https://github.com/kiwicopple/serverless-postgres Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} kiwicopple / serverless-postgres Public * Notifications You must be signed in to change notification settings * Fork 2 * Star 69 * 69 stars 2 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights kiwicopple/serverless-postgres This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 5 Commits docs docs oriole oriole .env.sample .env.sample .gitignore .gitignore README.md README.md docker-compose.yml docker-compose.yml View all files Repository files navigation * README Serverless Postgres (experimental) Serverless Postgres using Oriole, Fly Machines, and Tigris for S3 Storage. Overview This is a MVP for Serverless Postgres. 1/ It uses Fly.io, which can automatically pause your database after all connections are released (and start it again when new connections join). 2/ It uses Oriole, a Postgres extension with experimental support for S3 / Decoupled Storage. 3/ It uses Tigris, Globally Distributed S3-Compatible Object Storage. Oriole will automatically backup the data to Tigris using background workers. Usage Make sure you already have an account with Fly.io. Step 1: initialize your S3 store fly storage create # Keep the credentials, you'll need them in the next step Step 2: set up credentials Copy the sample env file and add your Tigris credentials from above: cp .env.sample .env Step 3: Running Postgres locally Start Postgres locally using docker compose up. (You may need to change the execute permissions on ./oriole/ entrypoint-s3.sh.) Step 4: Postgres usage Enable the Oriole extension: CREATE extension orioledb; Create a table and insert data: -- Create a table to store blog posts CREATE TABLE blog_post ( id int8 NOT NULL, title text NOT NULL COLLATE "C" ) USING orioledb; -- Insert 1 million blog posts INSERT INTO blog_post (select id, 'value' || id from generate_series (1,1000000) id); After this, you the Oriole background workers will store the data in Tigris. You can login to Tigris using fly storage dashboard to view the data: Serverless Postgres Deploy to Fly TBD Roadmap I wouldn't recommend using this in production just yet. The goal of this repo is to showcase Oriole and start gathering feedback from anyone who wants to test it out. Please submit any Oriole bug reports to the Oriole GitHub repo. * [ ] Deploy to Fly. I still need to document the secure deployment steps for Fly.io (PRs welcome). * [ ] Distributed read replicas. Oriole can have many read-replicas reading from the same S3 bucket. This is a good pairing with Fly.io that makes it simple to launch servers around the world. Not that if you do this it's very important that the read replicas do not write to the same bucket as your primary or the data will become corrupted. * [*] Distributed data, without Postgres replication. [DEL:Tigris will globally replicate objects that are less that 128 bytes. We would need to support global replication of all objects in this bucket if we want to create fast read replicas without Postgres replication.:DEL] + Tigris replicates any object that requires fast access regardless of the size. It is possible to control this per object. I need to test this implementation but it should work "in theory". * [ ] Non-forked Postgres. Oriole currently requires some patches to the Postgres TAM API. The goal is to make them available in Postgres core. Oriole Decoupled Storage Oriole has experimental support for S3. Oriole is a table storage extension for Postgres. It is designed to be a drop-in replacement for Postgres' existing storage engine. The Oriole storage engine's reduction in disk IO is significant enough that it unlocks performant databases backed by S3 compatible blob storage. Data most-often accessed is cached in local storage for performance. The data is synced with S3 asynchronously: S3 Workers Read more in the Oriole docs. About No description, website, or topics provided. Resources Readme Activity Stars 69 stars Watchers 2 watching Forks 2 forks Report repository Releases No releases published Packages 0 No packages published Languages * Shell 69.2% * Dockerfile 30.8% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.