https://github.com/splitgraph/seafowl-gcsfuse Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} splitgraph / seafowl-gcsfuse Public * Notifications * Fork 0 * Star 23 Scale to zero Seafowl hosting with Cloud Run www.splitgraph.com/blog/deploying-serverless-seafowl 23 stars 0 forks Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights splitgraph/seafowl-gcsfuse This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/s] Use Git or checkout with SVN using the web URL. [gh repo clone splitg] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @onpaws onpaws Update README.md ... e485177 Jun 6, 2023 Update README.md e485177 Git stats * 10 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time configs Update seafowl.toml to match the actual GCS config May 24, 2023 04:14 .dockerignore initial commit April 22, 2023 05:33 Dockerfile Consume seafowl nightly (not a fixed release) April 27, 2023 04:39 README.md Update README.md June 6, 2023 12:03 docker-compose.yml Introduce a docker-compose.yaml April 27, 2023 04:40 gcsfuse_run.sh Increase gcsfuse logging verbosity April 27, 2023 04:36 seafowl.toml Extract dummy seafowl.toml into own file April 27, 2023 00:11 View code Scale to zero Seafowl via gcsfuse How to use Steps What's in the repo README.md Scale to zero Seafowl via gcsfuse Because Seafowl was architected with the cloud in mind it's a good candidate for serverless/'scale to zero' hosting e.g. Cloud Run and similar. Database objects can be stored in S3 buckets, which avoids depending on a persistent volume. Meanwhile Seafowl's catalog can be backed by SQLite, also good for the scale to zero story because it avoids the usual persistent Postgres process. Platforms like Lambda and Cloud Run will forward incoming HTTP requests to the waiting Seafowl service, which is the ideal time for it to fetch the up-to-date SQLite catalog so the query can be handled with fresh data. By adding gcsfuse to the Seafowl container, the SQLite file is mounted from the bucket into the Seafowl container. While there is a performance penalty in doing so, the catalog is only metadata, not the raw database objects, so the penalty (at least observed so far) is negligible. Plus, so long as traffic is within the same region, GCP doesn't charge for bucket <-> Cloud Run traffic, an additional bonus for hosting costs. How to use Please check out blog post for the step by step details on how to set this up. In the end you'll have a bucket like this: [243754872-e9c1a8d2-8c5c-4bf8-a217-f78e56ad1300] and some endpoint similar to https://seafowl-gcsfuse-YourEndpointHere.a.run.app/q you can query from a browser, or your backend. Steps * Build the Docker image (or if you want, use the prebuilt image splitgraph/seafowl-gcsfuse) * Make it avaialble to Cloud Run (e.g. push to hub.docker.com or your own repo) * Deploy a new Cloud Run instance using this image, including the secrets * Test the endpoint using e.g. curl curl -i -H "Content-Type: application/json" \ -X POST "https://your-endpoint-goes-here.a.run.app/q" -d@- <