[HN Gopher] Show HN: Serverless Postgres
___________________________________________________________________
Show HN: Serverless Postgres
This is a MVP for Serverless Postgres. 1/ It uses Fly.io[0], which
can automatically pause your database after all connections are
released (and start it again when new connections join). 2/ It
uses Oriole[1], a Postgres extension with experimental support for
S3 / Decoupled Storage[2]. 3/ It uses Tigris[3], Globally
Distributed S3-Compatible Object Storage. Oriole will automatically
backup the data to Tigris using background workers. I wouldn't
recommend using this in production, but I think it's in a good spot
to provoke some discussion and ideas. You can get it running on
your own machine with the steps provided - connecting to a remote
Tigris bucket (can also be an AWS S3 bucket). [0] https://fly.io
[1] https://www.orioledb.com/ [2] Oriole Experiemental s3:
https://www.orioledb.com/docs/usage/decoupled-storage [3] Tigris:
https://www.tigrisdata.com/
Author : kiwicopple
Score : 76 points
Date : 2024-05-29 16:54 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| thomasfromcdnjs wrote:
| When would it make sense to use such a setup? What kind of
| applications?
| cowboylowrez wrote:
| I like to think this sort of project makes sense when putting
| screendoors on submarines.
| kiwicopple wrote:
| The "scale to zero" aspects mean that it's useful for very
| infrequently accessed applications
|
| Once I have experimented with Tigris' global data replication
| then it could be useful for a distributed user-base doing a lot
| of reads. The idea is that you can point a fresh postgres
| instance to the S3 bucket for read access (rather than keeping
| a hot standby using Postgres native replication)
| internet101010 wrote:
| You have data updates that happen at irregular intervals and
| don't want to be bothered with turning on and turning off a db
| instance.
| sea-gold wrote:
| How does this compare to Neon[1], Crunchy PostgreSQL for
| Kubernetes[2], and Citus[3]?
|
| [1] https://neon.tech / https://github.com/neondatabase/neon
|
| [2] https://access.crunchydata.com/documentation/postgres-
| operat... / https://github.com/CrunchyData/postgres-operator
|
| [3] https://www.citusdata.com /
| https://github.com/citusdata/citus
| craigkerstiens wrote:
| At very first glance this would be much closer to neon, with
| the separated storage and compute.
|
| Crunchy Postgres for Kubernetes is great if you're running
| Postgres inside Kubernetes, but is more of standard Postgres
| than something serverless. Citus also not really serverless at
| all, Citus is more focused on performance scaling where things
| are very co-located and you're outgrowing the bounds of a
| single node.
| scirob wrote:
| >citus is a bit out of date, now mostly native azure not
| flexible at all >neon has not released all of their kubernetes
| operator so its not fully open source yet > Crunchy is just a
| kubernetes operator for HA postgres. but they are working on a
| cool thing to natively use IceBerg as FDW. But their attitude
| is that this serverless separation of compute and storage is
| not the right approach for the primary operational data store
| canadiantim wrote:
| Very cool! I'm glad to see someone working with oriole, been
| wanting to do something similar for a long while.
|
| How tricky is it to with the S3 storage? Esp. In regards to
| permissions, etc.?
|
| Thanks for sharing eh, love what you're doing
| kiwicopple wrote:
| I think it's pretty simple with S3, and the permissions are
| basically "keep the bucket private"
|
| That said, this is very much an MVP and I'm sure I'll encounter
| some dragons as I push for scale or as I receive feedback from
| customers
| justinclift wrote:
| This seems more like "PostgreSQL using remote storage for the
| database" rather than "Serverless"?
|
| It seems like it still needs memory/cpu resources locally (sized
| properly) to handle the returned data.
|
| It also seems like an _incredibly_ bad idea to use with any
| remote data store that charges for network traffic. ie Amazon AWS
|
| You could easily give yourself a many thousand dollar bill for a
| month if you start using that backend data store in a serious
| way. :( :( :(
|
| ---
|
| That being said, it's a nifty idea conceptually. Just be aware of
| likely operational costs if you try using it. (!)
| kiwicopple wrote:
| I guess the term "Serverless" is pretty ambiguous, but since
| this originated from AWS Lambda (firecracker) and that's the
| same primitive as Fly.io then it seemed fitting. You can scale
| up/down compute and snapshot memory for fast startups
|
| The other tricky thing is data: there are several ways to
| handle this. Since Tigris has a lot of the data replication
| features that Postgres would traditionally handle then it looks
| promising. That said - it comes with tradeoffs that you point
| out (transfer costs) but these are also costs that you'd pay
| for postgres replication. I'll try benchmark some of these
| costs and compare them across several implementations. At this
| stage I just wanted to get an MVP working: the harder stuff is
| still ahead
|
| Thanks for your comment - a lot of good points
| justinclift wrote:
| > these are also costs that you'd pay for postgres
| replication.
|
| Some places define a 2nd (private) network interface for
| their VMs, and you can transfer data between your VMs over
| that private interface without cost.
|
| Not sure if AWS does it that way though.
| victorbjorklund wrote:
| Very cool.
| scirob wrote:
| now it would be mega cool if someone got a grant to do this with
| the open source stack. fly uses firecracker underneath etc...
| scirob wrote:
| Oh fun this is from the CEO of supabase, so i guess that's
| where the grant is coming from :))))
| scirob wrote:
| double funny, looks like they already acquihired the main
| devs from Orioledb
|
| https://github.com/akorotkov now supabase
| https://github.com/pashkinelfe now supabase
|
| silicon valley isn't capitalism awesome
| re-thc wrote:
| They acquired Oriole, see:
| https://supabase.com/blog/supabase-aquires-oriole
| BurnGpuBurn wrote:
| Serverless != serverless
|
| You use a lot of servers, you just call them resources. Your S3
| storage is a server, your background workers run on a server, and
| your postgress, well, it doesn't run on a bike.
|
| I hate this confabulating stuff. It's like people believing "the
| cloud" is an actual cloud. Couldn't we just leave this stuff to
| the marketing people?
| INTPenis wrote:
| Nevertheless serverless is an actual concept where you don't
| deploy any server instances, you use only services. Those
| services might scale up with server instances, but you don't
| have to know about that.
|
| I was against it too at first, but then I realized how to use
| it.
| tmountain wrote:
| Novel, and maybe I am just old school, but I truly believe that a
| right sized Postgres deployment is good enough for most use
| cases. I would trust my data to something vanilla barring a super
| niche scenario.
___________________________________________________________________
(page generated 2024-05-29 23:00 UTC)