[HN Gopher] Select 'Hello, World': Serverless Postgres Built for...
___________________________________________________________________
Select 'Hello, World': Serverless Postgres Built for the Cloud
Author : mxstbr
Score : 173 points
Date : 2022-06-15 14:23 UTC (8 hours ago)
(HTM) web link (neon.tech)
(TXT) w3m dump (neon.tech)
| melony wrote:
| How long before we get public access? I signed up for the beta a
| few weeks ago but I haven't heard back aside from having to fill
| in a bunch of surveys.
| nikita wrote:
| Let us bump you up. We received a lot of sign ups. Mostly due
| to the HN power and now onboarding as fast as we can as well as
| fixing small issues please email at beta@neon.tech. We will be
| asking for feedback in return.
| ranyefet wrote:
| I'm very excited about Neon, can't wait to use it in production
| :)
| nikita wrote:
| We will onboard you sooner in return for honest feedback and
| working with us through the early wrinkles. Please email
| beta@neon.tech
| ratherlongname wrote:
| How does this compare to a traditional Postgres deployment in
| terms of performance? Time-to-response for queries, latency and
| so on.
| nikita wrote:
| For many workloads it's on par. Few are slower. Pathological
| case, working set doesn't fit in memory allotted to pg but fits
| in file system page cache. There is not page cache in neon
| thomasfromcdnjs wrote:
| I find this one a little hard to believe.
|
| Regardless, what are some of the trade offs this
| implementation makes? (aka the cons)
| mattashii wrote:
| The cons are mostly:
|
| - Buffer cache evictions of freshly dirtied pages are bad.
| This is because we request pages with a hint on the latest
| change was evicted, and with newer and newer changes being
| evicted from buffers you might start to get limited by the
| write-through latency to Pageserver instead of only
| Safekeeper.
|
| - Commit latency can be not great due to cross-AZ
| communication -- with 3 safekeepers in as many AZs, the
| second slowest response is the limiting factor
|
| - Write amplification in the whole system is quite high.
| Plain PG does ~ 2x (WAL + Page), while we have many times
| that. Admittedly, these writes are spread around several
| systems, where any one system only really needs to write 1x
| WAL volume for the data that it is responsible for, but
| Pageserver is currently configured for something like 4x
| write amplification due to 4 stages of LSM-tree compaction.
| thomasfromcdnjs wrote:
| Thanks.
|
| So in layman terms, cache purging is slow (unreliable?)
| and writing is slower. (which is to be expected)
|
| Maybe another way of begging the question is, what apps
| would be suitable to build with Neon and what would not?
| j16sdiz wrote:
| S3 is not particularly good in latency. I guess there are lots of
| cache on pageserver? Must be lots of engineering work to keep it
| robust.
| bean_salad_123 wrote:
| Exactly. Latest database state and recent history is on the
| pageserver
| nikita wrote:
| Yes it's a lot of work. But a lot less compared to building a
| database from scratch. There is a blog coming out from Heikki
| and it covers the tricky spots
| moltar wrote:
| When will it be possible to self host this? Thank you.
| yeputons wrote:
| Depending on what parts you want to self-host.
|
| You can already build and run all base components on your local
| machine, see instructions here:
| https://github.com/neondatabase/neon/tree/d11c9f9fcb950ac263...
| . You can run the tests instead if you want more insight into
| how a particular piece works. You probably want to attach your
| S3-compatible storage to Pageserver and Safekeepers; some
| Ansible scripts with command-line flags are in the repository.
|
| To run Neon components on multiple machines, you should be able
| to create the `.neon` data directory via `neon_local init` and
| then share the generated configuration files across machines
| and tweak network settings. You can refer to our documentation
| to understand the terminology and the intended hosting
| configuration: https://neon.tech/docs/storage-
| engine/architecture-overview/
|
| However, there are still two missing bits: the self-hosting
| documentation and the Neon Control Plane (web UI + K8S-based
| compute nodes orchestrator). So you don't get automatic scale-
| to-zero at the moment out-of-the-box, although all hooks and
| the PostgreSQL proxy we're using at pg.neon.tech are there.
|
| We consider open-sourcing the Control Plane, so stay tuned. As
| for documentation and support, Nikita has already answered.
| nikita wrote:
| So you can selfhost our storage and we are making it easier and
| easier. Mostly we want to write one helm chart that will
| package it all.
|
| We are not planning to support on prem deployments
| commercially. We are working with partners like Percona to do
| it eventually, but those conversations are too early to commit
| to anything.
| [deleted]
| allaaloe wrote:
| Unbelievable! Great news, congrats!
| nikita wrote:
| Thank you! Somehow open source technologies often don't fully
| embrace cloud. This is an attempt to change that.
| uzochiapa wrote:
| Can the storage be switched from s3 to Cloudflare or R2 or Azure
| Blob?
| nikita wrote:
| Yes. There no dependencies on s3. Any object store will do. The
| next hill for us to take is more regions. Then more clouds
| uzochiapa wrote:
| This is amazing. Good product? Any update on pricing? I am
| trying to build a saas product that offers one serverless db
| per tenant.
| nikita wrote:
| We can start working with you on this please email
| beta@neon.tech
| ZeroCool2u wrote:
| If I could use this with Google Cloud Run, that would be a
| fantastic stack for so many side projects.
| nikita wrote:
| We would love to work with you on this. Please email
| beta@neon.tech.
| endisneigh wrote:
| How much will this cost?
| nikita wrote:
| Prob 60-70% margin over what is costs us.
| endisneigh wrote:
| How much is that, exactly?
| nikita wrote:
| We are not ready to answer this because pricing is always a
| model that is built on top of your COGS (cost of goods
| sold) and validated on real usage. We just started to
| onboard users - we have hundreds now, but it's just the
| first week of doing it.
|
| The rule of thumb of software cloud margins for
| infrastructure is 60-70%. So we need to be there over time
| at least. So sorry for non answer, I hope we will sort it
| out soon.
| Scorpiion wrote:
| Very interesting, tempted to apply to the company as this sounds
| very promising and fun to work on...
|
| I'm curious about the business side of things, to give some
| context, some open source models that startups use today are:
|
| 1. The "open core" model, Gitlab being a good example. They try
| to split features that are open or closed/enterprise depending on
| the buyer.
|
| 2. The AGPL model, Mongodb used to do this, today a popular
| example is Grafana and their collection of products.
|
| 3. The Apache + cloud backend model, the core being standalone
| working with Apache license while building a value added managed
| service. I think this is what Synadia is doing with NATS.
|
| 4. The "source available" model, not really open source, but
| worth mentioning as it's very popular recently. Examples Mongodb,
| Elastic, Cochroachdb and TimescaleDB. This is often combined with
| open source such that some parts are open source, others source
| available.
|
| With this as a reference Nikita, how would you explain how Neon
| thinks in regards to licensing and eventually building a healthy
| business? It's obvious a managed database service is the money
| maker, but how do you think around compeditors taking the project
| and building managed services without or with minimal code
| contributions? I'm sure you guys have thought a lot about this,
| would be interesting to hear some thoughts and reasoning for or
| against different options.
|
| (Note: This is not meant to be an extensive explanation of these
| business models just a high level overview. If I have
| miscategorized some company above feel free to correct me in a
| comment.)
| nikita wrote:
| It's 3. Our intention is to only monetize DbaaS revenue and
| opensource all/most of the tech with Apache 2.0 license. It's
| similar to that of Databricks. Databricks over time built out
| photon that is proprietary. We will stay away from this ideally
| forever. Enduring technologies are fully open source, we see an
| opportunity to build a standard scalable storage tier for
| Postgres and maybe for other engines over time (other engines
| are off strategy right now).
|
| Please do apply! we are hiring around the globe!
| btrees wrote:
| I am very tempted to apply because it's remote too, but do
| you also pay independent of the location?
|
| Also, do you ask leet code style questions in the 1:1 rounds?
| More details would be appreciated
| nikita wrote:
| Please apply! Are cost is location adjusted. In most areas
| we pay 90th percentile
| avinassh wrote:
| > Please do apply! we are hiring around the globe!
|
| Do you expect candidates to have background in database
| development?
|
| Also, any advice for someone looking to transition their
| career from backend to database development?
| nikita wrote:
| Systems >> databases for the engine. We also need fullstack
| or product engineers - modern DevX requires good UI. We
| also hire SREs and support
| nikita wrote:
| Tips for transitioning to database development: learn
| rust, start working on systems, ideally get a systems
| job, and optimize for being prolific. Write a lot of code
| Scorpiion wrote:
| Interesting and glad to hear, I think it makes a lot of sense
| in this case!
| speedgoose wrote:
| What is the consistency model?
| johtso wrote:
| https://github.com/neondatabase/neon/discussions/1853#discus...
| ksec wrote:
| So this is the PlanetScale version of Postgres?
| nikita wrote:
| People say that. The underlying tech is different. It's closer
| to aurora than to Planetscale. Planetscale is shared nothing
| which breaks MySQL compat. We are 100% Postgres compatible.
| metadat wrote:
| Looks like it's already released as FOSS. Stellar!
|
| https://github.com/neondatabase/neon
| nikita wrote:
| Don't forget to star us there :)
| ledgerdev wrote:
| What sort cold start times do you hope to achieve? If it's
| serverless and no cost while not using(aka scale to zero), you
| can't run those containers all the time. For comparison the
| serverless version of sql azure takes ~50 seconds to cold start.
| kelvich wrote:
| It is about 3 seconds to start now, with about 1.5 seconds out
| of 3 being k8s time quant. So we hope to drive it below 1
| second with some tweaks.
| ledgerdev wrote:
| Even 3 seconds is truly impressive!
| moltar wrote:
| Would be great to combine that with EdgeDB. Would be a dream!
| nikita wrote:
| I'm in contact with Yuri - CEO of EdgeDB. I'll make sure he
| sees this comment.
| dmw_ng wrote:
| Sounds very exciting, but curious about performance. Of course
| there will be downsides, it would be nice if this could be
| characterized somehow as part of the docs or site.
| nikita wrote:
| We will take a note of that. We are all catching our breath
| now. More will of course come! Thank you for pointing this out.
| mister_goo wrote:
| I'm not familiar with postgres or cloud architectures and I have
| a few dumb questions. Is the pageserver act as a page cache for
| s3? Does the postgres compute also have an internal cache? If so,
| this looks like multiple levels of memory cache connected by
| networks.
| nikita wrote:
| Yes. The issue is that compute doesn't have enough of the cache
| AND you need to have a scratch space to update pages. You could
| theoretically do it on the compute too and this would be a
| valid design. A bit harder to work with read replicas.
| mister_goo wrote:
| I read that pageservers are shared between users. What if the
| memory capacity of the pageserver becomes the bottleneck? eg:
| some users perform full table scans and make most page access
| fallbacks to s3. Sorry for one more dumb question.
| nikita wrote:
| The answer is more pageservers. Right now we have 1 to may
| relationships 1 page server many tenants. But one tenant
| one pageserver. We will shard pageservers and make it many
| to many. The good news is that pageserver workload is
| constant space so it's relatively easy to schedule. Unlike
| query processing workloads that have joins and those are
| not constant memory space.
| 7952 wrote:
| Looks cool!. Any support for PostGIS?
| nikita wrote:
| It's there!
| [deleted]
| agacera wrote:
| Depending on the cost per database, a service like this could
| allow one to build SAAS apps with Single Tenant model [1] in a
| really simple and elegant way.
|
| I always liked this model since it gives the best isolation and
| not having to code with multi-tenant in mind really simplifies
| development and security. But maintaing an infra for this is hard
| and costly.
|
| [1] https://docs.microsoft.com/en-
| us/azure/architecture/isv/appl...
| RedShift1 wrote:
| How do you handle upgrades for single tenant databases?
| nikita wrote:
| We shut down the old compute and start a new one with a new
| version (after loads of testing and making sure it'll work).
| nikita wrote:
| Yes, SQL Azure has customers with 10K+ tenants. I's anecdotal,
| but I'm pretty sure since I used to work at SQL Server!
|
| This makes it super easy.
| nikita wrote:
| Neon CEO here. Happy to answer any questions about Neon
| technology
| durkie wrote:
| can you describe how compute scaling works? I do a lot of work
| in PostGIS, which can have big CPU needs depending on the
| function being invoked, so my workload can look super bursty
| from a CPU perspective.
| kelvich wrote:
| We are still experimenting with compute scaling, and the tech
| preview includes only a small fixed compute container. We
| have a custom proxy in front of a compute, and we can quickly
| change the underlying compute container with a bigger/smaller
| one. That works fine if you don't use per-session semantics
| and transactions are short-lived. But there are a lot of
| tradeoffs on what to do if there is a long-lived transaction
| or session-level object. So if we need to scale up the
| container in the presence of long transactions, we can:
|
| 1) roll back long transactions and enforce upscale
|
| 2) wait for a better moment to upscale (potentially forever)
|
| 3) try to do a live migration of running Postgres to another
| node (like VM live migrations, or CRIU-like process
| migration) and preserve long-running transaction
|
| So far, we plan to start with some combination of 1+2 --
| should be fine for web/OLTP kind of load. But ultimately, we
| want to arrive at 3), but that approach has way more
| technical risks.
| koolba wrote:
| How does shared memory get handled across nodes? I'd imagine
| there's quite a bit of in memory state for things like
| sequences. For that specific example you could preallocate
| chunks of them as there's no guarantee of them being
| contiguous, but I'm guessing there's more complicated examples.
| [deleted]
| kelvich wrote:
| Our compute node is basically a usual postgres where we
| intercept WAL write and page read streams. In other words we
| don't do any compute sharding since we want to preserve
| vanilla postgres compatibility and any sharded solution will
| be riddled with a lot of issues like you've mentioned.
| panarky wrote:
| How do you differentiate against Google's AlloyDB, Cockroach
| Serverless, and fly.io?
| nikita wrote:
| AlloyDB is not serverless, cockroach is not Postgres, and fly
| doesn't separate storage and compute.
| jelder wrote:
| On the surface this looks pretty similar to AWS Aurora Postgres
| Serverless. Could you expand on the differences?
| mdaniel wrote:
| Not to steal the CEO's thunder, but this (tech|company) got
| quite a bit of commentary last time it was submitted:
| https://news.ycombinator.com/item?id=31536827
| nikita wrote:
| This was a good discussion indeed. The blog captures most
| of these points. We were pre launch then
| nikita wrote:
| Only serverless, only Postgres - that's focus.
|
| Multi-cloud and open source.
| gandreani wrote:
| I see how having multiple Postgres compute nodes scales up
| reads, which is great! Does separating the "write"
| functionality into the Pagekeepers service allow writes to
| scale as well?
|
| Seems like there's no upper limit for scaling up reads, just
| wondering how this architecture affects write throughput. Would
| love to hear more!
| nikita wrote:
| Yes you can't infinitely scale writes. You can make your
| storage bandwidth infinit but you are still limited to the
| how much Postgres WAL one instance can pump into the storage.
|
| I think we can do a lot of good things here over time and
| have plenty ideas. But for now it's a single writer system.
| Good news is that there is so much open source tech around
| Postgres that it might not be a gargantuan task in the future
| avinassh wrote:
| What's the 'page' in Pageserver? Is it a B-tree page? Or disk
| page?
|
| I have a hunch that Pageservers contain the disk pages, so
| storing the B-tree (or may be LSM) pages and compute traverses
| those pages to find the relevant page/pages. I am curious about
| how does it fit in together and fetches from disk/Pageservers
| work
| nikita wrote:
| It corresponds to Postgres 8k page. We store them in an LSM
| tree so we can move layers into S3
| throwaway892238 wrote:
| I see there's branching support. Are there plans for version
| control features? I'd like to be able to treat the database as
| a collection of versioned immutable artifacts. Diff database
| branches, merge/patch, revert to a previous known good version.
| This would allow me to eliminate "migrations" and simply stage
| a set of changes, test them, and then merge or revert them,
| knowing I'm getting exactly what worked before without writing
| additional code to ensure it or walk it back.
| nikita wrote:
| I think we need more details to answer this question. Can you
| describe your workflow with more details? Maybe even with
| some code examples ?
| throwaway892238 wrote:
| Use Case 1. Migrations.
|
| Everybody today knows that if you're gonna change your
| database [schema], you need a system to migrate the DDL
| changes intelligently so you don't break something, and
| then run that in lower environments to test, then promote
| it up to higher envs and run it, then deploy your apps that
| use the changes. But of course it may be _impossible_ to
| revert those changes after that point, requiring an entire
| database snapshot restore. So not only are there serious
| operational concerns, making any operations around this
| time-consuming and frought with peril, but you need to set
| up a migration solution (language-specific or framework-
| specific or agnostic) and make sure you architect your
| application to only make changes in a specific way. (all of
| this, by the way, is only necessary because the database is
| one big mutable state machine)
|
| ...whereas if it worked more like version control, you
| could make _any_ change, commit it, and get a commit ID. If
| that causes problems, if you could just _`cloudpg revert
| $change_id`_ , then there would be no need to carefully
| architect the app, changes wouldn't be fraught with peril,
| and we could be more agile with database-driven design. The
| database would obviously need to be intelligent enough to
| figure out _how_ to revert any change, which is why this
| has to be a database-specific feature and not just a _git
| revert_.
|
| Use Case 2. Merging Changes
|
| This sort of follows on the above (making changes more
| agile). If you have branches, and 4 different devs are
| working on 4 different database changes, how do you merge
| and deploy those all safely, and handle reversions safely?
| Well if all database changes had versions, and we could
| diff the changes between versions, then we could treat the
| database like a Git repo and merge/rebase all the changes
| to the database together at the same time as the code.
| Again, no need to go back and refactor migration scripts or
| the app design, because the database is essentially just
| version-controlled code.
|
| The same things would apply to upgrading/downgrading
| database versions, bringing up or restoring new servers,
| possibly even making replication easier, maybe other things
| we haven't thought of yet.
| claytonjy wrote:
| For migrations, how would you want neon (or anything
| else) to handle all the writes that happened since the
| commit you want to return to?
| throwaway892238 wrote:
| It would need to revert only the changes in that commit
| while preserving all new changes afterwards. I don't have
| the foggiest how to make that happen, that's up to the
| ninjas at neon!
|
| There would probably need to be a fallback mechanism if,
| for example, a new column was created and new data was
| entered into it, and then the revert removes the column.
| Probably it could keep pointers to such things ("there is
| a database D with a table T with a column C and rows
| [a,b,c,d]"), so that if the change is re-reverted later,
| an extra merge instruction could pop the reference back
| into place like nothing happened. Somebody with an actual
| CS background must have better ideas than me :)
| nikita wrote:
| This is a tricky problem to solve. Reverting to the
| commit - that's easy and we have all the infrastructure
| already to allow you to do that. The tricky part is
|
| * Make a schema change
|
| * push into prod
|
| * accumulate some new data
|
| * revert just the schema change
|
| In our discussions we call this separating schema and
| data and allow you to have different schemas on the same
| data.
|
| It's tricky to do in Neon due to the fact that storage
| knows nothing about schemas. It stores page with no idea
| what's on them.
|
| But we have some ideas how to do this with logical
| replication where we will run a transform on top of
| logical replication stream to keep two branches in sync.
| Not this year though.
| chwzr wrote:
| This would be the killer 2050 alien feature. Git is a
| really good mental model on itself. Git + Neon = you
| invent the future!
| mattashii wrote:
| It would be a killer feature, but the thing is that Git
| has a conflict resolution feature that puts the working
| tree in an invalid state until the user has manually
| fixed certain issues, so that you cannot commit until you
| manually fix the files and mark those as such.
|
| Doing so for a database seems less desirable from an
| availability perspective, especially with high-throughput
| databases.
| mike_hearn wrote:
| Does your Postgres support all the features? What about things
| like writing SQL functions in Java or other languages?
| nikita wrote:
| All the features. It's the same Postgres that you use
| elsewhere
| mike_hearn wrote:
| Huh, interesting. So your postgres can basically run
| arbitrary code on behalf of the user? I'm talking about
| stuff like https://tada.github.io/pljava/ where you can use
| the DB to invoke whatever code you like, outside of any
| sandbox. Like, could I upload a PL/Java function that
| probes your internal network? How are you making that
| secure?
| yeputons wrote:
| That is true. And that is why we do not have the UI for
| loading extension binaries and do not give root access to
| the compute node. Yet. Of course, some containerization
| is in place, but it is not as tight as we would like for
| arbitrary code execution.
|
| Still, there are no technical limitations. Our test suite
| already uses Neon-specific SQL functions from a C
| extension (https://github.com/neondatabase/postgres/tree/
| 7faa67c3ca53fc...). At the very least, providing a lot of
| popular extensions out-of-the-box is on our roadmap once
| we figure out the security, no special repacking needed.
| As compute nodes should already be pretty isolated from
| each other, I don't think allowing arbitrary code will
| require a redesign.
| andrewbarba wrote:
| Are there plans to release an HTTP API to make it easier to use
| with services like Fastly Compute@Edge and Cloudflare Workers?
| And if so would the API be global or region specific?
|
| One thing I haven't seen with "serverless" databases is an easy
| way to dictate where data is stored. Mongo has a pretty clever
| mechanism in their global clusters to specify which
| regions/nodes/tags a document is stored in, and then you simply
| specify you want to connect to nearest cluster. Assuming your
| compute is only dealing with documents in the same region as the
| incoming request, this ends up working really well where you have
| a single, multi-region db, but in practice reads/writes go to the
| nearest node if the data model is planned accordingly.
|
| A real world example of how I am using this in Mongo today: I run
| automated trading software that is deployed to many AWS regions,
| in order to trade as close to the respective exchange as
| possible. I tag each order, trade, position, etc. with the
| exchange region that it belongs to and I get really fast reads
| and writes because those documents are going to the closest node
| in same AWS region. The big win here is this is a single cluster,
| so my admin dashboard can still easily just connect to one
| cluster and query across all of these regions without changing
| any application code. Of course these admin/analytics queries are
| slower but absolutely worth the trade off.
| nikita wrote:
| Absolutely! We are working on it right now and call this
| "regions". We already have a proxy - you will notice that the
| connection string is project_name.cloud.neon.tech.
|
| We are working on deploying the proxy globally and routing read
| traffic to the nearest region.
|
| We also have some multi-master designs in collaboration with
| Dan Abadi. But this will take a second to build.
| gunnarmorling wrote:
| Congrats on the launch! Does it support logical decoding plug-
| ins? I.e. could I use for instance Debezium for streaming changes
| out of Neon to Kafka etc.?
| mattashii wrote:
| Any plugin that doesn't access the file system directly but
| uses the appropriate file system and buffer manager APIs should
| work. But because we scale to 0 when we detect no activity we
| might have issues with background tasks, so your milage may
| vary.
|
| Next, we have not yet optimized for replication outside Neon
| Cloud, nor do we have sideloading of extensions, so to use
| Debezium you'd have to self-host Neon for now.
___________________________________________________________________
(page generated 2022-06-15 23:01 UTC)