[HN Gopher] Amazon Aurora DSQL
___________________________________________________________________
Amazon Aurora DSQL
Author : aws_hero
Score : 127 points
Date : 2024-12-03 17:30 UTC (5 hours ago)
(HTM) web link (aws.amazon.com)
(TXT) w3m dump (aws.amazon.com)
| benterix wrote:
| > Amazon Aurora DSQL is a serverless distributed SQL database
| with virtually unlimited scale, the highest availability, and
| zero infrastructure management.
|
| I will ignore their claimed almost unlimited scalability (which I
| somewhat believe as it's in their best interest to make it
| expensive fast), and "highest" (as compared to what?)
| availability. But my pet peeve is _zero infrastructure
| management_. This is blatantly false and it always was - when
| they first announced their cloud (at that time called "web")
| services, when they marketed their solutions as "serverless", and
| now. You need to have well-trained staff to maintain your AWS
| infrastructure otherwise you will come across many problems.
| gjsman-1000 wrote:
| I think this is way, way too pessimistic and unfair. When they
| say zero, it's like a food item that says zero calories. It
| might have up to five calories while still saying zero, but
| that's very different than three hundred.
|
| Managing a SQL Database at scale is more like four thousand
| calories daily, if I continue stretching this analogy. :)
| benterix wrote:
| I don't disagree about the service itself having relatively
| few calories (I'd argue it's far more then 5, but that's a
| minor point). The reality is that you need a ton of other
| things to have it running properly - from a properly created
| and maintained AWS account, to proper IAM roles with correct
| permissions, to correct deployment/modification setup and
| other resources as a database rarely exists in a vacuum. And
| because Amazon decided not to implement spending cap, if you
| don't have the right competences, you are at risk losing
| money - and this is especially true for serverless services.
| I still remember the folks who set the same bucket for lambda
| notifications look with panic as their bill increases
| dramatically as they watch it growing, not knowing how to
| stop the whole process.[0]
|
| [0] They disabled the trap a few weeks ago, finally:
| https://aws.amazon.com/about-aws/whats-new/2024/10/aws-
| lambd...
| ndriscoll wrote:
| > I will ignore their claimed almost unlimited scalability
| (which I somewhat believe as it's in their best interest to
| make it expensive fast)
|
| I would believe it can scale your cost easily, but I'd be
| surprised if it can scale well in terms of capability. How many
| million QPS is "virtually unlimited scale"? The 10k row
| transaction limit and no FKs (or even views???) also just make
| this thing sound completely non-serious.
| redwood wrote:
| In my experience, multi-region databases are used for HA/DR more
| so than for "writing locally from multiple places". What this
| means is that the multi-region "consensus cost" is a far easier
| problem. It looks like DSQL is aiming to change that paradigm:
| the question is, will this actually mean multiple-region writes
| become democratized finally? or will this remain more of a
| database industry marketing talking point that essentially no one
| uses?
| politelemon wrote:
| I am confused, what is the difference between Aurora DSQL and
| Aurora Serverless?
|
| https://aws.amazon.com/rds/aurora/serverless/
| nextworddev wrote:
| Aurora might be tied to region
| asah wrote:
| Aurora is much more compatible:
|
| https://docs.aws.amazon.com/aurora-
| dsql/latest/userguide/wor...
| everfrustrated wrote:
| Aurora is the postgres binary running with patches to write to
| their storage backend.
|
| This appears wire-compatible with postgres but does things
| differently like locking.
|
| Depends a lot on how you use postgres as to whether its drop-in
| compatible or not.
| octernion wrote:
| love that i can spin this up and spend money before any of the
| documentation is available. neat-o
| aws_hero wrote:
| it's free during preview
| kbumsik wrote:
| So does it scale to zero, and scale from zero with <1s latency
| like Neon DB?
|
| AWS' recent "serverless" products, Aurora Serverless v2 and MSK
| Serverless for example, had been very misleading with
| "serverless" title.
|
| I hope it is not a lie this time.
| rhodysurf wrote:
| its supposedly serverless like dyanamo
| everfrustrated wrote:
| There's nothing in the cluster api to control scale or set
| instance type so this does look actually serverless for a nice
| change!
| evanelias wrote:
| This was announced only a couple weeks ago, but Aurora
| Serverless v2 can scale to zero now.
| https://aws.amazon.com/blogs/database/introducing-scaling-to...
|
| However, the latency scaling _from_ zero is still painful, "up
| to 15 seconds".
| epaulson wrote:
| It's frustrating that there's no pricing information. The tech
| looks cool and all, but without knowing how much it's going to
| cost there's no way to really evaluate it.
| gs17 wrote:
| It's really surprising that they don't have a page with
| information ready to go live when the announcement happens.
| E.g. the Nova AI models they announced look neat, but the
| Bedrock page doesn't mention them at all and the page for them
| has links to non-existent documentation.
| estebarb wrote:
| Too bad the details how it works aren't available yet:
| https://docs.aws.amazon.com/aurora-dsql/latest/userguide/get...
| Stay Tuned We're sorry. The documentation you are
| looking for is not yet available. Please check back soon.
| -The Amazon Web Services Documentation Team
| lechomp wrote:
| I think like other products, there will be a tech presentation
| and there they will talk in details. So hopefully later this
| week?
| taikahessu wrote:
| You know true Cutting-edge Technology when you see ... there's
| no documentation available.
| gs17 wrote:
| Some of the docs are up: https://docs.aws.amazon.com/aurora-
| dsql/latest/userguide/wha...
|
| EDIT: Your link is up now too.
| slackerIII wrote:
| The only announcement I want from AWS about databases is that the
| price for RDS is going down. Not that they have new chip that is
| more expensive but offers better price/performance, but that my
| bill is actually going to drop.
|
| I don't trust them enough to use non-portable technology like
| this until they give me confidence they are committed to lowering
| prices.
| vosper wrote:
| > The only announcement I want from AWS about databases is that
| the price for RDS is going down.
|
| If the price hasn't been going up in line with inflation then
| it's been going down in real terms. Not as nice as the number
| going down, too, but it's not nothing. Over time AWS prices
| have significantly declined in real terms simply by not having
| been raised in nominal value.
| panarky wrote:
| The inflation rate for chips, memory, storage and networking
| is negative.
| steepben wrote:
| It's Postgres compatible, so not exactly non-portable
| asah wrote:
| Cough, not exactly:
|
| https://docs.aws.amazon.com/aurora-
| dsql/latest/userguide/wor...
| crgwbr wrote:
| Of all the services AWS has, RDS is one of the best values and
| lowest risks. The peace of mind and reliability it gives you on
| something as important as a DB is well worth the cost.
| taldo wrote:
| Sooo they're finally launching a Spanner contender?
|
| I'm itching to read more details into what this actually is under
| the marketing blab.
| steepben wrote:
| There's a blog on the technical details here:
| https://aws.amazon.com/blogs/database/introducing-amazon-aur...
| hobs wrote:
| The most interesting page is always quotas and limits:
| https://docs.aws.amazon.com/aurora-
| dsql/latest/userguide/CHA...
|
| Seeing the No to configurable for some of the settings is the
| most telling hard limits that we can see up front.
|
| Some Very Noteable ones are: Maximum size of all data
| modified within a write transaction: 10 MiB Maximum
| transaction time: 5 minutes
| bobnamob wrote:
| The 10MiB transaction size limit smells a lot like
| FoundationDBs transaction size limit
| MapleWalnut wrote:
| Spanner provides serializability. It seems like DSQL only
| guarantees "repeatable read".
|
| https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
| timmg wrote:
| > Sooo they're finally launching a Spanner contender?
|
| It may be even better in one important way: (if I understand
| correctly) for Spanner, you have to provision servers -- and
| keep them running. So GCP version of Spanner really makes no
| sense for anything that isn't huge.
|
| This sounds more like "Spanner as a Service" -- which is
| something I've always wanted (for my hobby projects, no less :)
|
| Unless I misundertood, anyway.
| chikinpotpi wrote:
| So... its like cockroach db but with the added feature of more
| vendor lock in?
| Andys wrote:
| Like cockroachdb but with REPEATABLE READ instead of
| SERIALIZABLE as the default and only option.
| sidcool wrote:
| It doesn't talk too much about data consistency.
| steepben wrote:
| There's a blog with more technical details:
| https://aws.amazon.com/blogs/database/introducing-amazon-aur...
|
| It provides strong consistency for cross-region transactions
| bradhe wrote:
| DSQL = DHTML for data?
| PowerfulWizard wrote:
| Very interesting. My dream is to have something like this, a KV-
| store, a blob store, and pubsub all behind the same interface.
| mdaniel wrote:
| "like this" meaning you want wire PG compatibility, so you'd do
| something like this? UPDATE kv SET value =
| 'alpha' WHERE key = 'beta.charlie'; UPDATE s3 SET value =
| $b64$good luck$b64$ WHERE key = '/some/s3/path'; LISTEN
| whatever;
| PowerfulWizard wrote:
| I don't know what I want exactly but I'm thinking along the
| lines that SQL is already doing a lot so it would make the
| most sense to start with a database interface and augment
| from there, to try to build a system to handle all the common
| forms of durable storage used by applications.
|
| The type of situation I'm thinking about is for example
| storing a blob in S3, storing metadata and a reference to the
| blob's path in a database row, sending a message into a queue
| to trigger some async processing, and updating a cache. It
| would be nice to be able to do this through a single API or
| service, and it would be really nice to do all this within
| some type of transaction abstraction that would allow all
| operations to pass or fail collectively, really really nice
| if the whole thing could be pay-as-you and scale
| horizontally-ish on shared infrastructure without managing
| nodes or slots or whatever.
|
| I'm not a Postgres user so I don't know how far you can get
| currently and I should probably look into it in detail.
| Coordinating blob/ject storage, database, and pubsub
| operations is a pain point for me presently. I think that
| overall system design is going to prevent a database-type
| system from being a good idea for blob storage but I would
| still like to see someone try to put three systems in a
| trenchcoat and try to make it work behind one interface.
| e1g wrote:
| Just tried a quick test cluster - Identifies as
| PG 16.5 No views/triggers/sequences No foreign key
| constraints No extensions No NOTIFY ("ERROR:
| Function pg_notify not supported") No nested transactions
| No json(b)
|
| Unsupported PG features are now online
| https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
| acedTrex wrote:
| I feel like no FKs and no jsonb is really burying the lede
| there lol
| shayonj wrote:
| oof! that's rough. I was also not particularly impressed with
| "limitless". Esp. because you can't have partitions on the
| postgres limitless offering.
| craigkerstiens wrote:
| It feels very disingenuous to say "Postgres compatible" and
| have this as a missing feature set. I'm sure they'd quickly
| argue it's wire compatibility, but even then it's a slippery
| slope and wire compatible is left open to however the person
| wants to interpret it.
|
| There is no 'standard' or 'spec' for what makes something
| Postgres wire compatible.
|
| This feels like a strong overreach on the marketing front to
| leverage the love people have for Postgres to help boost what
| they've built. That is not to say there isn't hard and quality
| engineering in here, but slapping Postgres compatible on it
| feels lazy at best.
| benesch wrote:
| > I'm sure they'd quickly argue it's wire compatibility, but
| even then it's a slippery slope and wire compatible is left
| open to however the person wants to interpret it.
|
| I actually think that they'd argue they intend to close the
| feature gap for full Postgres semantics over time. Indeed
| their marketing was a bit wishful, but on Bluesky, Marc
| Brooker (one of the developers on the project) said they
| reused the parser, planner, and optimizer from Postgres: http
| s://bsky.app/profile/marcbrooker.bsky.social/post/3lcghj...
|
| That means they actually have a very good shot at approaching
| reasonably full Postgres compatibility (at a SQL semantics
| level, not just at the wire protocol level) over time.
| paulddraper wrote:
| Yeah, this is another DocumentDB.
|
| They'd save space by listing the supported features. It's not
| useless, but no existing application would run on it.
|
| > Unsupported objects
|
| > * Databases - Aurora DSQL supports only one database per
| cluster at this time.
|
| > * Views
|
| > * Temporary Tables
|
| > * Triggers
|
| > * Types
|
| > * Tablespaces
|
| > * UDFs / Functions other than functions using language = SQL
|
| > Unsupported constraints
|
| > Foreign keys
|
| > Exclusion constraints
|
| > Limitations
|
| > ...
|
| > A transaction cannot modify more than 10,000 rows
|
| If you want a distributed PostgreSQL-lookalike, look at
| CockroachDB.
| gigatexal wrote:
| Epic fail then. Wow. That's DOA.
| refset wrote:
| _> Aurora DSQL isolation level is equivalent to PostgreSQL
| Repeatable Read._
|
| So it lacks Serializable isolation also, but I guess that's the
| secret to how it "offers the fastest distributed SQL reads and
| writes".
| bobnamob wrote:
| https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHA...
|
| The quotas page is always the most interesting for any aws
| service.
|
| 10Mb transaction size and 5min transaction duration limits are
| interesting limitations that kill any "everything lives in the
| db" notions that one might have.
|
| A fully transactional, scalable, distributed data store + queue
| + db + data stream with triggers is my endgame
|
| Also, the 128mb limit on transaction temporary storage seems
| _very_ limiting for big ol' analytic queries
| jcmfernandes wrote:
| > No foreign key constraints
|
| Wat?
| ralusek wrote:
| what index types does it support?
| everfrustrated wrote:
| Postgres compatibility depends a bit on what you rely on. This
| doesn't appear to be postgres under the hood, more wire-
| compatible.
|
| >Aurora DSQL is PostgreSQL compatible, which means that it
| provides identical behavior for most supported features,
| identical query results for all SQL features, and supports many
| popular PostgreSQL drivers and tools with minor configuration
| changes. Supported SQL expressions return identical data in query
| results, including sort order, scale and precision for numeric
| operations, and equivalence for string operations. With a few
| documented exceptions, such as synchronous replication, no-lock
| concurrency control, and asynchronous DDL execution, Aurora DSQL
| behaves comparably to PostgreSQL.
|
| Aurora DSQL supports core relational features like ACID
| transactions, secondary indexes, joins, insert, and updates. See
| Supported SQL expressions for an overview of supported SQL
| features.
|
| Aurora DSQL doesn't support all PostgreSQL features. For more
| information, see Unsupported PostgreSQL features.
|
| https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
|
| No pricing that I can find
| bragrights wrote:
| https://aws.amazon.com/rds/aurora/dsql/faqs/ - it is free for
| preview
| Edwinr95 wrote:
| I find it to be super limited, and I'm sort of struggling to see
| the point given all these constraints.
|
| No temporary tables, no foreign keys, no views, no more than 10k
| rows in a transaction.
|
| Except for some basic wire compatibility with the postgres
| protocol, I'd hardly call this a "database", and more a key-value
| store.
|
| https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
| gjsman-1000 wrote:
| > Except for some basic wire compatibility with the postgres
| protocol, I'd hardly call this a "database", and more a key-
| value store.
|
| Hopefully that keeps the pricing reasonable. :)
|
| But seriously, for a smaller CRUD app, this could be
| sufficient, even "magical," if the price is right. For my part
| though, the lack of multiple databases per cluster puts multi-
| tenant systems completely off the table. Now that you mention
| it, I almost wonder if this is a giant hack on top of
| Valkey/Redis...
| everfrustrated wrote:
| >the lack of multiple databases per cluster
|
| I don't think that will matter. This feels much more like
| DynamoDB where you're charged for GB used/stored and no infra
| cost, so no reason to nest databases
| Edwinr95 wrote:
| Isolation can be quite important. I work with a micro
| services architecture, where we assign a database per
| service.
|
| That makes it very easy to manage access to sensitive data.
| adobrawy wrote:
| You do not need logical database isolation if you have
| even stronger isolation with virtual clusters. The
| pricing and quota will be important factor for that use
| case.
| Edwinr95 wrote:
| But a smaller CRUD app wouldn't need "virtually unlimited
| scale". I'm very curious what their target audience is.
| gjsman-1000 wrote:
| I don't know, I manage a PHP app with about 10K users. If
| it doesn't cost much more, just paying to have the peace of
| mind that I never need to think about it again as my app
| grows, would not be a bad trade. This is assuming, of
| course, that the price is not extortionate.
| nilamo wrote:
| Possibly a game server? Between matchmaking, leaderboards,
| achievements, you'll never be updating 10k rows at once...
| karmakaze wrote:
| Despite this number of limitations, I imagine that it's still
| far more usable from an application than DynamoDB for many use-
| cases. Adding Jsonb support would make it even more competitive
| for some.
| ajbourg wrote:
| Seems like it's mostly AWS' answer to Cloudflare D2, but using
| Postgres as the frame of reference instead of sqlite. (I'm not
| really sure what to call Postgres in this situation since it's
| so limited it's clearly not any full version of Postgres in any
| respect. Postgres-ish)
| xuancanh wrote:
| AWS tends to prioritize performance and scalability over
| functionality, which is reflected in the design of DynamoDB,
| SimpleDB, and now DSQL. I'm also not a big fan of this style.
| It doesn't give customers the flexibility to choose their own
| trade-offs like Spanner does and assumes that customers can't
| make these kinds of decisions on their own.
| Andys wrote:
| The fact that it uses "Firecracker microVMs" and a few other
| things makes me think this is an S3-backed Lambda-based database
| that emulates Postgres on the wire.
|
| Especially when you consider the recent feature additions to S3.
| joshstrange wrote:
| I won't even consider this without pricing, I can't imagine how
| anyone could.
|
| Serverless Aurora is stupid-expensive so I can't imagine this is
| going to be cheap but I keep hunting for that perfect DB that can
| scale to 0 (or very low) and be priced well.
|
| Right now neon.tech is what I'm using and I'm very happy but this
| looks like it would be interesting IF the pricing is good.
| dbbk wrote:
| I'm also on Neon. Seems like the primary benefit of this would
| be multi-region support, which Neon doesn't have. But I do
| wonder if multi-AZ is actually just enough, has there ever
| really been a time when all AZs have gone down?
| dalyons wrote:
| yes, every couple of years there's a region-wide outage. On
| us-east anyway.
| bobnamob wrote:
| Oof, press release proof readers must have been hit by layoffs
|
| > With its innovative active-active distributed architecture,
| Aurora DSQL is designed for 99.99% availability in single-Region
| configuration and 99.999% in multi-Region configuration, with an
| innovative active-active, distributed architecture, making it
| ideal for building highly-available applications
|
| We get that its architecture is innovative and active-active
| zokier wrote:
| Can Amazon please get RDS product management ducks in a row? You
| got RDS For PostgreSQL, Aurora PostgreSQL (provisioned), Aurora
| Serverless v1, Aurora Serverless v2, Aurora PostgreSQL Limitless
| Database, Aurora Global Database, and now Aurora DSQL (with
| PostgreSQL compatibility). It's very confusing, the docs are
| confusing, and the even the tools are confused.
|
| I think this is pretty big contrast to something like S3 where
| the product feels far more clearly and coherently managed, and
| where they have released lots of big improvements/changes over
| the years without disrupting or muddying the core product in
| major ways.
| bradhe wrote:
| RDS generally is a massive, massive business inside AWS. I'd
| imagine the politics and organization generally is an absolute
| nightmare.
| korkybuchek wrote:
| > It's very confusing, the docs are confusing, and the even the
| tools are confused.
|
| Not to worry -- you can use Rufus...uh "Q"....uhh...Nova...to
| summarize the docs into a decision tree on which confusing
| choice to use!
| OJFord wrote:
| Except you can't, because it will just misundertand you or
| say something hopelessly generic.
|
| Even asking it something on the same page that it's
| advertised 'now supports asking about' completely failed for
| me - think it was ENIs or something; just said something
| completely generic, not at all about the instances I'd asked
| about which it advertised being able to do (and I happened to
| have a concurrent need for).
| dakiol wrote:
| > It's very confusing
|
| As a very naive person myself, isn't "being very confusing" one
| of the main points of AWS? They know they are big and nobody
| will get fired for using AWS. So, the more confusing their
| products are, the more benefits for them (e.g., because you
| don't know how much you will pay, because you are misusing
| their products, because perhaps you don't actually need their
| products, etc)
|
| So, any manager at AWS probably thinks this is all working very
| fine.
| alexey-salmin wrote:
| Is it? Every solution has some advantages, switching between
| them is relatively easy (especially if you can tolerate some
| writing downtime), and IMO it's not that confusing if you have
| at least some experience in the space.
|
| Otherwise if you're just staring from scratch, I'd pick
| provisioned aurora if your load is steady and serverless v2 if
| not. Then IO optimized if the db doesn't fit into memory and
| non-IO-optimized until it does.
| oblio wrote:
| > Aurora Serverless v1
|
| This is deprecated.
| bobnamob wrote:
| Marc Brooker's blog has a ~bunch~ bit more technical information
|
| https://brooker.co.za/blog/2024/12/03/aurora-dsql
|
| Specifically, transaction latency is constant relative to
| transaction statement count, even across region, so I guess
| that's something
| DonnyV wrote:
| Soooooo....its just a basic SQL database. No views, functions or
| extensions supported. You might as well just buy a Hetzner server
| and throw Postgres on it. https://docs.aws.amazon.com/aurora-
| dsql/latest/userguide/wor...
| zokier wrote:
| How many multi-region active-active PostgreSQL clusters have
| you operated?
| qaq wrote:
| Virtually unlimited scale :)
|
| Maximum storage GB per cluster: 100GB Maximum size of all data
| modified within a write transaction 10 MiB Max: 10K rows per
| transaction
| dbbk wrote:
| 100GB doesn't seem very unlimited
| bobnamob wrote:
| While I love to take potshots, the yes(configurable) next to
| that 100gb does some heavy lifting.
|
| I imagine that number is tunable up to multi digit tb if you
| your monthly bill is worth Jassy's attention
|
| https://docs.aws.amazon.com/aurora-
| dsql/latest/userguide/CHA...
|
| The more interesting (i.e. constraining) limits are
| transaction size (both row count and data size) and
| transaction time, which aren't configurable.
___________________________________________________________________
(page generated 2024-12-03 23:01 UTC)