[HN Gopher] The dangers of conditional consistency guarantees
___________________________________________________________________
The dangers of conditional consistency guarantees
Author : greghn
Score : 28 points
Date : 2023-06-30 10:18 UTC (12 hours ago)
(HTM) web link (dbmsmusings.blogspot.com)
(TXT) w3m dump (dbmsmusings.blogspot.com)
| deathanatos wrote:
| The systems I work with seem to choose the even worse "PA/EA", to
| use the article's terminology, i.e., consistency never, even
| during apparent normal operation.
|
| We had whole slew of bugs that got discovered by this: we use a
| vendor for storing Docker images, and AFAICT empirically, their
| system does not obey the property "Read-Your-Writes". We knew
| this already about it for _images_ , but what tripped the other
| day seemed to be that _layers_ are also susceptible: so when the
| manifest of a new image was uploaded, it referenced layers _that
| the client has acknowledged writes for_ , but the write of the
| manifest apparently failed because the layer "didn't exist".
|
| (The script that was uploading the images was also written in
| Typescript, and Deno's API for executing child processes uses
| C-style "return the error" handling1. The script forgot to check,
| so our CI run "passed", all while missing an image. A deployment
| then bumped the image versions to the new (non-existent) version,
| and that deploy failed. Thankfully k8s keeps running the old
| versions, but the number of swiss cheese holes that went through
| was pretty astounding.)
|
| So many systems fail read-your-writes, and it just makes them all
| the more annoying to deal with. Inevitably you are writing to a
| system because you want to do something _with_ that write: I 'm
| not persisting data into a black hole, never to be see again,
| after all. But sanely serializing the next action in the sequence
| when you do not know if the write is "good" yet is impossible,
| and makes things brittle.
|
| (We asked the vendor about this, and the response from support
| types / vendors is just so utterly disappointing. "Can't you just
| write a loop to check if the image appears to exist?" This is one
| of the many spots where CS theory matters in the actual
| engineering! No, no I _cannot_ "just" write a "simple" loop to
| check, and the theory can tell you why that doesn't work. Not to
| mention the layers/manifest part means I need to re-write `docker
| push`...)
|
| (1but child process APIs are rife with this. Heck, even Rust's
| would bite you all the same. AFAIK, I think only Python exposes
| safe-ish APIs in this area?)
| YZF wrote:
| 2019 ...
|
| Interesting read though. I've seen Hazelcast in the wild but
| didn't really hear much about it.
| thamer wrote:
| Great insight from Abadi as always. To add to his point about
| guarantees and network partitions: I find that the way
| consistency is explained in documentation or even courses can be
| somewhat misleading or at least incomplete. If you read the
| Dynamo paper and take from it that you could model a system with
| a "classic" design of 3 replicas doing quorum reads and writes,
| you could easily convince yourself that it will be consistent and
| continue operating even with the loss of a replica.
|
| What this simple approach doesn't cover is _all_ the cases that
| aren 't your perfectly well-behaved read and write operations. It
| doesn't cover the case where your client gets a timeout and has
| no idea how many replicas were written to. You could have one
| replica that has persisted the (failed) write, and read back an
| old copy from the other two. Then your next read could pick up
| the new version, giving you alternating views of this data that
| was supposed to be consistent.
|
| With a database like Cassandra you also have to consider that
| only the most recent copy of a value is returned, regardless of
| how many replicas responded. In that case you could read an old
| value, then the new one, then the old one again, all at quorum
| when it was supposed to be consistent.
|
| Failures are often messy, and the source of most of the
| complexity in reasoning about consistency in distributed systems.
| Many concepts (like network partitions) are also often
| misunderstood, or being considered in a way that's too
| "theoretical". Nodes don't always shut down instantly. You likely
| won't see a clean network split that starts at a fixed point in
| time and is also resolved in an instant. It'll be partial,
| asymmetric, with nodes coming in and out, etc. Good luck
| reasoning about these scenarios...
| EdwardDiego wrote:
| > Nodes don't always shut down instantly. You likely won't see
| a clean network split that starts at a fixed point in time and
| is also resolved in an instant. It'll be partial, asymmetric,
| with nodes coming in and out, etc.
|
| Excellent point, it's precisely this scenario that has caused
| the biggest issues in distributed systems I've been working
| with.
___________________________________________________________________
(page generated 2023-06-30 23:02 UTC)