[HN Gopher] You cannot have exactly-once delivery (2015)
___________________________________________________________________
You cannot have exactly-once delivery (2015)
Author : babelfish
Score : 96 points
Date : 2023-03-01 18:52 UTC (4 hours ago)
(HTM) web link (bravenewgeek.com)
(TXT) w3m dump (bravenewgeek.com)
| tonymet wrote:
| the workaround is de-duplication & idempotency
| twawaaay wrote:
| Of course you can have exactly-once delivery. I mean... we know
| how to construct software that will transfer money from one
| account to another, exactly once. It really isn't rocket science
| but it does require a little bit of understanding of various
| tradeoffs that you are making.
|
| It is a bit like saying that we can't have straight lines. Of
| course, if you zoom in far enough to see individual atoms, every
| physical surface will look jagged. But in practical terms we can
| have straight lines and surfaces to a good enough approximation.
| It means specifying what "straight" means and figuring out how to
| measure it and how to produce "straight" according to
| specification and measurements.
|
| Engineering is about knowing and making tradeoffs. Every device
| we have ever created has to contend with limitations of physical
| existence. Engineering is about accomplishing goals in presence
| of those limitations.
|
| A person who says "you can't deliver a message exactly once"
| clearly lives in an idealised, theoretical world. I would urge to
| leave your ivory tower for a second and see how engineers in real
| world accomplish what you say is not possible.
|
| I get that this knowledge is useful -- but don't publish it as
| gospel. "You cannot have exactly-once delivery" is true, but not
| the same kind of truth as "you can't travel faster than light".
| No engineering can get you to travel faster than light. But
| engineering can get you as close to exactly-once delivery as you
| want to the point where the original statement stops being
| meaningful for real life problems.
| skybrian wrote:
| You can't do any transfer by any particular deadline if the
| network is down.
|
| Assuming the network will recover in time may be a reasonable
| assumption sometimes, though.
| [deleted]
| [deleted]
| dllthomas wrote:
| If "the network might be down arbitrarily long" is a
| possibility, you cannot have at least once delivery, either.
| preseinger wrote:
| technically you can, as at least once delivery is satisfied
| by not delivering :)
| [deleted]
| preseinger wrote:
| You can get exactly-once in a system if you design for
| consistency (in the CAP sense) and use a consensus protocol.
| those systems don't offer availability (in the CAP sense) by
| definition. and I guess when people say exactly-once is
| impossible they're speaking about systems that offer
| availability.
| urbandw311er wrote:
| Have you actually read the article? The title is just a summary
| and the author fully acknowledges that their argument is
| essentially based around edge cases, not that this in any way
| diminishes it for me.
|
| It's just an interesting piece of theorising.
|
| I think your comment (particularly your 4th paragraph about
| ivory towers etc) comes across as overly harsh and a little
| aggressive.
| bob1029 wrote:
| > I mean... we know how to construct software that will
| transfer money from one account to another, exactly once.
|
| Assuming you are talking about transferring between
| institutions, there is actually no _single_ piece of software
| with this responsibility. The business processes are
| effectively what provide these guarantees (typically by way of
| another 3rd party).
|
| In order to accomplish this, added latency (settlement time) is
| necessarily introduced into the process.
| monsieurbanana wrote:
| I don't understand where the anger comes from, the article
| makes it clear it's talking about distributed systems theory.
|
| Like someone else said, you can use at least once delivery and
| handle duplicate messages, but that's not quite the same as a
| distributed system guaranteeing that a message will be
| delivered exactly once.
| junon wrote:
| .... they said, misunderstanding the article entirely.
|
| This isn't an opinion. This is a fact of distributed systems.
| An axiom, if you will.
| convolvatron wrote:
| there is no distributed consensus. however we can reach
| consensus with an _arbitrarily high_ probability by adding
| additional machinery. until the point where is equally likely
| that the earth will simply fall into the sun one day.
|
| so while this is a hugely important result, it doesn't stop
| us from building useful systems.
| iainmerrick wrote:
| This article spends a long time discussing idempotency but only
| mentions deduplication in passing.
|
| Can't you de-dupe by attaching a UUID to messages?
| xg15 wrote:
| Maybe I didn't get the point. Of course we can't have exactly-
| once delivery directly in the layer of an unreliable network -
| but it seems pretty easy to construct it on a higher layer if
| your network stack supports at-least-once delivery: Just assign
| each message a unique ID during sending, then track those IDs in
| the receiver side and discard duplicates. And you need those IDs
| anyway so you can generate ACKs.
|
| Isn't this basically what every "reliable" transport (TCP, HTTP3,
| message queues...) does?
| oh_sigh wrote:
| The context is exactly once in a distributed system. When you
| construct that higher layer you will make your system highly
| coordinated, thus no longer a distributed system.
| xg15 wrote:
| I mean, the property of distributed systems is that they crop
| up everywhere you have an unreliable transport and generally
| only bring downsides. If you could un-distribute your system
| that easily, I bet that would make a lot of people very
| happy.
|
| The OP defines "distributed systems" like this:
|
| > _Web browser and server? Distributed. Server and database?
| Distributed. Server and message queue? Distributed._
|
| By that definition, as soon as I have a server, a client and
| an unreliable connection between them, I have a distributed
| system. In that context, nothing stops me from counting IDs.
| karpierz wrote:
| When you receive a message in your scheme, you have to do
| the following:
|
| 1. Some action with a side-effect (ex: update an entry on
| disk, send a message out to some third party, etc.). This
| might be a bank transaction, or a note saying "you gotta
| ship package X to person Y".
|
| 2. Some action to note that you've received ID X (ex: write
| to disk, send a message out to your DB, etc.)
|
| How do you set up your server to deterministically do
| neither in event of a crash, and both in event that your
| code turns to completion?
| hangonhn wrote:
| So when does the receiver record the IDs? When it receives the
| message but before processing or after it's processed the
| message? If the former, then what if it goes down during
| processing? Then the other receivers will keep rejecting the
| message even though it's never processed. So now it's less than
| once. If the receiver records it after it's done processing,
| then it could go down after processing but before recording it
| in the DB. So now you have more than once.
|
| Also, isn't the assumption here that you will have a reliable
| connection to a shared DB?
|
| You can have engineered solutions that is pragmatically close
| to deliver exactly once but it's not "pure" -- there are still
| scenarios, however unlikely, that it will fail.
| PaulKeeble wrote:
| Even if you record both when it arrives and when its
| completed a power failure mid way will still leave everything
| in between in a partial state unless everything has XA
| transactional behaviour and as we know that scales quite
| poorly.
| rowls66 wrote:
| XA scales well enough for many workloads. I believe that
| too many developers discard solutions like XA because it
| "scales quite poorly" without doing serious analysis of how
| much scalability they are likely to need and whether XA
| scales well enough to support it. On the flip side I
| believe that too many developers underestimate the
| complexity of managing state and failure in distributed
| systems.
| LgWoodenBadger wrote:
| XA transactions have the exact same problems, it doesn't
| solve it either.
| btilly wrote:
| Wherever you construct it you must necessarily have a machine
| whose failure mode is that "exactly once" degrades into either
| "at most once" or "at least once".
|
| What determines which failure mode you get is whether the
| machine will failover to a machine that retries uncertain
| messages (giving you "at least once"), or it doesn't (giving
| you "at most once").
|
| But, you say, why can't we have it failover to a machine that
| asks recipients what they have got and goes from there? Well we
| can, but the recipients don't know what messages are in the
| network still on their way to them.
|
| But, you say, why not have the recipients disregard those
| inbound messages once they know about the replacement machine?
| Well you can do that, but now the *recipients* become machines
| whose job is to ensure the deduplication. And now *they* become
| the machine with a bad failure mode.
|
| But, you say, does this not reduce the odds of failures? Why
| yes, it does. Which is why people do things like this. And
| there has to come a point where we accept SOME failure rate.
|
| The alternative, well, read _The Saddest Moment_ at
| https://scholar.harvard.edu/files/mickens/files/thesaddestmo...
| to see where madness leads.
| dooglius wrote:
| > And now _they_ become the machine with a bad failure mode.
|
| What is the failure mode the recipients have here?
| nimih wrote:
| This issues are in the context of distributed systems where you
| want to be able to recover from losing a receiver (f.ex., we
| want to be able to reassign partitions for a Kafka topic when a
| consumer goes down). If you don't mind your system grinding to
| a halt whenever you lose one of your receivers (that's
| perfectly fine in some circumstances!), then your proposed
| solution works great.
|
| Edit: also, i should be fair and acknowledge that you're
| effectively describing idempotency (i'm guessing you already
| knew that ;P ), which the article's author eventually points
| out is a way to recover "exactly-once" semantics. The point,
| maybe, is that someone needs to explicitly do this _somewhere_
| ; you can't really rely on your protocol to do it for you.
| danpalmer wrote:
| > then track those IDs in the receiver side
|
| Now you need a database. Do you also need exactly-once delivery
| to the database? Now the service is no longer stateless too,
| which means scalability is a problem. Maybe you decide to make
| it just an in-process cache for de-duping, but that needs
| expiring and now the semantics are exactly-once within a given
| time period, and not across service restarts.
|
| We can definitely solve this with higher level constructs, but
| they're not free, and they can introduce the same issues
| themselves.
|
| > Isn't this basically what every "reliable" transport (TCP,
| HTTP3, message queues...) does?
|
| TCP does this, to solve retries at the TCP layer. HTTP3 does
| this to solve issues at the HTTP3 layer. Message queues might
| solve this for the message queue, depends. But none of these
| solve the product level, or the user experience level, or other
| higher levels where these issues still crop up. They're issues
| you have to solve at every layer in some way.
| sorokod wrote:
| _Do you also need exactly-once delivery to the database?_
|
| Yes, with uniqueness constraint.
|
| _the service is no longer stateless too, which means
| scalability is a problem_
|
| Do you have a specific problem in mind?
| danpalmer wrote:
| Yeah often the best way to tackle exactly-once delivery to
| a database is a uniqueness constraint, but that isn't free
| - there's the index cost, additional write cost, and the
| error needs to be handled when it's thrown back to the
| client on a collision (something many applications don't
| handle well).
|
| Stateful services are far harder to scale than stateless
| ones. Typically a stateless service can be scaled out
| horizontally with relative ease, but when there's state
| storage involved this becomes harder. You can scale
| vertically, but only so far. You can scale horizontally,
| but then typically need to introduce some sort of
| sharding/consistent hashing/etc to ensure that the service
| has a consistent view of the world without needing to
| connect to every database instance.
| sorokod wrote:
| Not sure where the expectation of things being free comes
| from. If your stating point is stateless then you can
| consider the tradeoff of introducing state vs. processing
| the same request multiple times.
| hinkley wrote:
| It's probably more accurate to say that toy systems can
| maintain the illusion of exactly-once for a while, but it
| doesn't scale. You can't keep a record of every message
| ever seen in a message based system. Message passing
| systems exist to handle rates of traffic that cannot be
| achieved by rolling your own event queue as a thin wrapper
| around other tools like databases. It's not just the
| storage, it's the throughput.
|
| The first time I encountered RabbitMQ it could only handle
| 60 messages per second with delivery guarantees. We already
| had our own bespoke system that used a database to handle a
| couple multiples of that. So we ended up limping along with
| what we had.
| xg15 wrote:
| Or I just use a monotonically increasing ID and track the
| highest ID I've received in order. I might have to buffer a
| certain number of packets/messages/whatever to deal with out-
| of-order arrivals, but the entire state fits into RAM. (Edit:
| Actually I don't even have to. It's probably a good thing to
| do for efficiency, but in principle I can just drop out-of-
| order messages and wait until they are redelivered, hopefully
| in the correct order)
|
| But yes, even if I needed some stuff to archieve it, that
| doesn't make it _impossible_ as the OP claimed.
|
| > _But none of these solve the product level, or the user
| experience level, or other higher levels where these issues
| still crop up._
|
| I don't understand this point. Do you have some examples?
| danpalmer wrote:
| > Or I just use a monotonically increasing ID and track the
| highest ID I've received in order.
|
| This assumes you can generate monotonically increasing
| numbers. If you have many clients, now they all need to
| share a data source and may be performance bound by
| generating those numbers.
|
| > Actually I don't even have to. It's probably a good thing
| to do for efficiency, but in principle I can just drop out-
| of-order messages and wait until they are redelivered,
| hopefully in the correct order
|
| True (modulo first problem), but efficiency may be
| necessary here. With many clients, you may end up in a
| state where only a small fraction of messages get through
| successfully, and most traffic is unsuccessful, which is
| bad. This also makes performance commitments hard to
| maintain as it's perhaps just luck when a client manages to
| get a message through. Clients also now need more
| buffering, more state, etc.
|
| >> But none of these solve the product level, or the user
| experience level, or other higher levels where these issues
| still crop up. >I don't understand this point. Do you have
| some examples?
|
| Let's assume a simple client->server instant messaging app.
| As a user, if I send a message, I expect that to arrive
| exactly once. It's going over TCP which is "reliable", but
| it doesn't stop the HTTP request from failing and needing
| to be retried. It's using HTTP3, but that doesn't stop the
| server generating a 503 and needing to retry the POST
| request (or whatever). Maybe the server puts the message in
| a message queue, but that connection fails after sending a
| transaction commit, did it get committed?
|
| Idempotency tokens or an equivalent mechanism do solve
| this, but there isn't one magic trick to solving it in some
| base layer technology like TCP, this needs to be solved
| again and again whenever you have distributed systems.
|
| Also, this isn't just networking. Two processes on a single
| machine communicating via IPC may be effectively a
| distributed system! I've got some experience doing this on
| Android, and it's still hard.
| hinkley wrote:
| If everything smells like shit, check your shoes.
|
| How do you know what the last message you received is? You
| can crash in the middle of receiving a message. You can
| crash after you've written the ID to disk but before you've
| processed it, or you can crash after you've processed it
| but before you've written the ID to storage.
|
| If you're a distribution box (which is quite, quite common
| in these message queue systems), you can get the message,
| and send it to a box that just powered off. You saw it, you
| recorded it, and now you're not sure if you forwarded it
| successfully or not.
|
| Fun thing about power outages, not every box turns off at
| exactly the same nanosecond. PSUs are full of capacitors
| and inductors. Sometimes that's just enough to float
| through a brownout, too (and a bunch of machines booting
| can also cause a brownout)
| [deleted]
| jurschreuder wrote:
| Ok so maybe the letter gets sent more than once but the message
| gets sent exactly once, because the messages are numbered and you
| only process each number once.
|
| If you get a letter with the same number you already read you
| don't even open it.
|
| In Kafka this is also handled this way, events are numbered, and
| you request "latest" from the last one you processed.
|
| In our eventstreaming it's also done like this, it may surprise
| you that Kafka is just an implementation of eventstreaming and
| not the same as.
|
| With Kafka the offset of the consumers, or until which number it
| had already processed, used to be handled by the ZooKeeper but is
| migrated to the consumers.
|
| There is on "exactly one consumer gets the message" done by the
| ZooKeeper, all consumers get all the messages from the topics
| they subscribed to. If you want exactly on exactly one consumer
| you should create different topics.
|
| So not true.
| chias wrote:
| I think there is a useful distinction here though -- you're by
| definition doing processing there on a "non-exactly-once
| delivery" system in order to get your "exactly-once" result,
| and by definition anything that requires exactly-once message
| delivery must do this process itself: it cannot abstracted away
| into a separate system.
|
| So, in essence, you can never have "I will get this exactly
| once", and at best you can only ever have "I will have a plan
| for what to do if I get this more than once".
| xg15 wrote:
| > _it cannot abstracted away into a separate system._
|
| It absolutely can. That's the entire point of TCP.
| pixl97 wrote:
| At the same time, don't trust any network middleware not to
| break your expectations on TCP behavior... also don't trust
| that middleware will be visible and you'll know what's
| going wrong.
| andrewaylett wrote:
| TCP can't do exactly once in every scenario without fail.
| It does a pretty good job, but failure in the edge cases is
| the whole point of the difficulty we're looking at.
|
| If the network loses the last packet in a TCP stream, then
| goes down for an indefinite period, the two ends of the
| connection have irreconcilable differences of understanding
| about whether the entire transmission was received. As far
| as the recipient is concerned, they have an entire message
| that's fully ack'd and so they should process it. As far as
| the sender is concerned, they have no ack for the last
| chunk of data and must redeliver it. This is the crux of
| the Byzantine Generals problem.
|
| TCP can solve problems that happen in its own domain, and
| give reliable in-order delivery once over an unreliable
| network _up to a point_. It 's not able to provide
| _exactly_ once semantics in _all_ scenarios though. Because
| that 's logically impossible.
| chacham15 wrote:
| > So, in essence, you can never have "I will get this exactly
| once", and at best you can only ever have "I will have a plan
| for what to do if I get this more than once".
|
| I dont understand this at all. TCP has exactly once message
| delivery that the application layer is completely unaware
| of...
| majormajor wrote:
| TCP isn't guaranteed delivery if the receiver crashes;
| that's what queues and similar persistent systems are
| introduced to solve, but at-least-once is what you normally
| get outside of specialized two-phase (write then commit
| later) idempotent (so writing a second time is OK if the
| first one was never committed since your producer died and
| restarted from an earlier position, say) systems, AFAIK.
| marcosdumay wrote:
| This is something I don't understand with the common idiom.
|
| You need some processing for anything on the network, either
| you accept it or not. Yet, network protocols are described by
| the behavior they export to their consumers, not by their
| internals. Well, with the single exception of exactly-once
| delivery.
| still_grokking wrote:
| Looks like pure philosophical distinction.
|
| The question here would be who "you" are. Are "you" the low-
| level system processing raw messages, or are "you" a system
| on top of that?
|
| The high-level system can rightfully claim to that it
| receives messages "exactly once"--from the low-level system.
| tylersmith wrote:
| The article is saying a message queue service can't guarantee
| exactly once delivery and your comment is saying it's possible
| for a message queue consumer to handle duplicate delivery.
| Those are different things.
| majormajor wrote:
| > Ok so maybe the letter gets sent more than once but the
| message gets sent exactly once, because the messages are
| numbered and you only process each number once.
|
| Isn't that just shifting the goal a bit? Now the trick is "only
| process each number once" which seems to have its own
| transactional issues if you can crash between "taking action
| based on the message" and "recording that this number has been
| seen"? If you need non-idempotent actions wouldn't this still
| be a potential issue?
| javaunsafe2019 wrote:
| I mean isn't it just the same as with a real world
| package/message. It just depends on how much effort you are
| willed to put into your desired delivery model?! ... But
| guaranteed is nothing, never.
| stickfigure wrote:
| My company (among other things) routes print-on-demand orders to
| various print companies. Some of their APIs have mechanisms to
| ensure idempotency, some don't. The last time I pressed the
| issue, I was asked - and I quote - "Can't you just send the order
| only once?"
|
| The thing is, having a print company that gets the printing part
| right is more important than having one that gets the API right.
| I use them anyway, and accept the risk that there will very
| occasionally be duplicate orders. At least in my business, it's
| just tshirts.
|
| A few months ago I bought a fairly expensive cordless vacuum from
| hoover.com. I was charged once, but two of them arrived. I
| suspect I know why.
| piyh wrote:
| I was completely naive to distributed systems until I was field
| promoted to owning one after tons of attrition with no
| backfilling roles.
|
| It was a system built by people that also didn't have distributed
| system experiences. It was not enjoyable at all, and at least
| once delivery was a consistent headache that required infrequent
| but time consuming remediation.
| InCityDreams wrote:
| "was field promoted"....could you explain what you mean? To
| myself, it's always been a military term. As far as 386 pc's
| etc go....?
| sieabahlpark wrote:
| [dead]
| OnlyRepliesToBS wrote:
| [dead]
| deathanatos wrote:
| > _There are essentially three types of delivery semantics: at-
| most-once, at-least-once, and exactly-once._
|
| Oh, there's a fourth kind: "none-of-the-above", i.e., neither at-
| most-once or at-least-once. The message gets delivered between
| [0, [?]] times. Maybe it gets delivered ... maybe not. Your
| message is like UDP packet.
|
| A surprising number of systems exhibit this behavior, sadly.
| skyde wrote:
| if you don't explicitly re-transmit the UDP packet. It would be
| at-most-once delivery right? The IP routers in the path will
| not re-send the UDP packet on their own!
| tstrimple wrote:
| Not with multi-casting.
| jerf wrote:
| "The IP routers in the path will not re-send the UDP packet
| on their own!"
|
| First rule of networking: Every bad thing that _can_ happen,
| _will_.
|
| If nothing else, by sheer _bugginess_ you will certainly have
| something, at some point, retransmitting UDP packets for no
| good reason.
| tshaddox wrote:
| If you're admitting into the discussion literally any
| conceivable buggy behavior from public routers you have no
| control over, then isn't it pretty clear you can't have any
| guarantees about anything whatsoever?
| aflag wrote:
| In a broadcast scenario it only takes a temporary
| instability in the routing tables to cause packets to be
| duplicated. It's less of an issue nowadays, but you don't
| control the Internet. It's not a bug because it's not
| guaranteed that a message won't be delivered twice.
| rzzzt wrote:
| I'd like to send the request 0 times and still end up
| with the expected result, thank you very much
| dylan604 wrote:
| "I'm not a mind reader" --router in rack 3
| pixl97 wrote:
| Hello and welcome to the internet. Enjoy your stay!
|
| I think that is the point of these conversations. You
| shouldn't have application expectations that cannot be
| met in the real world. Even in your own data center you
| have far less control over your mirrored switches doing
| something dumb like sending a stream of packets twice out
| of their respective interfaces.
| convolvatron wrote:
| take it up with the authors of RFC 791. duplicate packets
| are allowed. the presence of one does not indicate a bug.
| convolvatron wrote:
| its very much not likely to happen given the implementations,
| but the IP layer does not guarantee at-most-once delivery
| depereo wrote:
| Buffer bugs can and will happen.
|
| Packets can and will be queued in multiple outgoing
| interfaces.
|
| Dumb shit happens in network kit.
|
| Also, the new SDN stuff sends packets multiple times over
| different paths _on purpose_. It 's supposed to discard
| everything except the one that got there first, but...
| hinkley wrote:
| In TCP in particular there can be a lot of duplication due
| to missed packets or packets that arrive just a little too
| late.
|
| On noisy wifi you're transferring data, and the destination
| finally gets enough packets to send an ACK for the sliding
| window, only the source never gets the ACK, so it sends the
| packets it thinks you want but already have. Some of those
| get through, and the destination realizes it needs to send
| the ACK again because clearly you didn't get it the first
| time. Finally you resync and start getting new data, until
| the next cup of coffee goes into the microwave and it all
| repeats again.
|
| Since many UDP protocols end up re-implementing half of
| TCP, you're going to have some of the same failure modes.
| oefnak wrote:
| So that would be 'at least once'. Since if it is 0, it is not a
| delivery.
| [deleted]
| qeternity wrote:
| Mirror - https://archive.is/VQsIr
| anonymousDan wrote:
| This article reads like someone who has a very superficial
| understanding of the theory he/she claims proves his/her point.
| For starters, the two generals problem does not prevent one party
| knowing a message they sent previously was delivered exactly
| once. It just prevents both parties knowing about some common
| knowledge in the presence of message loss. Not that I am claiming
| to be any better!
| Nican wrote:
| I have been toying with the idea lately of using a transactional
| database (like SQL) to manage some of the very important queues.
|
| Using a transaction to retrieve an item from the queue, and
| locking the row using "SELECT FOR UPDATE" and "SKIP LOCKED". Such
| that the row gets locked on read, and several workers can read
| from the table at the same time. Within the same transaction,
| other work is done, and everything gets committed to the database
| as a single atomic operation.
|
| CockroachDB (a consensus/raft distributed database) recently
| added supported for SKIP LOCKED, but I still have yet to work on
| this idea.
| scarface74 wrote:
| Using a database as a queue. What could possibly go wrong?
|
| I guess everyone has to make this mistake once in their career.
|
| Funny enough, when I searched for "database as a queue", my own
| comment from four years ago came up as the fourth result.
|
| https://news.ycombinator.com/item?id=18774559
| kevincox wrote:
| It can definitely work. It is a pretty good way to avoid the
| distributed systems problem by avoiding having a distributed
| system.
|
| Of course there are other concerns with using a database as a
| queue (mostly at high throughput) but for most cases it will
| work well.
| rowls66 wrote:
| I have worked on a system that took exaclty this approach for
| ~17 years. The database was Oracle, at the time we started
| 'SKIP LOCKED' was not even a documented feature of the Oracle
| DBMS. It is now. The approach worked quite well for us and
| happily working today at several large banks. Also, Oracle
| sells what I think they call AMQ (Advanced Message Queing) that
| provides a messaging API but uses the DBMS for storage. No idea
| how it performs relative to dedicated persistent messaging
| solutions, but I would guess that it probably good enough for
| many workloads.
| [deleted]
| aftbit wrote:
| If you want to reason about a world that has random software and
| hardware failures, than you cannot really have any kind of pure
| results. A backhoe could cut your network cable at exactly the
| wrong point, or a malfunctioning network switch could decide to
| insert the right extra few bytes in exactly the wrong place,
| changing the meaning of your message without altering any of the
| checksums. As the scale of your application increases, the chance
| of this sort of chaos increases as well. The question then
| becomes how to reason in the face of chaos, what sorts of error
| rates are acceptable, and how to build systems that can recover
| from supposedly impossible states. If your bank's software makes
| an error, they have established processes to determine that and
| correct the balance of accounts.
| hinkley wrote:
| Last month I had a deployment go wrong on one box, and the part
| of deployments outside of my control is all or nothing. No
| partial credit for 96% success. Some random consul call
| consumed the port we listen on, a shutdown timeout expired and
| the process was killed, and so that socket was left in
| CLOSE_WAIT (like seriously, Hashicorp, SO_LINGER has been
| around for at least 30 years).
|
| This led to an existential crisis because given the number of
| ports we open and the number of machines we run and the number
| of processes per machine, there must be over a 0.1% chance of
| any deployment blowing up this way. We do hundreds a year in
| prod and probably hundreds a month in preprod. We've been
| winning the lottery this whole time.
|
| Throw enough events around and a one in a million corner case
| will happen every week, every day, twice a day, three times in
| a row. That gets old really really quickly.
| metadat wrote:
| > SO_LINGER. Lingers on close if data is present. If this
| option is enabled and there is unsent data present when
| close() is called, the calling application program is blocked
| during the close() call, until the data is transmitted or the
| connection has timed out.
|
| I had to look this one up for a refresher, but 100% violently
| agree - Such behavior certainly warrants a bug submission.
| hinkley wrote:
| The obnoxious thing about CLOSE_WAIT is that it's supposed
| to time out after 2 minutes or 10 minutes, but I gave up
| and kicked the box out of the cluster after a half hour of
| trying to ask it nicely. Which is probably what everyone
| else does.
| bob1029 wrote:
| This line of "what if an asteroid hits the primary & DR data
| centers in the same microsecond" thinking is why we settled on
| running our product on 1 VM with SQLite in-proc.
|
| After taking our customers through this same kind of
| apocalyptic rabbit hole conversation, they tend to agree with
| this architecture decision.
|
| The cost of anticipating the .00001% that might never come is
| completely drowned out by the massive, daily 99%-certain
| headache that is managing a convoluted, multi-cloud cluster.
|
| Many times the business owners will get the message and finally
| reveal that they have always had access to a completely
| ridiculous workaround involving literal paper & pen that is
| just as feasible in 2023 as it was in the 18th century.
| morelisp wrote:
| In my experience the customers, and even the POs, are the
| easy ones to convince. "We get 99% of the uptime for 30% of
| the price? Great!"
|
| It's the resume-driven mid dev in the next office you've got
| to watch out for.
| candiddevmike wrote:
| I'd tell you an exactly once delivery joke but you may not get
| it.
| danw1979 wrote:
| I'd tell you an idempotency joke but I think you might have
| heard it before...
| yamtaddle wrote:
| You can go ahead. It won't modify the state-of-having-heard-
| it.
| setuids wrote:
| I'm sure it won't make a difference
| antod wrote:
| "Now listen very carefully, I shall say this only once."
|
| I suspect that quote will meet Resistance.
| AlbertCory wrote:
| I decided a long time ago in 3+Mail that we could _occasionally_
| have messages delivered twice, or not at all, but there was no
| easy way to be sure neither ever happened. So you bias it to
| "twice."
| dang wrote:
| Discussed at the time:
|
| _You Cannot Have Exactly-Once Delivery_ -
| https://news.ycombinator.com/item?id=9266725 - March 2015 (55
| comments)
| xg15 wrote:
| So evidently the article did fail exactly-once delivery.
| danw1979 wrote:
| Fail delivery of what ?
| xg15 wrote:
| Delivery to HN.
| bjornsing wrote:
| Why not mention the architecture that comes closest to exactly-
| once delivery? If you store a Kafka offset along with your
| application state in a transactional datastore, then for all
| intents and purposes you have exactly-once delivery semantics.
| This is something I really like about Kafka's design.
| infamouscow wrote:
| I suspect it's not mentioned because in the real world there
| are ways to work around this limitation. And because those work
| arounds actually work, there's nothing interesting to say. The
| post is much ado about nothing.
| kevincox wrote:
| This is again assuming that you have no side effects. Imagine
| that you want to email users based on a list in Kafka. You read
| the offset in a transaction and update it. But do you send the
| email inside the transaction or after closing it? You are back
| to picking between at-least-once and at-most-once.
| taspeotis wrote:
| Just put the offset in the email to users and make them keep
| track of it??
| fizwhiz wrote:
| Exactly Once = At least once + Idempotence
| paxys wrote:
| So, the opposite of exactly once
| naasking wrote:
| It can be exactly once at the application level just not
| exactly once at the more fine-grained message level. The fact
| that it's not exactly once at that lower level doesn't really
| matter, the semantics at the application level is what we
| care about.
| kybernetikos wrote:
| Exactly. In practice there are probably a bunch of other
| things happening over the wire we also don't care about,
| handshakes and keepalives and negotiation and splitting and
| encryption and latency measuring and backpressure... It
| doesn't matter, in a variety of systems, at the application
| layer it is fine for the user to assume they will see a
| delivery to their code exactly once and that's what the
| user cares about. A delivery didn't mean some internal
| bytes travelled across the wire, it means your clients
| received a call.
|
| That's why if you search for exactly once delivery you'll
| see a bunch of products claiming to have it (e.g kafka).
| sokoloff wrote:
| Not exactly. If you have a business problem where you're
| thinking "But I really, really need the effect of exactly-
| once; what can I do?", GP's post has the answer.
| fsckboy wrote:
| OP's idea should be
|
| idempotence + at least once
|
| idempotence isn't necessarily commutative.
| echelon wrote:
| No, if your datastore is online (the only way you're
| functioning anyway), store an idempotency key, vector clock,
| etc. with transactional semantics.
|
| In active / active setups, there are other strategies such as
| partitioning and consensus.
| fizwhiz wrote:
| With idempotence, you shift the problem from "deliver X
| exactly once" to "make it seem like X was delivered exactly
| once". In most systems, exactly-once is really "effectively
| exactly once".
| paxys wrote:
| That's my point. You are simply converting the problem to a
| new form, not actually solving it.
|
| Hey here's a solution to the halting problem - always
| assume yes, and then figure out the edge cases. How do you
| do that? Well that's on you, I did my job.
|
| In a distributed system that needs exactly-once delivery,
| implementing perfect idempotence is equally impossible.
| burnished wrote:
| Converting a problem to a new form that you know how to
| better solve, or at least hope is more tractable, is a
| time honored mathematical and CS tradition
| nawgz wrote:
| Idempotency - famously complex. No one has ever
| successfully implemented it, great point.
| paxys wrote:
| If you don't think idempotency can be complex then you
| haven't really worked on serious distributed computing
| problems.
| nawgz wrote:
| If you don't think your analogy is a miss then you
| haven't really read any serious literature.
| stonemetal12 wrote:
| Until the bill comes anyway. Having to provision extra
| bandwidth for useless dups, extra processing power for useless
| updates, etc.
| doctor_eval wrote:
| AFAIK the point of exactly once delivery, in the context of
| message passing, is to abstract delivery concerns away from the
| application layer and into the messaging layer, so that the
| application can depend on the exactly-once semantics without
| having to write logic for it.
|
| The problem with this is similar to the problems with two-phase
| commit in distributed databases: there are unavoidable failure
| cases. Most of the time it works just fine, but if you write
| your application to depend on this impossible feature, and it
| fails - which, given enough time, will certainly happen - then
| the cleaning up the mess can be much more effort (and have much
| wider business implications) than simply dealing with the
| undesirable behaviour of reality in the first place.
|
| Or to put it another way: exactly once semantics can never be
| reliably extracted away from the application, so if you need
| it, it needs to be part of your application.
| FooBarWidget wrote:
| Theoretically true, and easy to say. But the hard part is
| actually implementing this in the context of business problems.
| What if you need to call external services that you don't
| control, and they don't provide idempotence? Like sending
| emails. Or worse: you send a message to a warehouse to deliver
| an item, and they deliver duplicates...
| lll-o-lll wrote:
| Yeah the duplicate email thing is a classic problem, but I'm
| not sure it's one of "idempotence". This can happen in any
| (intended to be) transactional operation that creates a side
| affect.
|
| Hit an error, roll-back, side-affect can't be rolled back.
| Retry - side-affect happens again.
|
| Wouldn't the general approach be to have unique message
| identifiers and queue side-affects? Maybe I'm missing lots of
| subtleties.
| tunesmith wrote:
| This is called "Effectively Once".
| crazygringo wrote:
| Which the author admits three quarters of the way through:
|
| > _The way we achieve exactly-once delivery in practice is by
| faking it. Either the messages themselves should be idempotent,
| meaning they can be applied more than once without adverse
| effects, or we remove the need for idempotency through
| deduplication._
|
| Honestly I don't get why this is "faking it" though. It seems
| like the author's definition of "exactly once" is so purist as
| to essentially be a strawman. This _is_ "exactly once" _in
| practice_.
|
| Like are there other people claiming that this purist version
| of exactly-once _does_ exist?
| majormajor wrote:
| It can be very hard to get idempotency right.
|
| It can get _way harder_ when your initial design made
| incorrect assumptions about the delivery semantics you were
| using, so you didn 't know you'd need it.
|
| Edit for example:
|
| Someone could have a low-latency problem that seems like it
| could be a fit for a streaming application. They could look
| at docs and see "ooh, with Flink I can do exactly-once writes
| to Kafka" in one place, and choose to use that. But if they
| don't dig deeply into what that means, they may miss the
| latency impacts of having to checkpoint every time to commit
| a set of writes to Kafka. And by the time they figure this
| out, managing both "low latency" and "exactly once" in the
| code they wrote might be a really hairy problem.
| hn_go_brrrrr wrote:
| The distinction is how you design. You don't need idempotence
| with a mythical "exactly once" system. Conversely, when
| you're debugging a system built on top of "at least once",
| you need to keep that property in mind in case the bug you're
| tracking down is lost idempotence.
| nimih wrote:
| > Like are there other people claiming that this purist
| version of exactly-once does exist?
|
| In my experience, the purist version of "exactly-once" exists
| as a vague, wishy-washy mental model in the brains of
| developers who have never thought hard about this stuff[0].
| Like, once you sketch out why idempotency is important and
| how to do it, folks seem to pick up on it pretty quickly, but
| not everyone has trained their intuition to where they
| automatically notice these sorts of failure modes.
|
| [0] I don't mean this as a slight against those developers--
| the issues that arise from distributed systems are both
| myriad and subtle, and if you've spent your time learning how
| to make beautiful web pages or cool video games or efficient
| embedded systems, it seems reasonable to not know anything
| about the accursed problems of hypothetical Byzantine
| Generals. Or maybe you're fresh out of a bootcamp or an
| undergraduate program and haven't yet been trained to expect
| computers to always and constantly fail in every possible
| way.
| cowl wrote:
| Because both of this "solutions" are not part of the delivery
| mechanism but part of your problem space. So the delivery
| system is not guaranteeing even a fake exactly-once delivery,
| it's you usage that makes it a fake exactly once. What's more
| both of these solutions are very hard in practice.
| Idempontency can be applied only on special circumstances
| when you can design it that way. "Prepare an order" message
| for example can't be idempotent, it has side effects and it
| will prepare a new order every time you recieve the message,
| so you go the deduplication Route by considering the OrderID
| but if you have several Workers that process these messages
| how do you handle DeDuplication? if the first worker has
| never Ack-ed the processing, do you deliver it to a new
| Worker in the queue? How does the new Worker know if someone
| else is processing the same OrderID? Central Database? you
| are only hitting the can down the road...
| kevincox wrote:
| Because idempotence can be very hard to achieve. You usually
| can't just write the message ID to a DB and ignore messages
| with a matching ID because if you crash while processing then
| you need to start over again. But you can't just write it at
| the end because then all of your processing steps need to be
| idempotent (so why are you bothering to write the ID?).
|
| I've seen very few systems that have general idempotency
| baked in. Often it ends up being specific to the application.
| In some cases you can have simple solutions like upon
| crashing reload all of the state from an authoritative
| source. In some cases your messages result in simple
| idempotent operations such as "insert message with a unique
| ID" or "mark a message with a unique ID as read" but even
| then these are becoming quite related to business logic.
|
| Basically idempotency is a powerful tool to create a solution
| but it is no silver bullet. That is why it is important to
| understand the underlying problem.
| purpleblue wrote:
| It's not "equal".
|
| If you guarantee "exactly once", you design your systems
| differently than "at least one with idempotence". A system
| designed for exactly once will be less complicated than a
| system designed for at least once + idempotence, which is why
| it is ideal but impossible.
| hackerdad wrote:
| This!
| jerf wrote:
| I think we need to keep the concepts separate because otherwise
| people get confused. You can not receive a message exactly
| once. Yes, it's not that hard, _if_ you know this is an issue,
| to build a system where receiving the same message more than
| once won 't cause a bad thing to happen. There's a few
| principled ways to do this, and some less principled ways that
| will still mostly work.
|
| But that's not because you built a system that successfully
| delivers messages exactly once... you build a system that
| successfully _processes_ messages exactly once, _even if
| delivery occurs multiple times_. The delivery still occurred
| multiple times. Even if your processing layer handled it, that
| may have other consequences worth understanding. Wrapping that
| up in a library may present a nice API for some programmer, but
| it doesn 't solve the Byzantine General problem.
|
| Whenever someone insists they can build Exactly Once with
| [mumble mumble mumble great tech here] I guarantee you there's
| a non-empty set of human readers coming away with the idea they
| can successfully create systems based on exactly-once
| _delivery_. After all, I built some code based on exactly-once
| delivery last night and it 's working fine on my home ethernet
| even after I push billions of messages through it.
|
| We're really better of pushing "There is no such thing as
| Exactly Once, and the way you deal with is [idempotence/id
| tracking/whatever]", not "Yes there is such a thing as Exactly
| Once delivery (see fine print about how I'm redefining this
| term)". The former produces more accurate models in human
| brains about what is going on and is more likely to be
| understood as a set of engineering tradeoffs. The latter seems
| to produce a lot of confusion and people not understanding that
| their "Exactly Once" solution isn't a magic total solution to
| the problem, but is in fact a particular point on the
| engineering tradeoff spectrum. In particular, the "exactly
| once" solutions can be the _wrong_ choice for certain problems,
| like multiplayer game state updates, where it may be a lot more
| viable to think 1-or-0 and some timestamping and the ability to
| miss messages entirely and recover, rather than building an
| "exactly once" system.
| naasking wrote:
| > But that's not because you built a system that successfully
| delivers messages exactly once... you build a system that
| successfully processes messages exactly once, even if
| delivery occurs multiple times.
|
| I think the difference might be partly semantic. If
| processing at the messaging level is idempotent + at least
| once, then message delivery to the _application level_ is
| exactly once. People mostly only care about the application
| level not the lower levels where they might just build on a
| library or system that handles that logic for them.
| jerf wrote:
| I'd say it's _entirely_ semantic. I 'm very much arguing
| for where to draw the definition lines in the terms we use.
| It won't change the code one bit (give or take a few
| different names on things). I definitely think
| understanding carefully the issues involved in delivery,
| and understanding the various solutions to that problem, is
| the way to go, not to blur the questions of delivery and
| handling into one atomic element. They're not atomic.
|
| Alternatively we could come up with names for all the other
| combinations of delivery mechanism and handling mechanism,
| but since you can easily see we hit an NxM-type problem on
| that, this may well help elucidate why I think it's a bad
| idea to try to combine the two into one term. It visibly
| impairs people's ability to think about this topic clearly.
| naasking wrote:
| Well, my argument for erasing that line is that you
| generally don't care about TCP packets or SSL handshakes
| and such, so why is this one property relevant if it can
| be punted to a lower layer just like those others?
|
| I'll grant that it matters if you're trying to debug some
| problem and trying to find at what layer it failed, but
| it's basically the same process you use to debug all of
| those other layers too, so I'm not sure why this layer
| deserves special consideration.
___________________________________________________________________
(page generated 2023-03-01 23:00 UTC)