[HN Gopher] Webhook Failure Scenarios
___________________________________________________________________
Webhook Failure Scenarios
Author : luu
Score : 74 points
Date : 2023-09-14 21:02 UTC (1 days ago)
(HTM) web link (hermanradtke.com)
(TXT) w3m dump (hermanradtke.com)
| JimDabell wrote:
| Some interesting previous discussions here:
|
| - Best Practices for Using Webhooks (stripe.com):
| https://news.ycombinator.com/item?id=32521159
|
| - Collection of best practices for providing and consuming
| webhooks (webhooks.fyi):
| https://news.ycombinator.com/item?id=32517645
|
| - Give me /events, not webhooks (sequin.io):
| https://news.ycombinator.com/item?id=27823109
| caseysoftware wrote:
| Very cool article.
|
| I'm one of the creators of https://webhooks.fyi/ and will do a PR
| to add this one. Thanks!
| pyrolistical wrote:
| The way I solve all these problems is.
|
| Prerequisite: require receiver to be able to process duplicate
| webhook being resent.
|
| Algorithm outline:
|
| 1. Have webhook body stored by sender
|
| 2. Attempt to send body to receiver
|
| 3. If 2x status returned mark webhook as sent. Done
|
| 4. Otherwise for any other status or timeout increment attempt
| for webhook and try again later until maximum attempts (which
| then mark webhook as failed)
|
| This covers the tricky case where success response gets dropped
| in network by having the sender time out and retry later. The
| receiver gets a duplicate
| foobiekr wrote:
| You should describe how you are backpressuring internally while
| this is going on and also how you send an indicator to the
| receiver that they encountered a gap.
| sergiotapia wrote:
| At the most basic level, what I like to do is create a webhook
| response controller, the controller enqueues a job and
| immediately returns an HTTP 200.
|
| The job is saved in my postgres database, using Oban (Elixir) so
| I have observability, accurate bug stacktraces are saved in the
| job record's error field, and I can see how many times the job
| has been attempted.
| tasn wrote:
| There are so many more failure scenarios!
|
| I wrote a post about common TLS errors recently:
| https://www.svix.com/blog/ssl-tls-incomplete-certificate-cha...
| chunk_waffle wrote:
| For a server sending webhooks to endpoints entered in by users,
| take care that:
|
| a: The FQDN does not resolve to an RFC1918 address (You don't
| want to be POSTing payloads to endpoints within your internal
| network.)
|
| b: If you respond to redirect responses (easier to just not do so
| for other reasons as well) also make sure those don't resolve to
| internal addresses too!
| foobiekr wrote:
| The attacker-controlled DNS record pointing to an internal
| private address or an explicit redirect is a classic,
| especially if they can control the event template being used
| and the service relies entirely on edge filtering... Too much
| template control is a risk.
|
| I mean, there's a lot of things you should do when dealing with
| this that most people don't pay attention to:
|
| https://datatracker.ietf.org/doc/html/rfc2606
| https://datatracker.ietf.org/doc/html/rfc3927
| https://datatracker.ietf.org/doc/html/rfc4193
| https://datatracker.ietf.org/doc/html/rfc6761
|
| ... and so on. At least in Go some of the handy checks are
| simplified by IP.Is(Private|Loopback|Multicast|InterfaceLocalMu
| lticast()|LinkLocal*etc.)
| nicolaslem wrote:
| It's been years since I've looked into this problem but to
| tackle it properly one shouldn't just resolve the domain and
| check that the IP is acceptable. The HTTP client library needs
| to be involved by providing a way to run code just before
| creating the socket, which very few do.
| leetrout wrote:
| That needs better naming in my opinion.
|
| Just sender and receiver would be enough IMO.
|
| Maybe it is just me but it is unusual to see origin used in this
| context even if it is technically correct I only ever see it used
| with CDNs and load balancing.
| benburleson wrote:
| I've always referred to those client/server.
| leetrout wrote:
| I think webhooks are even more misleading because they are
| server to server.
| janci wrote:
| And the client is actually the sender.
| hermanradtke wrote:
| I was frustrated with my naming too.
|
| > technically correct I only ever see it used with CDNs and
| load balancing
|
| That is why I used it!
|
| I responded in https://news.ycombinator.com/item?id=37525971
| that maybe I should change it. Sender/receiver is better.
| naikrovek wrote:
| right out of the gate: why not use the terms "sender" and
| "receiver" instead of "client" and "origin"?
|
| the author uses "client" to refer to the application _sending_
| the webhook, and "origin" to refer to the application
| _receiving_ the webhook. this is all backwards to me.
|
| I often say that naming things is hard, but it isn't so hard that
| this needs be the result.
| hermanradtke wrote:
| Fair call out. I responded to similar feedback here:
| https://news.ycombinator.com/item?id=37525992
| bennyp101 wrote:
| Those terms are very confusing - to me orign would be the SENDER
| ... where the message ORIGNates from ...
|
| But yes, those are common scenarios, and most services that offer
| calling a webhook have retries baked in for such things. It is
| the responsibility of the consumer to verify if they have already
| received a message or not.
|
| You should not be consuming a webhook 'live' and attempting to
| work on it the moment you receive it - that way lies dragons!
|
| I have dedicated endpoints for services to send to, and they are
| verified and consumed as quickly as possible and put into a queue
| to be processed after. That way I can at least see what was sent,
| and if we missed something.
| hermanradtke wrote:
| > to me orign would be the SENDER
|
| My history with CDNs is that origin refers to the server
| sitting behind the proxy. I do agree that
|
| > where the message ORIGNates from
|
| makes sense too.
| paulddraper wrote:
| A CDN origin refers to the server where the content
| ORIGINates from.
| nonane wrote:
| Any suggestions to alternative to webhooks? I feel a 'pull' based
| model with cursors and long polling would be simpler and more
| reliable than webhooks.
| losvedir wrote:
| Webhooks are fun to think about. A couple more issues off the top
| of my head:
|
| * Ordering. Since network requests can take variable amounts of
| time, how do you ensure those two "foo.updated" events are
| processed in order, or that the receiver can tell their order?
| Especially something to consider if the webhooks will retry a few
| times on intermittent failures.
|
| * Consistency. Always a concern in distributed microservice land,
| but maybe more acute when generating webhooks right as things are
| updated: if the receiver uses the webhook to make an API request
| right back into the system, will the API have the same view of
| the data?
|
| * DDoS. How do you make sure the webhook destination URL is owned
| by the subscriber? If your system can generate high volumes of
| webhook traffic, that could be a problem.
|
| * Infinite loops. A silly one, but the user could conceivably
| point the webhook at a URL of the system that sends the webhook,
| in such a way that it will cause a new webhookable event to be
| generated.
| sarchertech wrote:
| Ordering is something that everyone always forgets. The only
| way to actually guarantee they are sent and received in order
| is to use a lock on whatever subset of messages you care about
| being ordered. Then only process one of those at a time.
|
| At my last company I built an ordered transaction outbox to
| handle this.
|
| Each message has a partition key. When the worker picks up a
| new message to process it locks the partition key, so that it's
| the only worker processing any messages with that partition
| key.
|
| Different messages use different things for their partition
| key. For some messages where we only care about the order
| within a given order (it's an order management system) we use
| the order_id. For other messages where we care about global
| ordering we use the tenant_id etc...
|
| You could just use tenant_id for every message, but by using
| the most granular partition that you can, you can get a lot
| more parallelism.
| foobiekr wrote:
| Well for starters, you should by-design not create an
| interface where individual discrete events are required to be
| delivered, without drops or strictly in order. Once you start
| down that path, you have to be able to answer what you do
| when the the destination is blocked (for any reason: network
| congestion or outage, service outage, etc.). This is the path
| where your internal queue (whether physical or virtual - that
| is, a cursor) starts requiring potential days of retention
| and the ability to stream from any point in that history.
| Been there, done that. You can implement that retention in
| all sorts of ways (rows in a database, messages in kafka,
| etc.) each of which has interesting cost dynamics and corner
| cases.
|
| You are almost better off with a state compressed reporter
| that has the semantics of always reporting a _sensible_
| series of events to the external endpoint where _dependent_
| events have sensible relative order ("x was added, x was
| deleted") that converges on current state no matter how slow
| the outbound endpoint is and without regard to how many of
| them there are. No key locking and no buffering, O(1) storage
| per walker - just a walk on conveying state. This requires
| careful design of both the system _and_ the schema (boolean
| state needs to be accompanied with a counter, for example, to
| convey collapsed/consolidated history of the state changes in
| this model).
|
| That's a lot more work, but it is possible, this kind of
| thing dates to the 80s or earlier.
|
| The only in-order/no-drops case that is really common is logs
| (which is a poor way of conveying event/state tracking) and
| webhooks are a very poor choice for that kind of thing.
|
| Also, in all cases, receivers should have to accept that they
| may receive the same notification more than once. Obviosuly
| there are ways around this, but they are expensive at scale.
| dalore wrote:
| Could you not timestamp your message and then let the client
| do the ordering. Granted they might get messages out of
| order, but I never promised them their webhooks would be in
| order.
| skeeter2020 wrote:
| Timestamps from where? There is not necessarily going to be
| a central authority or processor who can guarantee these
| are in sync across distributed systems
| devoutsalsa wrote:
| Regular timestamps can only be used reliably for ordering
| if all the timestamps come from a single system. Different
| systems have different clocks, so they'll never be
| precisely in sync. You can also have timestamp collisions,
| which makes ordering ambiguous.
| chrsig wrote:
| This is assuming the use of a monotonic clock. the system
| wall clock may jump backwards in time. Itt also assumes
| that only one id can be generated per millisecond (or
| whatever clock resolution). In the case of multiple
| threads or processes on multiple cores, it becomes more
| likely that there will be a collision.
|
| Timestamps don't make for a great ordering key.
| paulddraper wrote:
| Yes, you can push the problem to your users.
| paulddraper wrote:
| > At my last company I built an ordered transaction outbox to
| handle this.
|
| Almost all message queue/event streaming systems support FIFO
| (SQS, RabbitMQ, ActiveMQ, Kafka, Kinesis).
|
| If you use one of these, it's easy to solve ordering.
| skeeter2020 wrote:
| This is just for arrival order though. The GP mentions
| transaction-based operations; handling this without massive
| hits to parallel processing is not an easy problem to
| solve.
| paulddraper wrote:
| ? I'm not sure I understand.
|
| You choose an FIFO key, e.g. order ID.
|
| Any message with that key is processed serially. Ordering
| between keys is unspecified.
|
| It is important to choose a key that guarantees
| sufficient ordering (for correctness), but not too much
| (for performance).
| ignoramous wrote:
| > _You choose an FIFO key, e.g. order ID._
|
| This is the part of the system that is doing most of the
| heavy lifting. Ex A, _The struggles of Elasticsearch_ :
| https://archive.is/rm1UI
| tasn wrote:
| Ordering is even more compelx than that! I previously wrote a
| post about it as it comes up a lot with Svix.
|
| https://www.svix.com/blog/guaranteeing-webhook-ordering/
| Semaphor wrote:
| I've only ever worked with two webhook systems (Eventbrite and
| PayPal) and both have those issues handled. I'd say you should
| just look how others solved it.
___________________________________________________________________
(page generated 2023-09-15 23:02 UTC)