[HN Gopher] Give me /events, not webhooks
___________________________________________________________________
Give me /events, not webhooks
Author : todsacerdoti
Score : 258 points
Date : 2021-07-13 16:41 UTC (6 hours ago)
(HTM) web link (blog.syncinc.so)
(TXT) w3m dump (blog.syncinc.so)
| closeparen wrote:
| I have long felt there is a startup opportunity for a service
| that receives webhooks and lets you subscribe via message queue.
| rakoo wrote:
| History does repeat itself. All of those issues, and the
| solution, are the reason CouchDB is modeled the way it is:
| there's a single endpoint that gives you _all_ events happening
| in the database, in chronological order, with both document ids
| and "feed" ids, reachable with long-polling. All of this more
| than a decade ago already.
| recursivedoubts wrote:
| why not both?
| freedomben wrote:
| This is just my opinion of course, but I've consumed a great
| number of APIs and I love when there is. You don't even need
| /events as long as there's a good index endpoint for the object
| in question.
|
| For a great number of applications it's not that big of a deal to
| miss a webhook, and the extreme simplicity that it gives the
| developer is worth a great deal. With how enormously complex a
| lot of systems have gotten, I really favor simplicity whenever
| possible.
| lxe wrote:
| Holding requests open for long polling is probably difficult to
| scale. How do you persist connections when application servers
| drop in/out? How do you load-balance? What about socket
| descriptor limits? etc...
| maerF0x0 wrote:
| IMO the real solution is give me a better transport that RESTful
| HTTP! As many others have pointed out things like Kafka are built
| for these kinds of usecases. So often I see people trying to
| design around the flaws in REST while ignoring that we've had
| some pretty good progress in the ensuing 20yrs.
| bkrausz wrote:
| I was responsible for Stripe's API abstractions, including
| webhooks and /events, for a number of years. Some interesting
| tidbits:
|
| Many large customers eventually had some issue with webhooks that
| required intervention. Stripe retries webhooks that fail for up
| to 3 days: I remember $large_customer coming back from a 3 day
| weekend and discovering that they had pushed bad code and failed
| to process some webhooks. We'd often get requests to retry all
| failed webhooks in a time period. The best customers would have
| infrastructure to do this themselves off of /v1/events, though
| this was unfortunately rare.
|
| The biggest challenges with webhooks:
|
| - Delivery: some customer timing out connections for 30s causing
| the queues to get backed up (Stripe was much smaller back then).
|
| - Versioning: synchronous API requests can use a version
| specified in the request, but webhooks, by virtue of rendering
| the object and showing its changed values (there was a
| `previous_attributes` hash), need to be rendered to a specific
| version. This made upgrading API versions hard for customers.
|
| There was constant discussion about building some non-webhook
| pathway for events, but they all have challenges and webhooks +
| /v1/events were both simple enough for smaller customers and
| workable for larger customers.
| ctas wrote:
| Can you share a bit about how these events are stored on
| Stripes backend e.g. Kafka, Postgres?
| bastawhiz wrote:
| It's all just kafka and mongo. The event can be stored in any
| simple k/v storage. There's no magic.
| alexbouchard wrote:
| Shameless plug but I've built https://hookdeck.com precisely to
| tackle some of these problems. It generally falls onto the
| consumer to build the necessary tools to process webhooks
| reliably. I'm trying to give everyone the opportunity to be the
| "best customers" as you are describing them. Stripe is big
| inspiration for the work.
| Redsquare wrote:
| Do you provide the ability to consume, translate then
| forward? I am after a ubiquitous endpoint i can point
| webhooks at and then translate to the schema of another
| service and send on. You could then share these 'recipes' and
| allow customers to reuse well known transforms.
| spullara wrote:
| Pretty easy for a customer to setup an SQS queue and a lambda
| for receiving them rather than rely on their infrastructure to
| do all the actual receiving. Way more reliable than coupling
| your code directly to the callback.
| jon-wood wrote:
| This is precisely what we do where I work. We have a service
| which has just one responsibility - receive webhooks, do
| _very_ basic validation that their legitimate, then ship the
| payload off to an SQS queue for processing. Doing it this way
| means that whatever's going on in the service that wants the
| data, the webhooks get delivered, and we don't have to worry
| about how 3rd party X have configured retries.
| throwaway290232 wrote:
| I always laugh when people end up with designs like this. They
| could have just used SMTP! It's designed to reliably deliver
| messages to distributed queues using a loosely-coupled
| interface while still being extensible. It scales to massive
| amounts of traffic. It's highly failure-resistant and will
| retry operations in various scenarios. And it's bi-directional.
| But it's not "cool" technology or "web-based" so developers
| won't consider it.
|
| Watch me get downvoted like crazy by all Nodejs developers.
| Even though they could accomplish exactly what they want with
| much less code and far less complex systems to maintain.
| vidarh wrote:
| I actually did use SMTP as queuing middleware for a registrar
| platform years ago.
|
| It worked very well.
|
| EDIT: To add some context, my team had come off building a
| webmail platform, and so we'd done lots of interesting stuff
| to qmail and knew it inside out. We then launched the .name
| tld and built a model registrar platform that on registration
| would bring up web and mail forwarding for users that wanted
| it. We used SMTP to handle the provisioning of those while
| keeping the registration part decoupled from the servers
| handling the forwarding. We also used it to live-update a
| custom DNS server I wrote.
| rendall wrote:
| The suggestion to use SMTP is interesting.
|
| I didn't downvote you but I bet they come from this part.
| People don't like this kind of negativity.
|
| > _But it 's not "cool" technology or "web-based" so
| developers won't consider it. Watch me get downvoted like
| crazy by all Nodejs developers. _
| [deleted]
| forgotmypw17 wrote:
| I agree that people don't react well to negativity, but
| sometimes you have to say it. Node has a lot of very stupid
| (i.e. ignoring reality) decisions, and by extension, being
| exposed to this for a long enough period of time, tends to
| affect the developer as well.
|
| I say this from experience, as someone who's used a few
| stupid technologies over time.
| Aeolun wrote:
| Does Node have that? Or do node libraries have that?
|
| I find node to be surprisingly well rounded.
| daniellarusso wrote:
| The very first startup I worked at used this for a
| sweepstakes leadgen form to send to MySQL via a Perl script
| running from cron.
| [deleted]
| wruza wrote:
| SMTP would raise too many questions, from how both
| datacenters tolerate it (spam), to who will manage the
| receiving server itself and certificates on your side, and
| overall security of this setup. For a nodejs developer it's
| really easier to spin up a separate handmade queue process
| rather than managing SMTP-related things. Webhook (for
| runtime) and long-polled /events?since= (for startup) have
| all upsides with little downsides.
| aidenn0 wrote:
| SMTP no longer reliably delivers messages. Try setting up an
| MTA on a Hetzner VPS and see how many messages get through
| vidarh wrote:
| That is only relevant if you require delivery to arbitrary
| endpoints rather than to endpoints explicitly set up to
| process your messages.
| mrtesthah wrote:
| That's not an applicable criticism for SMTP running on a
| private network and/or dedicated set of "mail" submitting
| servers, as in the specific model outlined in the
| grandparent comment.
| alexgartrell wrote:
| I think the Stripe API stuff you did was fine, but you really
| did your best work as a concepts of mathematics TA.
| AtNightWeCode wrote:
| This is a fake post? Never encountered this level of
| incompetence in real life.
| avalanche123 wrote:
| I think this is also getting at the difference between pub/sub
| and state synchronization. While one might think they want the
| former, what they really want is the latter. Get some state and
| receive updates continuously rather than deal with unreliable
| stream of updates
| shadowgovt wrote:
| On HTTP, pub/sub is eventually guaranteed to drop some messages
| because TCP/IP itself is not a guaranteed networking protocol
| (it just makes some promises about failures being uncommon and
| probably detectable).
|
| If what you want is guaranteed state synchronization, pub/sub
| alone can't give it to you.
| rad_gruchalski wrote:
| > If the follower goes down, when it comes back it can page
| through the history at its leisure. There is no queue, nor
| workers on each end trying to pass events along as a bucket
| brigade.
|
| Sure, there is no queue. There is an append only log. Kafka is
| not a queue but an append only log.
|
| I do not like the proposed solution. I do not like it because it
| assumes that I have to maintain the infrastructure to do all the
| distributed logging on my end. As in, most likely I have to
| maintain a Kafka cluster.
|
| There's also one thing glossed over in this article. What if your
| consumer went past certain messages but it mishandled them? You
| can go to the past either.
|
| If your consumer needs ordered delivery, web hooks might not be
| the best solution, indeed. But it might cost you more because I
| need additional infra to provide you with that.
| toomim wrote:
| There's a much better approach than /events or webhooks: add
| synchronization directly into HTTP itself.
|
| The underlying problem is that HTTP is a state _transfer_
| protocol, not a state _synchronization_ protocol. HTTP knows how
| to transfer state between client and server once, but doesn 't
| know how to update the client when the state changes.
|
| When you add a /events resource, or a webhooks system, you're
| trying to bolt state synchronization _onto_ a state transfer
| protocol, and you get a network-layer mismatch. You end up with
| the equivalent of HTTP request /response objects inside of
| existing HTTP request/responses, like you see in /events! You end
| up sending "DELETE" messages within a GET to an /events resource.
| This breaks REST.
|
| A much better approach is to just fix HTTP, and teach it how to
| synchronize! We're doing that in the Braid project
| (https://braid.org) and I encourage anyone in this space to
| consider this approach. It ends up being much simpler to
| implement, more general, and more powerful.
|
| Here's a talk that explains the relationship between
| synchronization and HTTP in more detail:
| https://youtu.be/L3eYmVKTmWM?t=235
| mywittyname wrote:
| > To mitigate both of these issues, many developers end up
| buffering webhooks onto a message bus system like Kafka, which
| feels like a cumbersome compromise.
|
| Kafka solves exactly the issue that the author is complaining
| about. This is a safeguard to ensure that data isn't dropped in
| the event of an issue, and provides mechanisms to replay events.
|
| The tradeoff between pushing and polling have been argued since
| forever.
|
| In other news, mechanics who work with bolts often do so with
| ratchets. This is a cumbersome compromise, just give me Torx
| fasteners!
| aunty_helen wrote:
| It's a common writing style as of late, set down a premise and
| solve that premise decisively.
|
| Now, if that premise isn't based in reality, or if it's already
| been solved some other way, discredit it without giving it too
| much air time.
|
| A one liner about kafka being cumbersome and then building your
| own solution, warts and all, doesn't need to exist in the same
| thought if you've made the reader mentally disregard it as a
| possible solution.
| closeparen wrote:
| Are you going to expose your Kafka brokers directly to your
| integration partners? Are they going to use the Kafka client
| library and wire protocol to send you data? That's the thing
| about webhooks, HTTP is universal and if you're comfortable
| exposing _anything_ externally, it's going to be a web service.
| mrkurt wrote:
| We expose the NATs protocol to our users. Exposing non-http
| protocols is fun, sometimes.
| mywittyname wrote:
| I would not expose kafka directly. I would implement this as:
|
| HTTP Endpoint -> Push to message queue (kafka, SQS, etc) ->
| Acknowledge receipt
|
| That's a pretty straight-forward design that's widely used,
| robust, and easy to put together. I've probably done that
| same workflow 100s of times without issue.
|
| As long as you guarantee the message was pushed to the queue
| before acknowledging, that will be fabulously reliable. You
| need to make contingencies for duplicate messages, but that's
| not usually difficult.
| jerf wrote:
| It would if the source was pushing into the Kafka stream
| directly. It doesn't solve the problem of going out of sync if
| my code to push to the Kafka stream is entirely down and I miss
| POSTs.
|
| (And, of course, I don't want Kafka. I want Google PubSub. No,
| wait, I mean SQS. No, wait, I mean I want zeroMQ. No, I
| mean....)
| toomuchtodo wrote:
| You meant Apache Pulsar! :)
| jerf wrote:
| I'm feeling cosmopolitan today. I mean all the things!
| [deleted]
| mywittyname wrote:
| As long as you guarantee delivery to your message queue
| before acknowledging receipt, you should be golden.
|
| Also, swapping out one messaging system for another is
| trivial. Pick the one best suited to the environment you're
| working in, and if that environment changes, changing
| messaging queues is going to be one the easiest transitions
| you'll make.
| l_t wrote:
| Not disagreeing with your point, and I'm sure you already
| know this, I just wanted to point out (for the benefit of
| people that don't have other options) that it is possible to
| build "webhooks" in such a way that you're confident nothing
| is dropped and nothing goes (permanently) out of sync. (At
| least, AFAIK -- correct me if this sounds wrong!)
|
| Conceptually, the important thing is each stage waits to
| "ACK" the message until it's durably persisted. And when the
| message is sent to the next stage, the previous stage _waits
| for an ACK_ before assuming the handoff was successful.
|
| In the case that your application code is down, the other
| party should detect that ("Oh, my webhook request returned a
| 502") and handle it appropriately -- e.g. by pausing their
| webhook queue and retrying the message until it succeeds, or
| putting it on a dead-letter queue, etc. Your app will be "out
| of sync" until it comes back online and the retries succeed,
| but it will eventually end up "in sync."
|
| Of course, the issue with this approach is most webhook
| providers... don't do that (IME). It seems like webhooks are
| often viewed as a "best-effort" thing, where they send the
| HTTP request and if it doesn't work, then whatever. I'd be
| inclined to agree that kind of "throw it over the fence"
| webhook is not great and risks permanent desync. But there
| are situations where an async messaging flow is the right
| decision and believe it or not, it can work! :)
| ThrowawayR2 wrote:
| > " _Of course, the issue with this approach is most
| webhook providers... don 't do that _"
|
| Embedded systems don't do that for webhooks because they
| can't (very little RAM or non-volatile storage) but
| customers clamor for webhooks anyway because it's what
| their web developers know how to use. So inevitably they're
| going to lose data but they're only getting what they asked
| for.
| BasieP wrote:
| We have a system that pushes loads of messages (as in
| thousands a minute) and some consumer insists on using
| there http backend to push the messages to. There system is
| down every once in a while for quite some time. We're using
| an async queueing solution, but you can't keep those
| messages forever. We sometimes have milions of messages for
| them in there queue's, which take up space... If all of our
| consumers had those problems we would have to buy loads of
| storage.. We're simply dropping messages older than x, and
| have an endpoint that they can call to retreive the 'latest
| state of things'. This way when they come back from a
| failure, they simply get the latest state, and then
| continue with updates from our end.. It's far from perfect,
| but it works really well.
|
| I know the goal for most systems is just to be 'up to date'
| Not to get the entire history. So in most cases you don't
| need to stash all the messages, you just need to be able to
| retreive the latest state of stuff...
| atombender wrote:
| This misses the problem explained in the article, which is
| that there are scenarios where events are "acked" but
| things still go wrong because of bugs.
|
| For example, you rolled out code on the receiver side that
| did the wrong thing with each message. Now there's no way
| to replay the old webhooks events in order to reinstate the
| right behaviour; there's no way to ask the producer to send
| them again.
|
| The only way around this is to store a record of every
| received message on the receiver side, too, which the
| article author thinks is an unnecessary burden compared to
| polling.
|
| Personally, I think push is an antipattern in situations
| where data needs to be kept in sync. The state about where
| the consumer is in the stream should be kept at the
| consumer side precisely so it can go back and forth.
| nine_k wrote:
| The question is: who maintains the queue of events, and pays
| for it?
|
| Certainly the event producer is in a better position to
| maintain a queue without missing events, but it also means
| they need to buffer more data in their queue system to
| accommodate for your receiver's downtime
| BasieP wrote:
| this!
| danudey wrote:
| Having helped manage a Kafka cluster, I do not want to run a
| Kafka cluster just so that Microsoft Teams can webhook me
| events now and then.
| Floegipoky wrote:
| Yeah I was scratching my head reading this article; they're
| bending so far backwards to avoid the obvious solution that I
| thought they were gearing up to pitch some competing tech.
|
| > If the sender's queue starts to experience back-pressure,
| webhook events will be delayed, and it may be very difficult
| for you to know that this slippage is occurring
|
| I've never before seen anyone try to argue that properly
| dealing with backpressure is a bad thing. The author's proposed
| model makes this situation even worse. With kafka, consumers
| can continue processing the event stream and you can continue
| to serve reads from your primary datastore. With the author's
| model the event stream lives in your primary datastore, so if
| that starts to lock up the blast radius is much larger.
| alexbouchard wrote:
| Totally, things can get very reliable if you start processing
| webhooks asynchronously. Personally I've found it pretty
| cumbersome and complicated to build the necessary
| infrastructure in the past. I've been building
| https://hookdeck.com as a simpler alternative specifically to
| ingesting incoming webhooks.
| klysm wrote:
| This seems like another instance of the never ending war of push
| vs pull
| fiddlerwoaroof wrote:
| It seems to me that you could build a protocol where normal syncs
| happen through webhooks where each webhook event refers to the
| event id of an immediately previous event. If the system
| receiving notifications doesn't have that event, it makes an API
| request for all events between the latest event it has and the
| one it just received.
| aboodman wrote:
| It the system is moving fast this is still somewhat complicated
| to implement robustly because by the time the "catchup" request
| to t0 returns, more time has passed and more events have
| happened, so you still can't resume consuming the webhooks.
|
| To be correct with such a system you have to be prepared to
| queue the incoming webhook events, do the catchup query, then
| replay the queued events.
| MrStonedOne wrote:
| only if the order of events matters.
| fiddlerwoaroof wrote:
| Yeah, if the events can be handled idempotently and this
| handling properly accounts for time, you don't have to stop
| processing webhooks while filling in the gaps.
| aboodman wrote:
| OK but, it usually matters. If the events represent
| changes to some object, those changes almost always have
| to be handled in order if you are to arrive at the same
| end state as the source.
|
| _edit:_ Also, idempotent is not the right term here.
| Idempotent just would mean the event could be handled by
| the receiver multiple times w /o changing the meaning. If
| you need the events to be applicable out of order, then
| you need them to be commutative. This is a much more
| difficult property to ensure and in practice, I am
| guessing, almost non-existent in deployed webhook APIs.
| williamdclt wrote:
| That is pretty much what TCP does, except that it doesn't
| "request" missing packets: the client just acknowledges the ID
| of the last packet it was happy with.
|
| It requires a sequence though, so that you know that the client
| can know that packet it just received isn't the one it
| expected, but you could build that with a chain of IDs like
| you're proposing.
| tlarkworthy wrote:
| long polling is not good for serverless consumers. webhooks are
| great for compute on demand so we should work towards that (it's
| cheaper coz it's more effecient).
|
| You do not need storage in the producer AND in the consumer, you
| just need a queue in the producer. Yes, even that is annoying,
| but the suggested architecture will still lose data if the long
| poller is down unless there is storage in the producer... so
| nothing significant has really been solved
| kureikain wrote:
| I think with a right a strategy for retry then webhooks are
| great. On my email forwarding app https://hanami.run we offer
| both method to give user access to their email:
|
| 1. webhook with retry up to 7 days. 2. a REST api to fetch all
| data
|
| Sending webhook out properly require lots of effort, especially
| idempotent key concept to avoid duplicate data. And control
| concurency to avoid swarming the webhook endpoint.
|
| So at the end of day, both are require same amount of resources,
| either on the sender side or the receiver side.
| throwaway290232 wrote:
| The premise of this post is wacky. They're trying to argue for
| how web applications should provide consistency to operations on
| remote systems. You know what that's called? Distributed
| Computing. I don't know if you know this, but Distributed
| Computing Is Hard. You can't solve it with a new interface or
| polling really fast.
|
| Webhooks are perfectly fine for what they're intended, which is
| inconsistent push-based notifications to loosely coupled web
| apps. If you require "consistency", you supplement with polling
| and queues and other junk. If you require _real_ consistency, you
| must use a distributed consensus algorithm.
| MrStonedOne wrote:
| distributed consensus/distributed computing appies for
| mutual/full-duplex/two-way systems, syncing changes one way
| from a 3rd party is not distributed computing, and not
| something you would want to throw a distributed consensus
| algorithm at
| himoacs wrote:
| Ah, I hear that a lot from customers I work with. A true event-
| driven system requires both events and webhooks. You will always
| have apps that only interact via REST so you can't really use
| streaming architecture here but you can make them more real-time
| via webhooks.
|
| The article talks about issues with webhooks such as not being
| reliable if the service goes down and messages are lost. It also
| talks about developers daisy-chaining multiple services together
| to put forward a solution which is not robust.
|
| That's why you need a broker that does event distribution,
| supports multi-protocols (REST, AMQP, MQTT, WebSockets...)
| natively without any proxies and supports Webhooks. You can push
| messages to your REST clients and if they disconnect, the
| messages will pile up in a queue, ready to be consumed when the
| client reconnects.
|
| Solace PubSub+ Broker does all of this. Disclaimer: I work at
| Solace.
| graton wrote:
| Makes me think of how the OpenStack project watches for events to
| Gerrit. They open an SSH connection to the Gerrit server with the
| stream-event command. It then stays connected to the Gerrit
| server and all the events that occur show up in the stream and
| the program can take actions based on the events it sees.
|
| Gerrit stream-events doesn't solve the issue if the connection is
| dropped and events occur while disconnected.
|
| I personally (for my hobbyist use case) would prefer the
| article's /event system over webhooks as webhooks require you
| have a system that is available on the Internet to receive the
| webhook. Where having this /event system would not require that.
| danudey wrote:
| My company has recently switched to Microsoft Teams, where
| unsupported integrations happen via webhooks. For example, if we
| wanted to be able to trigger builds in Jenkins or Gitlab, or
| acknowledge alerts via AlertManager, we'd have to set them up as
| webhooks to the appropriate service.
|
| The problem is that all of those services are internal to our
| network, and aren't accessible from the outside world. We cannot
| set up a webhook to Jenkins because Jenkins does not have a
| publicly accessible URL. We cannot set up a webhook to Gitlab, or
| to Prometheus, or to Sentry, or anything else, because those are
| all internal services.
|
| The only option there would be to create a new, public-facing
| server, set it up with a domain name and SSL certificate, expose
| it to the world, and then give it access to those services -
| which defeats the point of having those services internal and
| secure if we just create a non-internal system and give it access
| to them.
|
| Alternately, we have that new, public-facing server buffer those
| requests and have other services poll them, somehow, so that it
| cannot connect in, but now we're getting into the same situation
| as described in the article.
|
| If there were an API, I could easily create a small daemon that
| would watch for events and dispatch them accordingly, and then
| respond to them as needed; instead, my only option is to build
| some kind of Frankenstein - or to give up entirely, which is the
| more reasonable solution.
|
| Then again, this is Microsoft Teams, where creating an
| application requires an Azure account and jumping through a ton
| of hoops, so they're no stranger to stupid ideas that no one
| wants to deal with.
| oscargrouch wrote:
| I'm finishing a browser based application platform where the
| applications installed expose a RPC api, so in the end all
| applications can call others in the same local(or remote)
| node/s.
|
| The beauty of this is that you also can compose with other
| nodes and for a distributed service by calling the local
| service as a proxy and routing the requests to the other nodes
| of the same api.
|
| It took more time than i've predicted because its also expected
| to deliver UI and most of the 'HTML5' api to native
| applications (instead of Javascript), which is a massive
| platform by now (and the #1 reason why newcomers to browser
| technology cant compete, giving the feature creep tax imposed
| to them).
|
| The idea is also to distribute over a DHT so you can just serve
| your application over torrent without needing to register
| anything..
|
| The only way to get there is by empowering users and developers
| and taking some of the control from the cloud platform giants.
|
| In my point of view the only way to break the browser monopoly
| now is to create a new path forward, a branch.. its not the
| time to follow the rules, its time to break them or else the
| future doesn't look so bright in my opinion..
| bastawhiz wrote:
| > The only option there would be to create a new, public-facing
| server
|
| This is a problem with receiving any inbound data from a third
| party. At least with HTTP, it's pretty trivial to set up a
| robust reverse proxy with nginx.
| orf wrote:
| A small Lambda (or your cloud equivalent) is perfect for this
| graton wrote:
| You have the same sort of issue that I do.
|
| You might look into Cloudflare Tunnel (formerly Argo). It is
| free and allows you to poke a hole in your firewall to a
| specific service. If that meets your security requirements.
|
| https://www.cloudflare.com/products/tunnel/
| jffry wrote:
| I don't believe Cloudflare Tunnel is free, the free tier
| pricing page [1] lists Argo Smart Routing at "Starting at $5
| per month" ("Argo includes: Smart Routing, Tunnel, and Tiered
| Caching")
|
| [1] https://www.cloudflare.com/plans/
| fidesomnes wrote:
| A similar idea from a hn member https://tailscale.com/
| iamtheworstdev wrote:
| check out https://smee.io/
| historyloop wrote:
| I tried skimming through the article to attempt and derive the
| missing elevator pitch, but I saw them reimplement precisely the
| system they maligned at the start (push notifications with
| polling).
|
| Did anyone here understand what their value proposition even is?
| aboodman wrote:
| Did you check the website? The value prop is incredibly clear:
| they give you a Postgres database you can query with SQL rather
| than having to either (a) learn and code to a custom API or (b)
| setup your own sync system to get that Postgres db.
| historyloop wrote:
| I'm talking about the elevator pitch of their blog post, not
| of their company. Which are actually two distinct things
| (they're not describing their service in this blog post).
| leowoo91 wrote:
| It just depends which side you would like to put the
| responsibility on. Webhooks is mostly for owning it at source.
| sergiomattei wrote:
| Building polling infrastructure is substantially more complex
| than setting up a GET endpoint to handle webhooks.
|
| If you need strict consistency guarantees, sure. Otherwise, don't
| piss off your API consumers, webhooks work just fine.
| toomuchtodo wrote:
| > If you need strict consistency guarantees, sure. Otherwise,
| don't piss off your API consumers, webhooks work just fine.
|
| Ahh, but all problems are queues. What happens if your webhook
| destination is down and your source system sending queue backs
| up? What happens if your destination endpoint is up, providing
| 200s to requests, but throwing away the data quietly due to a
| mistake during a rollout? Or your webhook source quickly ramps
| to a volume that is effectively a DoS attack? (These are all
| problems I encountered in a role at a low code/no code
| product).
|
| I strongly endorse others in this thread who indicate long
| polling events is a suitable pattern. You want the best worlds
| of data durability and consistency through polling
| functionality, but also as close to real time event firing as
| possible.
|
| Of course, some APIs you have no control over, and are stuck
| building robust, chatty polling infra to support because your
| (paying) users demand access to those APIs. Such is the schlep.
| superbaconman wrote:
| Why do I need two different ways of retrieving events?
| Wouldn't the data also be available via the usual rest api?
| If events fail on the consumer end it's their responsibility
| to resync. Presumably code was written for an initial sync
| anyway no?
| crooked-v wrote:
| > Building polling infrastructure is substantially more complex
|
| That really depends on the setup you're using. There are plenty
| of server platforms out there where setting up a cron is no
| more complicated than making a GET route handler.
| sergiomattei wrote:
| I'm looking at it from the perspective of a web dev. If
| you're already building a web application, it is _almost
| always_ more work to do the polling setup.
|
| In Python, it's either a cron script or Celery beat. In PHP,
| usually a cron script.
|
| That all means more processes running outside of your web
| framework, more stuff to deal with, more complexity. Now you
| have to manage e.g. running a celery process, managing
| crontabs wherever you deploy...
|
| Even in languages like Elixir where long-running scheduled
| processes are cheap and native, you still have to write a
| GenServer in comparison to just using your web framework.
|
| I'm fine with the approach in the article. I actually like
| that they're giving the option for multiple ways of getting
| those events. Just please, please don't make /events the only
| option...
| justsomeuser wrote:
| I actually have the opposite opinion. I'm an experienced
| web developer. Setting up polling is easier for me than
| setting up an HTTP endpoint.
|
| I typically use a language with an event loop
| (async/await), so there is always something like
| `setInterval(poll, 500)`. All that code needs is a
| connection to the internet. If the server is down for 24
| hours I can start it up and it'll read the missed events. I
| can set up 10 dev environments with just an API key
| difference in the config. I can batch apply events in a
| single database transaction, ensuring consistency.
|
| But with a webhook, I need to ensure that my server can
| accept incoming connections, the external API knows that
| location. Each dev env needs to replicate this public HTTP
| server set up. I need to monitor the uptime of the server
| closely as missing events or erroring on a subset of events
| could leave my database in an inconsistent state.
| masklinn wrote:
| > There are plenty of server platforms out there where
| setting up a cron is no more complicated than making a GET
| route handler.
|
| Or way simpler if you're not building a web application in
| the first place.
| shadowgovt wrote:
| It's less about cost of building and more about cost of
| running.
|
| Polling is chatty when updates are infrequent.
| justsomeuser wrote:
| The Stripe `/events` endpoint is kind of like a Stripe-internal
| webhook with a cache of 30 days.
|
| One issue is that webhooks and HTTP clients can be pinned to a
| version, but the events listed at `/events` are whatever the
| Stripe account default version was at the time of the event
| creation.
|
| So all of your code clients polling `/events` needs to ensure it
| can handle many different versions.
|
| Another issue is that child lists are limited to 10 items, which
| means that you need to do a direct download to get list items >
| 10. This means the event list is lossy as items > 10 are never
| contained in the event stream.
|
| Stripe feature request:
|
| - An option to include all list items.
|
| - `/events` that can be version-pinned / contains events with the
| same API version.
| bno1 wrote:
| A pattern that I like to use in toy projects is to have an events
| endpoint and just send the newest event cursor through a
| notification channel (e.g. webhooks) when a new event occurs. You
| get new event quickly, you don't have to keep connections open
| for long-polling, and you can keep polling with a low period in
| case the event channel stops working for some reason.
| mbrevda1 wrote:
| Are events and webhooks mutually exclusive? How about a
| combination of both: events for consuming at leisure, webhooks
| for notification of new events. This allows instant notification
| of new events but allows for the benefits outlined in the
| article.
| alexbouchard wrote:
| This is the way to go and I'd love to see more API's with
| robust events endpoint for polling & reconciliation. Deletes
| are especially hard to reconcile with many APIs since they
| aren't queryable and you need to instance check if every ID
| still exist. Shopify I'm looking at you.
| coldacid wrote:
| I think that's what the author was getting at, after reading
| through the whole article. The idea isn't to get rid of
| webhooks, but provide an endpoint that can be used when
| webhooks won't necessarily work.
| saurik wrote:
| Yeah... I'd go so far as to argue that this is the only
| architecture that should even ever be considered, as only
| having one half of the solution is clearly wrong.
| snarkypixel wrote:
| Very similar to how I built my previous application.
|
| 1) /events for the source of truth (I.e. cursor-based logs) 2)
| websockets for "nice to have" real-time updates as a way to
| hint the clients to refetch what's new
| sb8244 wrote:
| What about supporting fast lookup of the event endpoint, so it
| can be queried more frequently?
|
| I think that a combo of webhooks / events is nice, but "what
| scope do we cut?" is an important question. Unfortunately, it
| feels like the events part is cut, when I'd argue that events
| is significantly more important.
|
| Webhooks are flashier from a PM perspective because they are
| perceived as more real-time, but polling is just as good in
| practice.
|
| Polling is also completely in your control, you will get an
| event within X seconds of it going live. That isn't true for
| webhooks, where a vendor may have delays on their outbound
| pipeline.
| jacobr1 wrote:
| The article advocate for long-polling
| sb8244 wrote:
| Yea, you're right. I am reading the advocacy as "if you
| need real-time, then support long-polling."
|
| I see the value in this, but I actually disagree with the
| article in terms of that being the best solution. Long-
| polling is significantly different than polling with a
| cursor offset and returning data, so you wouldn't shoe-horn
| that into an existing endpoint.
| shvedsky wrote:
| Yes to the combination of both. I worked on architecture and
| was responsible for large-scale systems at Google. Reliable
| giant-scale systems do both event subscription and polling,
| often at the same time, with idempotency guarantees.
| j_san wrote:
| Sorry if I'm daft, could you/someone explain why one would
| want to use both at the same time for the same system?
|
| One thing that makes sense: if you go down use polling so you
| can work at your own pace. But this isn't really at the same
| time. When/why does it make sense to do both simultaneously?
| shvedsky wrote:
| There is an inherent speed / reliability tradeoff that is
| extremely difficult to solve inside one message bus. When
| you get to truly large systems with a lot of nines of
| reliability, it starts to make sense to use two systems:
|
| 1. Fast system that delivers messages very quickly but is
| not always partition-tolerant or available 2. Slower,
| partition tolerant system with high availability but also
| higher latency (i.e. a database)
|
| The author goes through this in the very first section.
| Webhook events will eventually start getting lost often
| enough for the developer to think about a backup mechanism.
|
| Long-polling works if you have a lot of memory on your
| database frontend. Most shared databases want none of your
| long-running requests to occupy their memory which is
| better used for caches.
|
| Even if your message bus has the ability to store and re-
| deliver events, you might want to limit this ability (by
| assigning a low TTL). Consider that the consumer
| microservice enters and recovers from an outage. In the
| meantime, the producer's events will accummulate in the
| message service. At the same time, the consumer often
| doesn't need to consume each individual event but rather
| some "end state" of some entity or a document. If all lost
| events were to get re-delivered, the consumers wouldn't be
| able to handle them, and would enter an outage again. This
| is where deliberately decreasing the reliability of the
| message bus and rely on polling would automatically recover
| the service.
|
| There are other reasons, of course. The author is
| absolutely correct in their statement, though: whenever a
| system is implemented using hooks / messages, its
| developers _always_ end up supplementing it with polling.
| kissgyorgy wrote:
| What's the point of implementing webhooks once you implemented
| long polling for the /events endpoint?
| luuio wrote:
| I don't think the original comment meant long polling (i.e.
| keeping the connection alive), they meant periodically call
| the endpoint to check for events.
| cwyers wrote:
| The article advocates for long polling of endpoints.
| mbrevda1 wrote:
| I'd argue against long/persistent polling. Webhooks allows
| for zero resource usage until a message needs to be
| delivered.
| polote wrote:
| The article is interesting but is misleading. What the article
| really says, is that if in average you receive a webhook every
| second or faster, it is better to poll every second an /events
| endpoint.
|
| Well the best advantage of Webhooks over polling is that you
| receive events straightaway no matter the volume of events. If
| you already know the volume and the volume is high, of course
| polling is going to be better for everyone.
| pmlnr wrote:
| Basically: RSS for anything.
| atian wrote:
| > One idea for Stripe and other API platforms: support long-
| polling!
|
| It's great that we've went full circle. But make no mistake that
| this only means one thing: that servers are cheaper than ever. We
| can now afford to entertain previously extravagant ideas.
| Anunayj wrote:
| I've have had a couple issues implementing Long Polling in the
| past. At times I've had the firewall or reverse proxy drop
| client connections if it detects no data transfer. Which meant
| I had to timeout all the requests every 30 secons or so. Make a
| new request before the other one ends, and it all becomes
| messy.
|
| At this point, its honestly just easier to just have a
| websocket + events endpoint with a cusor both.
| [deleted]
| simonw wrote:
| Long polling is a lot easier to support now than it was a few
| years ago, thanks to the wide availability of async server
| frameworks - Node.js, Python ASGI etc - which make supporting
| thousands of simultaneous long-polling connections with a
| single server much less expensive.
| maerF0x0 wrote:
| also the OP seems to ignore all the issues with webhooks that
| longpolls suffers or is worse at. eg: you'll lose all those tcp
| connections if your service is down -- one of the main
| complaints about webhooks...
| coldacid wrote:
| Which, if the parts discussed _before_ long-polling was
| mentioned were implemented, wouldn 't be an issue since you'd
| be starting from the first event following your last-recieved
| cursor anyway.
| historyloop wrote:
| > It's great that we've went full circle. But make no mistake
| that this only means one thing: that servers are cheaper than
| ever. We can now afford to entertain previously extravagant
| ideas.
|
| We've not come full circle. This is just one blog saying "how
| about long-polling". While also ignoring that since then we've
| gained web sockets, HTTP/2 and HTTP/3, each of which make long-
| polling pointless in three different ways.
| MrStonedOne wrote:
| >While also ignoring that since then we've gained web
| sockets, HTTP/2 and HTTP/3, each of which make long-polling
| pointless in three different ways.
|
| None of those have strong support across language frameworks
| for using them as a _client_ from a server context.
| Especially http /3.
| historyloop wrote:
| What language exactly you use that can't do web sockets,
| which is a thin layer over plain TCP sockets?
| BrianOnHN wrote:
| It's an incentives issue. Webhooks or for when the service
| provider is aiming for the bare-minimum functionality with the
| least overhead. If they have the incentive to maximize your
| consumption of the data, then they should offer both, and may
| even overnight a physical copy.
| keeganj wrote:
| For infrequent events, why not both? A webhook to push the event
| and a full list at /events?
| paxys wrote:
| There are lots of reasons to want to immediately respond to an
| external event besides building an eventually consistent data
| syncing system. Polling an API endpoint works fine for the latter
| case, but not much else.
|
| A good platform should offer both of these and more (for example
| Slack does webhooks, REST endpoint, websocket-based streaming and
| bulk exports), and let the client pick what they want based on
| their use case.
| benlivengood wrote:
| Long-polling is the way to immediately retrieve events. It's
| more efficient and lower latency than waiting for a sender to
| initiate a TCP and TLS handshake.
| sk5t wrote:
| If the webhook events are coming at some sort of a brisk
| pace, the sender well may be able to reuse an already-open
| connection. And if they're rather infrequent, is the
| efficiency or latency likely to be a significant concern?
| IshKebab wrote:
| If you're using HTTP use websockets or server-sent events,
| not long polling. Long polling is obsolete.
| gremlinsinc wrote:
| Websockets can cause issues especially if you're not
| closing sockets properly, or have too much activity on a
| small server etc... Livewire for instance accounts for this
| by just polling every 2 seconds for changes, this is much
| more performant than keeping 10000 sockets open if people
| leave open the page/app but don't actually do anything...
|
| Straight long-polling should be avoided, but intermittent
| polling is a good solution for performance when you don't
| want to use all your socket bandwidth.
| azinman2 wrote:
| My understanding is that long polling is the thing that
| will reliably work at scale. Perhaps this changed in the
| past few years, but I've asked various companies like
| PubNub why they only use long polling and the answer was
| that there are too many incompatibilities out there in the
| wild for anything but that.
| [deleted]
| hakunin wrote:
| Someone has to maintain an always-running listener for
| `/events`. If a server does that, and triggers client calls,
| we call that webhooks. If a client does that, and triggers
| internal functions, it's what the op describes. I think that
| for APIs, `/events` should indeed be the fundamental feature,
| and "webhooks" should be a nice-to-have service on top of
| `/events`, for those who don't want to maintain a local
| subscriber.
| mikepurvis wrote:
| One nice benefit of long polling is the built in catch-up-
| after-a-break functionality: When the client initiates the
| poll, it tells the server the state it knows about
| (timestamp, sequence number, hash, whatever), and the server
| either replies right away if it's different, or waits and
| replies _once_ it 's different.
|
| With webhooks, as in the article, you only get state changes;
| you need some separate mechanism to achieve (or recover) the
| initial state.
| tshaddox wrote:
| That's true, although it's also true of any `/events`
| endpoint that doesn't go back to the beginning of time.
| Stripe's endpoint only goes back 30 days, so you still need
| to solve for the initial state unless you have launch all
| of your desired functionality at the very beginning of your
| Stripe account!
| mikepurvis wrote:
| Hopefully if it's a system like payments where you not
| only need to know state, you _also_ need to know the time
| and nature of all transitions, there 's a way to query
| all of that information.
|
| I'm thinking of simpler situations like my source host's
| CI spinner that seems to get stuck all the time due to
| missing the ping back from Jenkins about build statuses.
| In that case it really would be fine to always just say
| "I think the state is X, please answer me now or in the
| future whenever the state is other than X." I don't care
| about anything other than an up to date sync.
| andrewstuart2 wrote:
| A persistent connection has a cost. Your statement may be
| true in some circumstances but definitely not all. Namely,
| for infrequent events it is much more efficient to be
| notified than to be asking nonstop. Sure, the latency is
| lowest if the connection is already established, but for
| efficiency the answer is not cut and dry but is rather a
| tradeoff decision based on the expected patterns.
| dnautics wrote:
| Exactly. Perhaps an event happens once or twice-ish a day
| per customer, and never on the weekends.
| eptcyka wrote:
| Also, there's the case of ISP's just dropping idle TCP
| connections. It can also take a while to determine that a
| TCP connection is broken.
| runeks wrote:
| What's the issue with that? This will be discovered as
| soon as the endpoint tries to send an event, right? At
| which point the client will see that the connection has
| been closed, reconnect, and receive the event.
| kelnos wrote:
| No, the _server_ will try to send an event, and the
| _server_ will notice the connection has dropped. The
| client will still have no idea until some sort of timeout
| is reached, as the client will usually not be sending any
| data over the connection, as the connection 's sole
| purpose is for the server to send events to the client.
|
| A way to fix this is to use an application-level
| keepalive (TCP keepalives are generally useless), but
| then that increases the load on the server and adds a
| scaling burden.
|
| Meanwhile, unless the event stream is stateful (more
| overhead!), the client has lost all events since the
| connection has dropped, and the client can't even be sure
| _when_ the connection _actually_ dropped.
|
| With webhooks, assuming the callback sending service has
| a generous retry policy, and the customer's receiving
| service does not return 200 unless the webhook has been
| completely processed, or persisted to storage, you won't
| lose events.
|
| I've been at Twilio for the past 10 years. We recently
| started offering an event stream service (that customers
| had been requesting for some time), but it's complicated
| to get right (on both the server and client side) and
| difficult to scale, and, frankly, webhooks have worked
| fine for most customers for a very long time.
| rad_gruchalski wrote:
| > No, the server will try to send an event, and the
| server will notice the connection has dropped. The client
| will still have no idea until some sort of timeout is
| reached, as the client will usually not be sending any
| data over the connection, as the connection's sole
| purpose is for the server to send events to the client.
|
| Exactly why mqtt has the ping packet for the client.
| [deleted]
| fragmede wrote:
| yeah. a(n improperly configured) firewall is going to
| start dropping packets if it thinks a connection is idle
| for too long, so the system never sees an RST and think
| the connection's been terminated.
| jandrese wrote:
| Why for the love of God does the firewall not send the
| RST when it drops the connection?
| FpUser wrote:
| I have the same thing in my backend. The backend has main
| business server that among the other thing has an API endpoint
| one can query for a list of events in date-from - date-to
| fashion. I dismissed the idea of webhooks right at design stage
| as to me it looked like a minefield choke full of potential
| problems.
| cryptonector wrote:
| YES PLEASE. Give us a GET on /events with support for hanging
| GETs.
|
| A hanging GET is where you get chunked encoding (which is the
| only option in HTTP/2 anyways) and possibly never-ending stream.
| I've implemented a (proprietary, for now) "tail -f" over HTTP
| that does this when Range: bytes=0- (i.e., end offset not
| specified), completing the transfer (i.e., final, empty chunk
| sent) only whenever the file is removed or renamed away.
|
| You'll want to add a heartbeat to any hanging GET /events.
| cryptonector wrote:
| My little tail-f-over-http server is for plain files, but this
| scheme works for anything. Of course, if you can output an
| indefinite stream (e.g., PG NOTIFYs, build logs, etc.) you can
| redirect that to a file and then serve up that file.
| qwertox wrote:
| I definitely don't want to see long polling. In that case I'd
| prefer a combination of /events and websockets, where websockets
| can push (or pass via a GET param) the last read event from
| /events to notify the server which is the last known event.
| airstrike wrote:
| TFA addresses this by suggesting long-polling as an option
| rather than the _only_ way to request ` /events`
|
| _> In our integration with Stripe, it would be neat if we
| could request /events with a parameter indicating we wanted to
| long-poll. Given the cursor we send, if there were new events
| Stripe would return those immediately. But if there wasn't,
| Stripe could hold the request open until new events were
| created. When the request completes, we simply re-open it and
| repeat the cycle. This would not only mean we could get events
| as fast as possible, but would also reduce overall network
| traffic._
| fart32 wrote:
| Long polling doesn't scale very well. Webhook/websocket and
| events endpoint combined sounds like the sweetspot to me.
| raksoras wrote:
| One of the complexity of the polling approach on a consumer side
| is having a long running poller. This is trivial to do in Java
| apps - start a polling thread - but not so straight forward in
| case of PHP apps, for example. In that case you'd have to setup a
| cron job or a separate polling script under some sort of process
| supervisor like systemd to poll periodically/continuously.
|
| I wonder if the two approaches could be combined to simplify
| things for consumer apps at the cost of slightly more complexity
| on the producer side? Instead of POSTing the actual event data to
| webhook, the producer just uses consumer's webhook to "poke" it -
| to tell the consumer app "hey, you have new events waiting for
| you". On receiving the poke the consumer endpoint handler/PHP
| script can just turn around and do a GET to "/event" with
| anything > last downloaded event id query. That way you don't
| have to support long polling on the producer's servers and it's
| not a big problem if consumer misses couple of webhook "pokes".
| The next time it does receive a webhook "poke" successfully, it
| will download all the events and be all caught up. If real time
| notifications are not strictly required then producer side can
| even run the webhook dispatching code on a scheduled basis to
| coalesce multiple events in a single "poke" to a consumer to be
| more efficient, if desired.
| candiddevmike wrote:
| Why wouldn't service providers want to offer long polling? It
| seems a lot easier to build (no webhook registration backend) and
| no retry logic/SLAs outside of your control. SSE seems so much
| simpler.
| judge2020 wrote:
| Keeping those connections open probably isn't super cheap and
| complicates deploying rolling updates - you'd need to kill all
| connections when you update and that would require some sort of
| RPC (so that you only kill those existing connections after
| they're done sending in-flight data).
| Nullabillity wrote:
| Why bother? Just kill them and let the clients reconnect on
| their own. They'll have to handle that case anyway...
| judge2020 wrote:
| With polling you have to make sure you finish sending data
| for the current in-flight event. If you don't then the
| client doesn't receive that event, unless you also send
| them events from the past 30 seconds on first poll.
| Nullabillity wrote:
| Don't delete the events from your queue until you receive
| a client ack?
| thrower123 wrote:
| With some of the load-balancers and gateways and things in that
| space I've had to use, long-polling doesn't work at all because
| of short timeouts.
| paxys wrote:
| > It seems a lot easier to build
|
| Quite the opposite. HTTP servers and clients are essentially a
| solved problem. Massive scale-out, load balancing, retries,
| authentication, authorization, rolling deploys etc. can all be
| done out of the box by a hundred different providers. Anything
| to do with maintaining a large number of open TCP connections
| is still a massive pain on the server side.
| handrous wrote:
| Yep. Consider: you can build a damn reliable & resilient
| Webhook handler out of a few lines of PHP or Lua, ready to
| accept a fairly heavy load, zero dependencies, and only
| default-available packages for most any distro or BSD
| (anything where nginx or Apache2 with standard modules is
| available by default) and without tweaking the config at all.
| You can be live in hours, or even inside a single hour if you
| want to cowboy it up pretty hard, and despite not taking a
| lot of care, the webhook-handling part of your service
| probably won't get you woken up at night with a
| everything's-on-fire support call (what it does with the data
| might, of course). Logging? Trivial and standard. Service
| management? It's the OS' default service definition for a web
| server daemon, and that's it. Config and deployment? So tiny
| it'd be nearly no work to document it in a run-once shell
| script, if you don't have anything fancier at hand.
| Operationally, it doesn't get much simpler. "Is it working?"
| checks? You can test it with curl, from any address that's
| able & allowed to talk to it.
|
| With long-polling, now you're managing a custom daemon,
| basically. That's a big step down in reliability-by-default,
| and a bunch more work to do it right.
|
| In either case, you'll be looking at more work if you want to
| check any kind of log on the other end for missed messages,
| but that looks pretty similar for either, and not all systems
| need that level of accuracy (and if they do, they probably
| need _even more_ and this whole thing is Doing It Wrong)
| skybrian wrote:
| If the webhook just triggers a download from /events then both
| can be made idempotent. If you miss an event then you can get it
| later.
| abnercoimbre wrote:
| I like having a one-off command-line app triggered by a
| webhook. If you miss an event, invoke the app manually. If you
| pass it the same event twice it won't matter (idempotency!)
| tlarkworthy wrote:
| This article is premised on the incorrect strawman that webhooks
| are complicated _because_ the consumer has an extra persisted
| message bus.
|
| But if the producer retries and the consumer does not respond
| with 200 until it has processed the message, no consumer side
| message queue is needed, the consumer can rely on the producer to
| reach at-least-once delivery.
|
| In both cases (webhooks and /events) storage is needed producer
| side, so nothing consequential has changed, only with /events you
| need long lived TCP connections which ties up (e.g. you can't do
| this on FaaS endpoints)
|
| Functions as a service are absolutely ideal for low frequency
| webhook receivers. SO SO SO cheap.
| alexbouchard wrote:
| I partially agree but the issue with relying on the producer
| delivery is that you effectively give up control on what the
| retry logic is. If it doesn't fit your use case, too bad for
| you. While ideally every platform would provide those
| configuration I think it's unreasonable to think all platforms
| will offer excellent webhook tools & configuration. You better
| just take things under your own hands.
| coder543 wrote:
| > But if the producer retries and the consumer does not respond
| with 200 until it has processed the message, no consumer side
| message queue is needed, the consumer can rely on the producer
| to reach at-least-once delivery.
|
| Part of the point of the article was that you may deploy bad
| code which returns 200, but doesn't actually take the correct
| action with the events, and then you have lost all that data
| and have no way to get it back, which is why you have a
| consumer-side message bus to hold the webhook history, so that
| you can replay the webhooks if you made a mistake. Your comment
| does not address this at all.
|
| If the service exposes a /events page, and especially one that
| supports long polling (or SSE), then you no longer need a
| consumer-side message bus, and you might not even need webhooks
| at all.
|
| I definitely think webhooks should be offered, but I agree with
| the article that webhooks shouldn't be the only thing.
| villasv wrote:
| The article specifically exemplifies the case of the web hook
| having faulty code (introducing nulls) while still not failing
| with errors. In this case if you don't have storage on the
| consumer side, you have to ask for the producers to be merciful
| gods.
| einrealist wrote:
| I am in favor of doing both: provide Webhooks (Callbacks) and
| Feeds. Webhooks are great as triggers. The payload can be also
| minimal, especially if data is sensitive and authentication /
| authorization is an issue. And Feeds provide data (can be static
| / cached, served by CDNs, optimized for batch processing in
| different variants) at the consumers' pace. The combination of
| both are ideal.
| z3t4 wrote:
| Something that is underestimated in messaging systems are
| incremental numbers, eg. add + 1 for each message, so the
| receiving end get 1,2,3,4,5,6 etc, and if it then get 8 it knows
| it missed the 7th message. And it will know if the messages are
| out of order. And it can pick up if it goes down by requesting
| the missing messages.
| Redsquare wrote:
| Or put simply, a sequence number!
| mrkurt wrote:
| We have (almost) the opposite problem, webhooks are too
| synchronous for what we need to ship to people. We're
| experimenting with giving people a NATs endpoint to listen for
| logs and other events: https://community.fly.io/t/fly-logs-over-
| nats/1540
|
| Having an ephemeral messaging system and a ledger to reconcile
| against is a nice, simple way to provide immediacy and eventual
| consistency (where eventual could be days). It's a pattern we're
| using all throughout our infra.
| joelcollinsdc wrote:
| We have this problem too. When we send a request to create an
| entity in an external system, before we get the response back
| with the entity Id, we already have received a webhook saying
| said entity was created. Makes a basically trivial workflow
| quite confusing.
| rkalla wrote:
| FWIW, this is what CouchDB has done from Day 1 and it _always_
| seemed like one of the most magical and surprising things tool
| or platform providers would suddenly realize and then LOVE
| about using the DB.
|
| There was nothing fancy about it, you could just listen to an
| endpoint and it was a stream of the append only log of events
| occuring in the DB to the point that you could literally use it
| to feed a replicated master or slave (or backup).
|
| I imagine your use-case is a bit more nuanced, but I sure do
| love that model.
| grejdi wrote:
| Webhooks are great for producers of events, and I'd argue that
| it's too cumbersome for them to provide an '/events' endpoint
| primary because of scaling. With webhooks, they can offload
| events at their own pace.
|
| For consumers, I agree with most here that Kafka is certainly
| overkill. We've gotten away with a very simple architecture to
| have reliable event consumption. We point all webhooks to an
| (AWS) API Gateway backed by Lambdas. The Lambdas push the events
| to an SQS queue (FIFO-queue, if it needs some sort of sequence),
| and we take our time consuming the events through a very generic
| poll.
| jcrites wrote:
| > With webhooks, they can offload events at their own pace.
|
| They can't offload webhooks at their own pace if the two
| parties want reliable delivery. The server providing the
| webhook might be experiencing a prolonged outage, in which case
| the sender needs the ability to buffer the events anyway.
| masklinn wrote:
| > Webhooks are great for producers of events, and I'd argue
| that it's too cumbersome for them to provide an '/events'
| endpoint primary because of scaling. With webhooks, they can
| offload events at their own pace.
|
| TBF they could do something similar with `/events`, instead of
| pushing events to a webhooks-sending queue just push them to
| the events buffer, which could even be a circular buffer just
| to point out that the essay is completely wrong. TFA is not
| asking for /events, they're asking for a very specific kind of
| /events with a large non-drained buffer. Something which would
| only ever work for low number of events: $dayjob's github
| integration takes in several events per second.
|
| A proper event stream would be nice though, github's webhooks
| delivery system is not exactly reliable.
| deniska wrote:
| I'm building integrations with various marketplaces at a company
| I work at (fulfilled by seller kind of deal), and I can confirm,
| it's much easier for us to schedule an HTTP request once in 15
| minutes than to create a custom HTTP service responding to
| specific requests from 3rd parties.
|
| We're in the business of selling things, we're not in the
| business of building HTTP services. Our ERP-like thing is down
| for maintenance from 10pm to 11pm, so we can't use it as a
| platform for responding to webhooks.
|
| I'll hack something together in a pinch when it's the only way to
| get orders from a marketplace service, but then eventually I'll
| have to explain to my colleagues how to linux, how to HTTPS, how
| to python, how to WSGI, and all other stuff our company typically
| doesn't do, but has to do now, because this particular
| marketplace wants to POST orders to us.
| mkherlakian wrote:
| You might want to check out https://hookdeck.com (I work on
| it). We built it precisely for this use case, you shouldn't
| have to spend of bunch of time building webhook ingestion
| infrastructure.
| boring_twenties wrote:
| Just curious, how would you _prefer_ to receive these
| notifications?
| deniska wrote:
| By polling an endpoint provided by the marketplace with
| parameters "since" and "to" to filter events by the time they
| happened. We typically set "since" to two days ago and "to"
| to tomorrow. We're not in the hurry, we have an hour or two
| of leeway between receiving a message and having to act on
| it. I certainly prefer the polling solution for that usecase.
| Easier to set up, easier to debug, easier to notice that
| something is wrong.
| dceddia wrote:
| Maybe there's an opportunity here for some kind of
| buffering service that would receive webhooks and present
| them as a stream of events. Or maybe something like this
| already exists?
| deniska wrote:
| That's more or less what I implemented with a bit of
| python and sqlite. It works, but it's another piece of
| infrastructure to care about in a shop full of people who
| never had to care about that kind of infrastructure. For
| example we (well, I, really) forgot to configure certbot
| to restart nginx after renewing a cert, and only noticed
| that after a marketplace notified us that they're
| temporarily pulling off our SKUs due to our HTTP service
| being misconfigured.
|
| Can this be a 3rd party service? It certainly can be, but
| it's hard to make a generic one for any kind of webhook.
| Some marketplaces expect a dynamic response, like
| replying the order number we assigned internally (I
| typically just echo back the number they gave us with
| some prefix, but it's still more smarts than just
| replying with empty 200 OK).
|
| And I've seen services which aggregate popular local
| marketplaces into API which is easier to work with, but
| they require to concede some other parts of the business
| we'd rather keep in-house, like assortment and inventory
| management.
| boring_twenties wrote:
| I was thinking the same thing, it's kind of why I asked
| the question. :)
|
| On the one hand, it seems like something too simple to
| expect people to pay for.
|
| On the other, it's so simple it wouldn't be a huge loss
| to try it out and see if they will.
| supergeek133 wrote:
| Working in IoT land, we rely quite a bit on EventHub/AMQP type
| delivery for events to say a 3rd party (e.g., Alexa/Google/Etc).
|
| That being said, I've also lobbied for a similar endpoint so I
| don't get support tickets for "missing data".
|
| Both? Both are good.
| dgudkov wrote:
| I think the author is fighting the wrong problem. Webhooks are a
| _notification_ mechanism first of all, not a data transfer
| protocol. You can view it as a control plane which can be mixed
| or not with a data plane.
|
| What they offer is a data plane and it makes sense. Although, it
| doesn't contradict the idea of webhooks, but rather complements
| it. A consumer can get notified via a webhook when new data is
| available. Whether the data itself comes with the webhook, or is
| available via an additional API request, is a matter of design.
| Personally, I like the idea of separating the control plane from
| the data plane. However, in some cases it can be an overkill.
___________________________________________________________________
(page generated 2021-07-13 23:00 UTC)