[HN Gopher] Give me /events, not webhooks
___________________________________________________________________
Give me /events, not webhooks
Author : todsacerdoti
Score : 525 points
Date : 2021-07-13 16:41 UTC (1 days ago)
(HTM) web link (blog.syncinc.so)
(TXT) w3m dump (blog.syncinc.so)
| closeparen wrote:
| I have long felt there is a startup opportunity for a service
| that receives webhooks and lets you subscribe via message queue.
| rakoo wrote:
| History does repeat itself. All of those issues, and the
| solution, are the reason CouchDB is modeled the way it is:
| there's a single endpoint that gives you _all_ events happening
| in the database, in chronological order, with both document ids
| and "feed" ids, reachable with long-polling. All of this more
| than a decade ago already.
| recursivedoubts wrote:
| why not both?
| freedomben wrote:
| This is just my opinion of course, but I've consumed a great
| number of APIs and I love when there is. You don't even need
| /events as long as there's a good index endpoint for the object
| in question.
|
| For a great number of applications it's not that big of a deal to
| miss a webhook, and the extreme simplicity that it gives the
| developer is worth a great deal. With how enormously complex a
| lot of systems have gotten, I really favor simplicity whenever
| possible.
| lxe wrote:
| Holding requests open for long polling is probably difficult to
| scale. How do you persist connections when application servers
| drop in/out? How do you load-balance? What about socket
| descriptor limits? etc...
| maerF0x0 wrote:
| IMO the real solution is give me a better transport that RESTful
| HTTP! As many others have pointed out things like Kafka are built
| for these kinds of usecases. So often I see people trying to
| design around the flaws in REST while ignoring that we've had
| some pretty good progress in the ensuing 20yrs.
| dotdi wrote:
| I agree that webhooks have their problems, but, as somebody
| maintaining software that uses long-polling heavily, I will say
| that long-polling is quite difficult to handle reliably "over the
| internet".
|
| ISPs have very different views on how long a connection is
| allowed to be kept open, and will absolutely kill long-polling
| connections without remorse. This extends beyond ISPs, too.
| Practically all network infrastructure between API consumer and
| API will have to accommodate long TCP connections, which is
| unfortunately not as trivial as it sounds.
|
| The next cool feature of long-polling is that the server might
| not know that the connection is broken until it actually tries
| writing to it, so if the back-end relies on having or not having
| a connection, this will make for some interesting edge-cases.
|
| Load-testing products using long-polling also has interesting
| implications, such as hitting the "65k" problem on test clients
| and network infrastructure.
|
| Add another layer on top (or below, like Kubernetes, OpenShift,
| whatever) and you should be getting a prescription for anti-
| depressants from the very beginning, because you WILL need them.
| webmaven wrote:
| _> Add another layer on top (or below, like Kubernetes,
| OpenShift, whatever) and you should be getting a prescription
| for anti-depressants from the very beginning, because you WILL
| need them._
|
| I guess this is the current zeitgeist's version of "stock up on
| $PAINKILLER" or even "keep a bottle of $LIQUOR hidden in the
| bottom drawer of your desk".
| bkrausz wrote:
| I was responsible for Stripe's API abstractions, including
| webhooks and /events, for a number of years. Some interesting
| tidbits:
|
| Many large customers eventually had some issue with webhooks that
| required intervention. Stripe retries webhooks that fail for up
| to 3 days: I remember $large_customer coming back from a 3 day
| weekend and discovering that they had pushed bad code and failed
| to process some webhooks. We'd often get requests to retry all
| failed webhooks in a time period. The best customers would have
| infrastructure to do this themselves off of /v1/events, though
| this was unfortunately rare.
|
| The biggest challenges with webhooks:
|
| - Delivery: some customer timing out connections for 30s causing
| the queues to get backed up (Stripe was much smaller back then).
|
| - Versioning: synchronous API requests can use a version
| specified in the request, but webhooks, by virtue of rendering
| the object and showing its changed values (there was a
| `previous_attributes` hash), need to be rendered to a specific
| version. This made upgrading API versions hard for customers.
|
| There was constant discussion about building some non-webhook
| pathway for events, but they all have challenges and webhooks +
| /v1/events were both simple enough for smaller customers and
| workable for larger customers.
| ctas wrote:
| Can you share a bit about how these events are stored on
| Stripes backend e.g. Kafka, Postgres?
| bastawhiz wrote:
| It's all just kafka and mongo. The event can be stored in any
| simple k/v storage. There's no magic.
|
| Edit: not sure why I'm being downvoted. I work at stripe and
| this is literally how it works.
| ctas wrote:
| Thanks for the input. We're currently working on a similar
| solution, so I was really curious to learn more.
|
| One thing I really admire is how Stripe makes it
| transparent which events were fired both in general through
| the Developer area, and on specific objects like customers,
| subscriptions, etc..
| bkrausz wrote:
| Hi Basta! Can confirm both that he works at Stripe and is
| right.
|
| Years ago there wasn't even a Kafka portion, that's newer.
| Eiriksmal wrote:
| At Lawn Love, we naively coupled our listening code directly to
| the Stripe webhook... but it worked flawlessly for years. I
| wasn't a big fan of the product changes necessitating us
| switching from the Transfer API for sending money to the
| complicated--and very confusing for the lawn pros--Connect
| product, but its webhooks also ran without issue from the
| moment we first implemented them. So thanks for making my life
| somewhat easier, Mr. Krausz.
|
| Like many others, I now pattern my own APIs after Stripe's.
| bkrausz wrote:
| Don't fully thank me, I was also the architect of the
| Transfers API to Connect transition :). There's a lot I would
| have done differently there were I doing it again, though
| much of the complexity (e.g. the async verification webhooks)
| were to satisfy compliance needs. Hard to say how much easier
| the v1 could've been given the constraints at the time,
| though I'm very impressed with the work Stripe has done since
| to make paying people easier (particularly Express).
| alexbouchard wrote:
| Shameless plug but I've built https://hookdeck.com precisely to
| tackle some of these problems. It generally falls onto the
| consumer to build the necessary tools to process webhooks
| reliably. I'm trying to give everyone the opportunity to be the
| "best customers" as you are describing them. Stripe is big
| inspiration for the work.
| Redsquare wrote:
| Do you provide the ability to consume, translate then
| forward? I am after a ubiquitous endpoint i can point
| webhooks at and then translate to the schema of another
| service and send on. You could then share these 'recipes' and
| allow customers to reuse well known transforms.
| alexbouchard wrote:
| Transformations is something we haven't built yet but we
| have our eyes on it as your are not the first one to bring
| that up. You can use Hookdeck in front of lamda and to the
| transformation there, you'd still get the benefit of async
| processing, retries, etc
| rkazokas wrote:
| Do you have an idea of when hookdeck will have
| transformations. It's not something we need immediately
| but would be the win over something like:
| https://webhookrelay.com/ if it's something you have on
| your roadmap for sometime soon.
| alexbouchard wrote:
| Can you reach out to me, I'd love to talk about your use
| case and prioritize accordingly. Email is alex at
| hookdeck dot com
| JimDabell wrote:
| This is a fairly common use of no-code glue services like
| Zapier, IFTTT, Cyclr, etc.
| dools wrote:
| You can do this with BenkoBot, we just launched custom
| webhooks (although it's not in the interface yet). So you
| can receive a webhook and run some arbitrary javascript to
| transform it then send it on somewhere else:
|
| http://www.benkobot.com/
|
| Our main focus is on handling Trello notifications, and the
| Trellinator library I wrote is built in, our objective is
| to create more API wrappers over time to make it as simple
| as possible to deal with as many APIs as possible. You can
| see some example code here:
|
| https://trello.com/b/IoHmhz5c/benkobot-community-board
|
| You currently require a Trello account API key/token to
| sign up, but you can use it as you described to be a
| generic endpoint, transform however you want with JS then
| post the data onto another endpoint.
| rattray wrote:
| > We'd often get requests to retry all failed webhooks in a
| time period.
|
| (I worked on the same team as bkraus, non-concurrently).
|
| For teams that are building webhooks into your API, I'd
| recommend including UI to view webhook attempts and resend them
| individually or in bulk by date range. Your customers are
| guaranteed to have a bad deploy at some point.
| spullara wrote:
| Pretty easy for a customer to setup an SQS queue and a lambda
| for receiving them rather than rely on their infrastructure to
| do all the actual receiving. Way more reliable than coupling
| your code directly to the callback.
| jon-wood wrote:
| This is precisely what we do where I work. We have a service
| which has just one responsibility - receive webhooks, do
| _very_ basic validation that their legitimate, then ship the
| payload off to an SQS queue for processing. Doing it this way
| means that whatever's going on in the service that wants the
| data, the webhooks get delivered, and we don't have to worry
| about how 3rd party X have configured retries.
| tasn wrote:
| These reasons are exactly why we started Svix[1] (we do
| webhooks as a service). I wish we existed to serve you guys
| back when you started working on it. :)
|
| [1] https://www.svix.com
| throwaway290232 wrote:
| I always laugh when people end up with designs like this. They
| could have just used SMTP! It's designed to reliably deliver
| messages to distributed queues using a loosely-coupled
| interface while still being extensible. It scales to massive
| amounts of traffic. It's highly failure-resistant and will
| retry operations in various scenarios. And it's bi-directional.
| But it's not "cool" technology or "web-based" so developers
| won't consider it.
|
| Watch me get downvoted like crazy by all Nodejs developers.
| Even though they could accomplish exactly what they want with
| much less code and far less complex systems to maintain.
| andyxor wrote:
| going down this non-traditional path you might also consider
| using XMPP and ejabberd for machine-to-machine messaging
| nijave wrote:
| >so developers won't consider it
|
| I think it depends on the developer. There's developers
| hammering out boring business logic as fast as possible and
| there's developers with a deep understanding of machine
| internals, protocols, and infrastructure. For the former,
| SMTP is black magic they'd probably never think of and
| involves engaging the one infra person that's always busy
|
| It also means standing up and managing "infrastructure"
| throwaway290232 wrote:
| I sort of agree, but somebody already has to manage the
| "infrastructure" of their web apps, dns. They never mind
| adding more of their own home-grown services. If they used
| Kinesis instead that's another piece of infra to maintain.
| But you would never hear them say "what about Postfix
| instead". Regardless of infra, if it's new, they want to
| use it, even if something older and more boring would work
| better.
|
| If I ever heard a dev at work say _" No I won't use that
| new tech, it's too untested/I'll have to spend more time
| figuring out how to make it work well"_, I would shit my
| pants. Whereas if it's _old_ tech, _" it's not modern/I'll
| have to spend more time figuring out how to make it work
| well"_. It's practically software ageism...
| wruza wrote:
| You're likely blinded by a "nodejs monkey developer"
| stereotype which prevents you to see that node is what
| _everyone_ wanted back then. It's very, very easy to
| create an http-based analog of any "traditional" service
| in node and to free yourself from learning all the shady
| details (which there is a lot) of configuring it and
| keeping it alive at all levels, were it based on
| traditional software. Node is extensible configurable
| networking itself, and http(s) is a quintessence of all
| text protocols. All that we wanted back then is available
| now in node at much finer granularity and much less
| configuration or headache. "They" spin up home-grown
| services because it is a natural one-page-boilerplate
| straightforward thing to do in node, not because of
| ageism or something similar.
|
| I tell you that as someone who fiddled with sendmail.cf's
| and other .conf's way too much long before nodejs became
| a thing. Now it's a relief.
| vidarh wrote:
| I actually did use SMTP as queuing middleware for a registrar
| platform years ago.
|
| It worked very well.
|
| EDIT: To add some context, my team had come off building a
| webmail platform, and so we'd done lots of interesting stuff
| to qmail and knew it inside out. We then launched the .name
| tld and built a model registrar platform that on registration
| would bring up web and mail forwarding for users that wanted
| it. We used SMTP to handle the provisioning of those while
| keeping the registration part decoupled from the servers
| handling the forwarding. We also used it to live-update a
| custom DNS server I wrote.
| buro9 wrote:
| I remember interviewing someone who worked on a DNS
| platform where IIRC the DNS zone files were propagated by
| SMTP to DNS servers. The details on this were that there
| was a 5-minute SLA (I believe) on the loading of zone
| records, essentially that the DNS servers were polling the
| mailbox and parsing new records since some last loaded time
| stamp.
| vidarh wrote:
| For a second there I wondered if you'd ever interviewed
| me (but having looked at your profile: no; I don't think
| we've met, though I'm in London too).
|
| We had similar-ish constraints. SLA was internal, not
| imposed (the .name _registry_ had externally imposed SLA
| 's, but the registrar platform did not), but the zones
| were very simple - either NS records pointing elsewhere,
| or identical CNAME/MX records, so we needed only a short
| string per address.
|
| I don't remember if we used CDB files or if we stored
| individual records directly in ReiserFS filesystems (our
| mail platform had relied heavily on the ability of
| ReiserFS to handle vast quantities of tiny files, so were
| comfortable with that), but it was definitively something
| simple.
|
| Similar for the web forwarding, which just required a url
| to redirect to.
|
| If a node should ever need to be replaced, all we'd need
| to do would be to start a queue on a new box but not
| process it, then rsync over the dataset from another
| server, and start processing the queue, and add it into
| rotation when up to date. If we'd needed stricter
| consistency guarantees it'd have been a different
| consideration.
|
| For many types of workloads I'd pick another queuing
| system today, but the amount of readily available tooling
| for e-mail, especially once you need federation,
| reflection/amplification etc. does make it an interesting
| choice for some things.
|
| It also made debugging the message flow trivial: just add
| a real mailbox to the cc:....
| thakoppno wrote:
| You have this Nodejs developer's upvote.
|
| At this point in my career (10 years in the game), let me
| simply defend node as the tool that got me here. Using it
| then to bootstrap my career was just as practical as using
| SMTP as you describe now.
| tylerscott wrote:
| I absolutely love your perspective. I feel the same way.
| s/Ruby+Rails/node for my situation. I believe there needs
| to be more respect paid to "bad" technologies. The
| measuring stick should include things outside pure
| benchmarks. Low barrier to entry technologies provide broad
| access and real life changes to folks that are able to pick
| them up and get hacking.
| pmelendez wrote:
| >And it's bi-directional. But it's not "cool" technology or
| "web-based" so developers won't consider it.
|
| I might be missing a point or two here, but I don't see how
| SMTP can work for this case at all. You would require every
| API consumer to setup a SMTP server (which is another piece
| of infrastructure to maintain), and then somehow have a layer
| of authentication so the recipient can control who post
| messages on that server (overhead for the publisher per new
| customer). Then we still haven't resolved the issues on the
| customer side (a bad code that could pop all messages and now
| we might require the publisher to replay them again).
|
| I haven't even started to think about security and network
| hardening challenges yet. Again, I might be missing the point
| but this is not a case of cool tech overuse to me.
| vidarh wrote:
| SMTP servers supports SSL. Using client certificates and/or
| HMAC signed messages takes care of the security. You have
| the same security consideration for HTTP.
|
| As for "setting up an SMTP server", the point is that
| compared to the current requirement of a webhook, you're
| _going to need_ a queuing mechanism _or_ a pull mechanism
| _or both_ anyway. So you can build a custom solution, or
| you can pick an existing queuing mechanism that people have
| spent literally _decades_ providing a vast array of
| software options for.
|
| And yes, you're right, you can always ending up needing a
| way to trigger a replay because no matter what you do the
| customer might do something stupid. Nothing you do will get
| you away from that. So either you require them to always
| pull, or you provide an option to push and an API to
| trigger redelivery for when they've done something stupid.
| _If_ you opt for push, SMTP is an option worth considering,
| because _no other queuing mechanism_ has as many available
| ready-made and battle hardened queuing options.
|
| There are many cases where it'd not be suitable, but in the
| situations where SMTP is a bad choice, webhooks are likely
| to be an absolutely awful choice.
|
| I speak from actually having run messaging on SMTP both as
| an e-mail provider with a couple of million users _and_
| having used it as messaging middleware in production.
| rendall wrote:
| The suggestion to use SMTP is interesting.
|
| I didn't downvote you but I bet they come from this part.
| People don't like this kind of negativity.
|
| > _But it 's not "cool" technology or "web-based" so
| developers won't consider it. Watch me get downvoted like
| crazy by all Nodejs developers. _
| [deleted]
| forgotmypw17 wrote:
| I agree that people don't react well to negativity, but
| sometimes you have to say it. Node has a lot of very stupid
| (i.e. ignoring reality) decisions, and by extension, being
| exposed to this for a long enough period of time, tends to
| affect the developer as well.
|
| I say this from experience, as someone who's used a few
| stupid technologies over time.
| kreetx wrote:
| Since it doesn't relate at all, why even add the
| negativity?
|
| "Here's a crazy idea ... This are the properties for why
| it works ..." Would have sounded much better.
| strken wrote:
| This was not a time when you had to say it. None of the
| existing conversation was about languages or ecosystems,
| and taking cheap shots was wholly unnecessary to the
| suggestion of using SMTP.
|
| SMTP itself is interesting, although it comes with fun
| new footguns like STARTTLS.
| cutemonster wrote:
| What makes STARTTLS a footgun? (I'm curious since I use
| it sometimes)
| virtue3 wrote:
| "STARTTLS is an email protocol command that tells an
| email server that an email client, including an email
| client running in a web browser, wants to turn an
| existing insecure connection into a secure one."
|
| I'm sure absolutely nothing bad will come from that last
| bit. Oh look:
|
| "And yes, STARTTLS is definitely less secure. Not only
| can it failback to plaintext without notification, but
| because it's subject to man-in-the middle attacks. Since
| the connection starts out in the clear, a MitM can strip
| out the STARTTLS command, and prevent the encryption from
| ever occurring."
| vidarh wrote:
| That's only a footgun if your system is set up to allow
| an insecure connection to continue. Just because the
| protocol allows it does not mean you can't add additional
| requirements.
| teddyh wrote:
| DANE is meant to fix that. If someone asserts, via DNS
| records (signed by DNSSEC), that their SMTP server is
| able to use TLS, then you should only accept connections
| using TLS to that SMTP server.
| tptacek wrote:
| And DANE is never going to happen; DANE advocates have
| been saying this for over a decade, and the only change
| has been that the IETF and all the major email providers
| moved forward on a new protocol, MTA-STS, specifically to
| avoid needing DNSSEC (which nobody uses) to solve this
| problem.
| teddyh wrote:
| Almost every time anyone mentions DNSSEC here on HN, you
| pop up like a jack-in-the-box to claim that nobody is
| using it and that it is dead. And it's always _you_ ,
| nobody else. Whereas, from where I sit, I work at a
| registry and DNS server host (among other things) where
| about 40% of all our domains have DNSSEC (and that number
| is constantly climbing). Every conference I go to, and in
| every webinar, people seemingly always talk about DNSSEC
| and how usage is increasing.
|
| You might have some valid criticism about the
| cryptography; I would not be able to judge that (except
| when you are basing it on wildly outdated information).
| I'm not an expert on the details; you could most
| assuredly argue circles around me when it comes to the
| cryptography, and possibly about the DNSSEC protocol
| details as well. But, from my perspective, your
| continuous claim that "nobody uses" DNSSEC is simply
| false. DNSSEC works, usage of DNSSEC is steadily
| increasing, and new protocols (like DANE) are starting to
| make use of DNSSEC for its features. Conversely, I only
| relatively rarely hear anything about MTA-STS.
| tptacek wrote:
| Take any list of the top domains on the Internet --- any
| of them at all --- and run them through a trivial script,
| like: #!/bin/sh while read
| domain do ds=$(dig ds $domain +short)
| echo "$domain $ds" done
|
| ... and note that virtually none of the domains, in any
| sane list of top domains, are signed. That was true
| several years ago and remains true today, despite the
| supposed "increase in usage" of DNSSEC.
|
| What's actually changed is that registrars, especially in
| Europe, now apparently auto-sign domain names. That
| creates a constant stream of new, more-or-less ephemeral
| signed zones that gives the appearance of increasing
| DNSSEC adoption. Of course, this is also security theater
| (the owners of the zones don't own their keys!). The real
| figure of merit for DNSSEC adoption is adoption by sites
| of significance, and that has been static, and
| practically nonexistent, for a decade.
|
| It is no surprise to me that people working on the DNS
| talk quite a bit about DNSSEC. People who worked on SNMP
| talked quite a bit about SNMPv3, and IPSEC people
| probably really believed there would be Internet-wide
| IKE. None of those things happened, because what matters
| in the real world is what the market decides. Most
| especially at the companies with serious security teams,
| DNSSEC is a dead letter standard.
| cutemonster wrote:
| And, reading more at StackOverflow, from where Virtue3's
| quotes are?, This:
| https://serverfault.com/questions/523804/is-starttls-
| less-sa...
|
| I find:
|
| > If the client is configured to require TLS, the two
| approaches are more-or-less equally safe. But there are
| some subtleties about how STARTTLS must be used to make
| it safe, and it's a bit harder for the STARTTLS
| implementation to get those details right.
|
| I previously thought that was the default, good to know
| it isn't / might not be
|
| Thanks everyone :-)
| rendall wrote:
| > _Node has a lot of very stupid (i.e. ignoring reality)
| decisions, and by extension, being exposed to this for a
| long enough period of time, tends to affect the developer
| as well._
|
| As a developer who used Salesforce for nearly a year once
| upon a time, I can confirm that exposure to stupid
| decisions in a platform can affect the developer.
|
| Node, though? Could you expand on the stupid decisions in
| Node? And does Deno address those?
| IggleSniggle wrote:
| I use and love nodejs daily, and I think I can speak to
| some of the stupid. A lot (but not all) has to do with
| ecosystem.
|
| Some of the stupid in node just comes from the fact that
| there's still a lot of reinventing the wheel, and doing a
| less good job of it. Like, we've got all these backend
| frameworks, but still nothing at all that compares to eg
| Spring. Can you even find a nodejs lib that does HATEOAS
| properly and completely? How often do you find yourself
| doing string parsing, or handling a JSON object, when you
| know it would be more efficient to be handling a stream,
| or that really the kind of work you're doing ought to be
| handled by your framework but isn't?
|
| As for nodejs itself, it's much better in 2021 than it
| was in the past. But it's still a massive runtime. And I
| have mixed feelings about eg Worker threads. As for
| node_modules, I get the sense that we're just replaying
| the history of Microsoft's dll story, needing to relearn
| all the lessons that should have been learned already.
|
| As for Deno, I think it comes with great ideas. In many
| respects, I like it better than Nodejs. Most of its good
| ideas, Nodejs is flexible enough to accomodate. One of
| Deno's main advantages is that it doesn't have any legacy
| to support, so it can embrace things like ECMAScript
| modules more easily. Its library system is closer to Go,
| although I think the end result is that a lot of folks
| end up doing one-off systems that end up looking a lot
| like the nodejs module resolution system in the end.
| Deno's main disadvantage is that it is not compatible
| with nodejs libraries. That's also an advantage insomuch
| as you have a clear module import spec from the get-go.
|
| In short, the stuff that Deno can do, Nodejs can do, and
| I'm not sure that it's cleaner system can overcome the
| fact that the same is accomplishable in Nodejs. I'd be
| more than willing to use Deno in a greenfield project
| because I like all the technology choices it makes, but
| fundamentally, the technology choice you're making is
| whether or not to use V8, and adopting Deno is almost
| just a way of pressing the reset button the ecosystem,
| which may or may not be a good thing depending on your
| needs.
| Aeolun wrote:
| Does Node have that? Or do node libraries have that?
|
| I find node to be surprisingly well rounded.
| daniellarusso wrote:
| The very first startup I worked at used this for a
| sweepstakes leadgen form to send to MySQL via a Perl script
| running from cron.
| [deleted]
| mrzimmerman wrote:
| Honestly this is so stupid brilliant I love it (stupid as in
| I can't believe I hadn't considered this). Honestly it really
| is about storing, sending, and checking messages so SMTP
| makes so much sense!
|
| I've been building for the web for 15 years and it shows how
| far I can hyper focus on certain communications
| implementations that I'm not looking at pre-existing options
| that really meet a large number of use cases. I suppose it
| also means making sure your data consumers are comfortable
| working with the protocol but it's a really top notch idea.
| johannes1234321 wrote:
| E-Mail isn't trustworthy. You may get a confirmation that an
| initial SMTP server accepted a mail, but that's it. There's
| also no good way to detect that an endpoint (receiver
| address) is gone for good to stop sending messages.
|
| You will probably point me to SMTP success messages, but a
| removed mailbox might only be known by a backend server.
|
| Also mail infrastructure will potentially include heavy spam
| filters etc. making it quite inconvenient. Not even
| mentioning security aspects with limited availability of
| transport layer encryption with proper signatures.
| vidarh wrote:
| What you're saying is true of public e-mail infrastructure,
| but that's besides the point. As a queuing solution
| _internally_ in a system you can make it as resilient as
| you like with ease because there 's a huge ecosystem of
| resilient software you can use for it.
|
| Same goes for security - your objection is true for public
| e-mail delivery without additional requirements on the
| servers or clients, but that is not relevant for a private
| infrastructure.
| johannes1234321 wrote:
| In a private environment you have tons of options. The
| post however refers to notification between independent
| entities on the public network.
| vidarh wrote:
| Running over the public internet does not mean you rely
| on unknown third party mail servers. If I address a
| message to foo@apiendpoint.mycustomer.com, only the
| servers configured to handle mail for
| apiendpoint.mycustomer.com and my sending server is
| involved in the exchange. And that is if you trust MX
| records for this exchange rather than have the customer
| input the address of the receiving SMTP server directly.
| paulddraper wrote:
| > SMTP
|
| But... Why?
|
| The HTTP protocol is so much easier to manage, load balance,
| use, etc.
| berkes wrote:
| The article is almost entirely the answer to your question.
|
| > there are risks when you go down.
|
| Solved by SMTP on protocol level. With HTTP, must be solved
| on _both_ client and servers application level.
|
| > webhooks are ephemeral. They are too easy to mishandle or
| lose.
|
| SMTP has this baked into its heart. Loosing messages is
| possible, certainly, but rather hard to do. With HTTP, it's
| really simple.
|
| > In the lost art of long-polling, the client makes a
| standard HTTP request. If there is nothing new for the
| server to deliver to the client, the server holds the
| request open until there is new information to deliver.
|
| SMTP is push, not polling. So all those issues are solved
| for you.
| teddyh wrote:
| > _SMTP is push, not polling._
|
| Yes. And if you want to poll, POP is polling, and IMAP
| has both polling and immediate notifications.
| salamander014 wrote:
| Yes, but SMTP is a protocol wherein a system opens a
| connection to another system and says, hey here's a
| message.
|
| And if that doesn't succeed for some reason, it reliably
| queues and retries.
|
| That's a push.
| teddyh wrote:
| I was not disagreeing, merely providing additional
| information. I have edited to clarify.
| fny wrote:
| So how are people supposed to consume this? With an SMTP
| client?
|
| I think the bigger issue is that consumption isn't
| particularly friendly. Also, you still haven't solved the
| versioning issues.
| jmiserez wrote:
| There are better options than SMTP. Basically any message-
| oriented middleware / message queuing service can provide
| this. It's great for both sides, maintenance/outages can
| happen independently, as long as the queue stays online and
| has space everything is fine.
| teh_klev wrote:
| We used to do this for domain name registrations and it
| worked fairly well for years. However once you've been added
| to a spam blacklist it quickly breaks down, especially for
| time critical operations such as domain name renewals when
| you're scrabbling around trying to appease the Spamhaus gods.
| NicoJuicy wrote:
| And far too slow for a lot of use-cases
| vidarh wrote:
| There's nothing about SMTP that makes it slow. There's lots
| about _public e-mail infrastructure_ that sometimes makes
| it slow.
| user5994461 wrote:
| SMTP won't work for the customers.
|
| Developers won't be able to use the existing email systems of
| the company, too critical and managed by another team. They
| will never be able to reconfigure it and get API access to
| read emails. Note that it may or may not be reliable at all
| (depends on the company and the IT who manages it).
|
| Developers won't be able to setup new email servers for that
| use case. Security will never open the firewall for email
| ports. If they do, the servers will be hammered by
| vulnerability scanners and spam as soon as it's running. Note
| that large companies like banks run port scanners and they
| will detect your rogue email servers and shut it down
| (speaking from experience).
| vidarh wrote:
| Nothing preventing offering delivery on alternative ports
| for people with incompetent security teams that thing port
| numbers is sufficient to determine if it's a threat.
|
| As for "being hammered", rejection of invalid recipients
| before even getting to the DATA verb is cheap.
|
| Having actually _run_ both an e-mail service and SMTP used
| as messaging middleware, I have dealt with these issues.
| user5994461 wrote:
| The security team is not incompetent. Large companies do
| not permit developers to spin up their own email systems
| without audit and regulatory retention. The port number
| is sufficient to determine that the request should be
| rejected.
|
| You could work around it but should you? You're exposing
| the company to fines and risking your job.
|
| Better think of another way to integrate with the vendor,
| or find another vendor.
|
| P.S. SMTP is easy to identify on ANY port, it's replying
| a distinctive line of text when TCP connection is opened.
| vidarh wrote:
| > do not permit developers to spin up their own email
| systems without audit and regulatory retention
|
| If they freak out over an SMTP server but don't freak out
| over a web server, then the are indeed absolutely utterly
| incompetent fools that should never work in this space.
|
| In both cases code written by the company developers will
| eventually process untrusted textual input, and you need
| to deal with that with the same level of caution, and the
| protocol does _nothing_ to change that.
|
| > You could work around it but should you? You're
| exposing the company to fines and commiting a fireable
| offense. Better find another product that's easier to
| deploy.
|
| I would not work around it - I would make the case that
| there's no difference in exposing a carefully chosen SMTP
| server than exposing a web server, and if the security
| team fail to understand that, I'd resign, because it'd be
| a massive red flag, and I've been successful enough to be
| in a position to not need to work for companies like
| that.
|
| For that matter, in 25 years in this business I've yet to
| run into your hypothetical scenario, including at large
| companies, so I'm not at all convinced it'd be a genuine
| problem. Yes, I've been at companies where I'd need to
| provide a justification for getting a port opened. But
| never once had an issue getting it approved - including
| SMTP.
|
| > P.S. SMTP is trivially identifiable on ANY port, it's
| giving a line of text when the TCP connection is opened.
|
| I was responding to "Security will never open the
| firewall for email ports.". Point being that if they care
| about the specific port numbers, it doesn't matter.
|
| [And I'll again point out I've actually run
| infrastructure like this].
| user5994461 wrote:
| Never worked in a bank? Never worked in defense?
|
| I'm speaking from real experience too. It takes a while
| to open firewall in some environments, if you ever can.
|
| One bank was the worst. There was a super stringent
| process to expose things externally. Opening the firewall
| port was just the beginning and that'd take 2-4 weeks if
| all goes well.
|
| You'd struggle like hell to expose a SMTP server though
| because it would be immediately be rejected and flagged
| based on the port. Banks have to store, monitor and
| ensure the origin of all emails, they don't allow shadow
| email servers. And it's plain text so more reasons to ban
| (also a problem with HTTP, you should do HTTPS if
| anything).
|
| Defense was simpler, mainly because there was no external
| connectivity in many cases. You don't need to worry about
| how to open a firewall when there's none :D
| kbenson wrote:
| SMTP used to be a lot more reliable than it is now. Now, with
| all the changes to help with blocking spam, you have to be
| very careful or have a lot of control over the receiving
| server to ensure you actually get delivery. Some anti-spam
| systems will just discard if the matching rules indicate the
| spam likelihood score is above a certain threshold, and
| mistakes in rules at system levels can and do happen.
|
| But here's another way you could (ab)use the mail system for
| delivery, provide a mailbox for the client and just allow
| IMAP or POP access and throw the messages into that. The
| client can log in to access and process them (which they
| would likely be automating on their own mailboxes anyway). It
| does mean it's housed at the provider, but it's also pretty
| easy to scale. There's lots of info on how to set up load
| balanced dovecot clusters out there, and even specialized
| middleware modes (dovecot director) to make it work better so
| you can scale it to very very large systems.
| R0b0t1 wrote:
| It's not uncommon to set up daemons that only talk to each
| other for infrastructure monitoring and reporting.
| mlk wrote:
| at that point you'd better off using an atom/rss feed
| kbenson wrote:
| The difference is that using a mail subsystem to handle
| this handles a lot more of the implementation than "use
| an atom/rss feed".
|
| Notably, in choosing to use an atom/rss feed, you need to
| determine what the webserver serving it is, how to
| implement authentication on top of it (is it a
| token/oauth, HTTP auth, param auth, etc), what is the
| underlying data store (SQL/NoSQL, some message system),
| how to scale that system if you expect it to be large and
| span multiple servers and/or datastores (mail systems
| right now deal with hundreds of thousands of users and
| gigabyte plus mailboxes of millions of messages).
|
| Choosing IMAP to deliver this info means there there are
| well worn solutions for all the decisions you need to
| make (including howtos to implement oauth at the server
| level), as well as client level libraries in almost every
| language. Basically, you could decide to use it and not
| have to worry about forging a new path on that system
| basically ever, because there's plenty of people that
| have already implemented it at a larger level and with
| the same features (even if you would be using them to
| slightly different effect), and they've contributed the
| info on how to do it and what the performance
| ramifications are to the public domain.
|
| I'm not _seriously_ advocating for it, but that 's more
| because clients will look at you funny than for any
| technical reason. Technically, it actually has a lot
| going for it. Unfortunately as an industry we fetishize
| the new and bespoke because obviously our own unicorn
| projects are so new and special and will serve so many
| people that some off the shelf solution could never be as
| good....
| sobani wrote:
| I don't think mr throwaway was advocating to use email to
| send the events, only to use SMTP. Email is an entire
| ecosystem, SMTP is only a protocol.
|
| If the distinction is too hard to make: think of it as
| using the 'Simple _Event_ Transfer Protocol ' that just
| happens to use exactly the same protocol as SMTP.
| kbenson wrote:
| > Email is an entire ecosystem, SMTP is only a protocol.
|
| Yeah, but it's a protocol for transferring email. As I
| noted with "you have to be very careful or have a lot of
| control over the receiving server to ensure you actually
| get delivery", you can amstract most of the mail system
| out as long as you ensure you are running the server they
| deliver to, but you would also need to rely on them
| making sure their outgoing server is good for this, which
| probably means dedicating it to this and not running any
| real mail through it (so you avoid outgoing company email
| filters, etc). At that point, both sides are running
| specific bespoke mail servers, which cuts down on the
| usefulness of the solution because of how much setup and
| administration it requires.
|
| It used to be nobody ran incoming and outgoing filtering
| on email, so it was a robust channel for communication
| with retries, and notifications for failed delivery, etc.
| These days it's not exactly that because of all the spam
| mitigations and company compliance and risk mitigations
| that might be in place, etc. In fact, just setting up a
| new mail server and attempting to send to microsoft
| (live/hotmail), yahoo or gmail is extremely hard, because
| they have a high bar for acceptance, and large swaths of
| the easily obtained IP space have already been
| blacklisted from prior spam use so you start at a bad
| reputation and have to work to get it to a level you've
| even be allowed to talk to other by working with all the
| third party (and first party) blacklist maintaining
| entities.
| fouc wrote:
| Yeah it does make more sense to have the IMAP/POP setup
| rather than actively sending out emails through consumer
| level email services like gmail etc where deliverability
| might become a concern.
| dataflow wrote:
| I'm confused, "use SMTP" doesn't even type-check for me.
| Isn't SMTP just a _transfer protocol_? Meaning it defines a
| bunch of commands and gives them meanings (like EHLO and DATA
| and such), just like how HTTP defines commands like GET and
| POST and all that? Isn 't the problem here about e.g. the
| storage & retry logic rather than about the data transfer
| itself? Can't you retry transmission as frequently as you
| like using whatever protocol you like? How does transferring
| the data over SMTP gain you anything compared to HTTP?
| wccrawford wrote:
| "use SMTP" here is a short way of saying "send mail to a
| mail server that will store the requests indefinitely"
| instead of webhooks that are constantly retrying on a
| protocol that was hand-written instead of being baked into
| the whole internet already.
| dataflow wrote:
| > "use SMTP" here is a short way of saying "send mail to
| a mail server that will store the requests indefinitely"
|
| So the suggestion is to use email? That's not how others
| are interpreting it. [1] And it doesn't make sense to me
| either. Emails as they are "baked into the whole internet
| already" are unencrypted with tons of middlemen, and even
| their transport isn't guaranteed to be encrypted. Email
| is also munged and messed with in weird ways, with fun
| stuff like each middleman tacking on their own headers
| and filtering it out based on unknown rules. It also
| introduces a ton of latency and severely prioritizes
| "eventually reaching the destination" over timeliness.
| And more downsides I can't think of off the top of my
| head. That seems like a _really_ poor choice for an event
| delivery mechanism.
|
| [1] https://news.ycombinator.com/item?id=27830705
| erikpukinskis wrote:
| Your suggestion about SMTP is a good one. Disappointing that
| I had to downvote your comment for the ad hominem on us old
| Node developers.
|
| Why you need to insult a whole body of people, rather than
| just make a claim about the technology, I don't know.
| eyelovewe wrote:
| There is a significant faction of node.js who, hmmm how
| shall I say this, I had a js fanatic tell me here in HN
| that you can write device drivers in js, and that the only
| thing stopping js from dominating embedded dev was
| prejudice. Unfortunately, such attitudes color the
| perception of the entire field, and a technology becomes
| shorthand for a certain attitude, here's an example : what
| do you think of when I say Visual Basic developer? The
| individual could be highly competent and work somewhere
| that Visual Basic is the tool, there's surely a core
| competence for VB, I know that there is one for js, the
| frontend web, and certainly node.js has a set of core
| competencies too
| staz wrote:
| so someone told you he was prejudiced against, you
| respond with more prejudice and then you pretend it's his
| attitude that colored your perception. Are you for real?
| deleted-account wrote:
| Programming language choice isn't a protected class, it
| isn't prejudice to say that JS developers are mostly
| dumb.
| Jonnax wrote:
| Prejudice is also an English word that has a meaning
| unrelated to law.
|
| Calling JS developers most dumb is prejudice.
|
| The word's meaning is literally spelled out "pre" +
| "judge".
| deleted-account wrote:
| It's honestly disgusting to see JS developers abuse
| social justice language in this way. You should be
| ashamed. Go make a donation.
| tremon wrote:
| Node developers are old now?
| go_prodev wrote:
| I think that would be a great solution for these types of
| scenarios.
|
| In an enterprise setting it becomes more complex if a 365
| subscription is required, or active directory authentication
| is needed to receive emails. Does someone need to monitor the
| inbox to confirm it's working etc.
|
| But after you mentioned it, I do wish that this was an
| alternative to webhooks that more service providers offered.
| bkrausz wrote:
| When designing something like this as a service, the biggest
| question is what other developers will find easy to use.
| Every cheap host supports inbound HTTP requests, and most web
| developers know how to receive them.
|
| Stripe needs to be usable by both the developers building
| intense, scalable, reliable systems and the people teaching
| themselves to code in a limited context on a limited
| platform.
| chaps wrote:
| I pitched an idea like this years ago to essentially backfill
| one ticketing system to shiny new system that could read an
| email inbox. The idea was that if we dropped an email in that
| inbox with its desired format for each old ticket's updates,
| the new system would do all the necessary inserts and voila.
| They told me no -- not because of any technical reason, but
| because their email infrastructure was required to be audited
| by the SEC, they would have opened themselves up to
| significantly more auditing. Instead, I ended up having to do
| it through painful, painful SQL.
|
| Lesson being, that sometimes there are unexpected reasons why
| a specific piece of technology shouldn't be used.
| darkerside wrote:
| Weren't these all emails already? Weren't you required to
| retain them for the SEC even before falling into this
| specific (hypothetical) inbox?
| user5994461 wrote:
| It's not clear from the message whether the software is
| setting up their own email system, if so it will need to
| be audited and certified, which is a major hassle.
|
| Either way, the auditors and the infrastructure might not
| want to handle an order of magnitude more traffic (API
| usage is really in a different league than occasional
| human email). Expect all emails to have to be stored for
| around a decade.
| wruza wrote:
| SMTP would raise too many questions, from how both
| datacenters tolerate it (spam), to who will manage the
| receiving server itself and certificates on your side, and
| overall security of this setup. For a nodejs developer it's
| really easier to spin up a separate handmade queue process
| rather than managing SMTP-related things. Webhook (for
| runtime) and long-polled /events?since= (for startup) have
| all upsides with little downsides.
| kgwxd wrote:
| What about events that need faster than 1 minute response
| times? Any push notification like system is going to be just
| as error prone. And what about multiple message handlers? And
| what happens when the send fails? Did someone write the code
| to check the inbox for them and handle them? When a send
| fails multiple times, is that logged and is there a system
| for clients to check that log? Message transfer isnt the hard
| problem in this domain.
| vidarh wrote:
| There's nothing about SMTP that dictates response times or
| in any way makes it much slower than HTTP. A non-pipelining
| clients will require a few more network roundtrips if it
| connects and disconnects for every message, that's all.
|
| > Any push notification like system is going to be just as
| error prone.
|
| E-mail servers are built with retry logic and queuing logic
| already. The point is if you need queuing anyway, it offers
| a tried and tested mechanism with a multi-decade history
| and a vast number of interoperable software options. While
| there is now a relatively decent number of queuing
| middleware options, _none of them_ have as many server and
| client options as SMTP.
|
| SMTP isn't the best choice for everything, but it _works_
| (I 've used it that way), it's reliable, and it scales with
| relative ease.
|
| > And what happens when the send fails?
|
| It gets retried. Retries are built in to mail servers.
| That's part of the point.
|
| > And what about multiple message handlers?
|
| What about it? Most SMTP servers provides mechanism for
| plugging in message delivery agents rather than delivering
| to a mailbox, or you let it deliver to a mailbox and pick
| it up from there. Or you plug in whatever routing mechanism
| you want to distribute the messages further. The sheer
| amount of ready-built options here is massive.
|
| > Did someone write the code to check the inbox for them
| and handle them? When a send fails multiple times, is that
| logged and is there a system for clients to check that log?
|
| Pretty much every e-mail server ever written provides a
| mechanism for handling persistent failures, and _many of
| them_ offers heavily configurable ways of doing it. But
| yes, you 'd need to decide on what to do about persistent
| failures. But you need to do that whatever queuing system
| you use.
|
| > Message transfer isnt the hard problem in this domain.
|
| The point of the article is exactly that reliable message
| transfer _is_ the hard problem in this domain.
| alephu5 wrote:
| Another option would be to publish an AMQP endpoint, I'm not
| sure what the security implications of this are though.
| aidenn0 wrote:
| SMTP no longer reliably delivers messages. Try setting up an
| MTA on a Hetzner VPS and see how many messages get through
| vidarh wrote:
| That is only relevant if you require delivery to arbitrary
| endpoints rather than to endpoints explicitly set up to
| process your messages.
| mrtesthah wrote:
| That's not an applicable criticism for SMTP running on a
| private network and/or dedicated set of "mail" submitting
| servers, as in the specific model outlined in the
| grandparent comment.
| alexgartrell wrote:
| I think the Stripe API stuff you did was fine, but you really
| did your best work as a concepts of mathematics TA.
| AtNightWeCode wrote:
| This is a fake post? Never encountered this level of
| incompetence in real life.
| avalanche123 wrote:
| I think this is also getting at the difference between pub/sub
| and state synchronization. While one might think they want the
| former, what they really want is the latter. Get some state and
| receive updates continuously rather than deal with unreliable
| stream of updates
| shadowgovt wrote:
| On HTTP, pub/sub is eventually guaranteed to drop some messages
| because TCP/IP itself is not a guaranteed networking protocol
| (it just makes some promises about failures being uncommon and
| probably detectable).
|
| If what you want is guaranteed state synchronization, pub/sub
| alone can't give it to you.
| rad_gruchalski wrote:
| > If the follower goes down, when it comes back it can page
| through the history at its leisure. There is no queue, nor
| workers on each end trying to pass events along as a bucket
| brigade.
|
| Sure, there is no queue. There is an append only log. Kafka is
| not a queue but an append only log.
|
| I do not like the proposed solution. I do not like it because it
| assumes that I have to maintain the infrastructure to do all the
| distributed logging on my end. As in, most likely I have to
| maintain a Kafka cluster.
|
| There's also one thing glossed over in this article. What if your
| consumer went past certain messages but it mishandled them? You
| can go to the past either.
|
| If your consumer needs ordered delivery, web hooks might not be
| the best solution, indeed. But it might cost you more because I
| need additional infra to provide you with that.
| toomim wrote:
| There's a much better approach than /events or webhooks: add
| synchronization directly into HTTP itself.
|
| The underlying problem is that HTTP is a state _transfer_
| protocol, not a state _synchronization_ protocol. HTTP knows how
| to transfer state between client and server once, but doesn 't
| know how to update the client when the state changes.
|
| When you add a /events resource, or a webhooks system, you're
| trying to bolt state synchronization _onto_ a state transfer
| protocol, and you get a network-layer mismatch. You end up with
| the equivalent of HTTP request /response objects inside of
| existing HTTP request/responses, like you see in /events! You end
| up sending "DELETE" messages within a GET to an /events resource.
| This breaks REST.
|
| A much better approach is to just fix HTTP, and teach it how to
| synchronize! We're doing that in the Braid project
| (https://braid.org) and I encourage anyone in this space to
| consider this approach. It ends up being much simpler to
| implement, more general, and more powerful.
|
| Here's a talk that explains the relationship between
| synchronization and HTTP in more detail:
| https://youtu.be/L3eYmVKTmWM?t=235
| derptron wrote:
| The website seems to be crammed into the left side of my screen
| unnecessarily.
| top_kekeroni_m8 wrote:
| Centering a div is hard!
| hnbad wrote:
| Not really. They could have slapped a `margin: 0 auto` on
| that div they decided to set a `max-width` on and called it
| a day.
| wruza wrote:
| You may send POST /events instead. It also breaks "REST", which
| is just a sort of obsession rather than a requirement here, but
| more importantly it wouldn't break idempotence and proxy
| caching that GET implies.
|
| Edit: from the network point of view, it's either call-back or
| a persistent call-wait/socket, or polling. The exact protocol
| is irrelevant, because it's networking limits and efficiency
| that prevent everyone from having a persistent connection to
| everyone. A persistent connection can't be much better than any
| other persistent connection in that regard, and what happens
| inside is unrelated story. Or am I missing something?
| jonny_eh wrote:
| > just fix HTTP
|
| Oh yes, changing HTTP is so easy.
| toomim wrote:
| HTTP is actually quite malleable, and adding synchronization
| _is_ easy.
|
| You can add it to your own website with a few simple headers,
| and a response that stays open (like SSE) to send multiple
| updates when state changes:
| https://datatracker.ietf.org/doc/html/draft-toomim-
| httpbis-b...
|
| ...and you can get these features for free using off-the-
| shelf polyfill libraries. If you're in Javascript, try
| braidify: https://www.npmjs.com/package/braidify
| mywittyname wrote:
| > To mitigate both of these issues, many developers end up
| buffering webhooks onto a message bus system like Kafka, which
| feels like a cumbersome compromise.
|
| Kafka solves exactly the issue that the author is complaining
| about. This is a safeguard to ensure that data isn't dropped in
| the event of an issue, and provides mechanisms to replay events.
|
| The tradeoff between pushing and polling have been argued since
| forever.
|
| In other news, mechanics who work with bolts often do so with
| ratchets. This is a cumbersome compromise, just give me Torx
| fasteners!
| aunty_helen wrote:
| It's a common writing style as of late, set down a premise and
| solve that premise decisively.
|
| Now, if that premise isn't based in reality, or if it's already
| been solved some other way, discredit it without giving it too
| much air time.
|
| A one liner about kafka being cumbersome and then building your
| own solution, warts and all, doesn't need to exist in the same
| thought if you've made the reader mentally disregard it as a
| possible solution.
| closeparen wrote:
| Are you going to expose your Kafka brokers directly to your
| integration partners? Are they going to use the Kafka client
| library and wire protocol to send you data? That's the thing
| about webhooks, HTTP is universal and if you're comfortable
| exposing _anything_ externally, it's going to be a web service.
| mrkurt wrote:
| We expose the NATs protocol to our users. Exposing non-http
| protocols is fun, sometimes.
| mywittyname wrote:
| I would not expose kafka directly. I would implement this as:
|
| HTTP Endpoint -> Push to message queue (kafka, SQS, etc) ->
| Acknowledge receipt
|
| That's a pretty straight-forward design that's widely used,
| robust, and easy to put together. I've probably done that
| same workflow 100s of times without issue.
|
| As long as you guarantee the message was pushed to the queue
| before acknowledging, that will be fabulously reliable. You
| need to make contingencies for duplicate messages, but that's
| not usually difficult.
| jerf wrote:
| It would if the source was pushing into the Kafka stream
| directly. It doesn't solve the problem of going out of sync if
| my code to push to the Kafka stream is entirely down and I miss
| POSTs.
|
| (And, of course, I don't want Kafka. I want Google PubSub. No,
| wait, I mean SQS. No, wait, I mean I want zeroMQ. No, I
| mean....)
| toomuchtodo wrote:
| You meant Apache Pulsar! :)
| jerf wrote:
| I'm feeling cosmopolitan today. I mean all the things!
| [deleted]
| mywittyname wrote:
| As long as you guarantee delivery to your message queue
| before acknowledging receipt, you should be golden.
|
| Also, swapping out one messaging system for another is
| trivial. Pick the one best suited to the environment you're
| working in, and if that environment changes, changing
| messaging queues is going to be one the easiest transitions
| you'll make.
| l_t wrote:
| Not disagreeing with your point, and I'm sure you already
| know this, I just wanted to point out (for the benefit of
| people that don't have other options) that it is possible to
| build "webhooks" in such a way that you're confident nothing
| is dropped and nothing goes (permanently) out of sync. (At
| least, AFAIK -- correct me if this sounds wrong!)
|
| Conceptually, the important thing is each stage waits to
| "ACK" the message until it's durably persisted. And when the
| message is sent to the next stage, the previous stage _waits
| for an ACK_ before assuming the handoff was successful.
|
| In the case that your application code is down, the other
| party should detect that ("Oh, my webhook request returned a
| 502") and handle it appropriately -- e.g. by pausing their
| webhook queue and retrying the message until it succeeds, or
| putting it on a dead-letter queue, etc. Your app will be "out
| of sync" until it comes back online and the retries succeed,
| but it will eventually end up "in sync."
|
| Of course, the issue with this approach is most webhook
| providers... don't do that (IME). It seems like webhooks are
| often viewed as a "best-effort" thing, where they send the
| HTTP request and if it doesn't work, then whatever. I'd be
| inclined to agree that kind of "throw it over the fence"
| webhook is not great and risks permanent desync. But there
| are situations where an async messaging flow is the right
| decision and believe it or not, it can work! :)
| curryst wrote:
| If you want to be 100% sure that you get all the webhooks,
| the sender could implement an incrementing "webhook ID". If
| the receiver knows the last webhook ID was 53 and the
| sender sends one for 55, you can tell one has been dropped.
| There are some other concerns around that like if 54 has
| been sent but they arrived out of order, or if they arrive
| almost simultaneously. Nothing that isn't solvable afaict
| though.
|
| Of course, then you need a way for the receiver to
| retrigger or view the webhook if one gets missed, which
| starts to look like you have to have a polling endpoint
| anyways, though.
| ThrowawayR2 wrote:
| > " _Of course, the issue with this approach is most
| webhook providers... don 't do that _"
|
| Embedded systems don't do that for webhooks because they
| can't (very little RAM or non-volatile storage) but
| customers clamor for webhooks anyway because it's what
| their web developers know how to use. So inevitably they're
| going to lose data but they're only getting what they asked
| for.
| BasieP wrote:
| We have a system that pushes loads of messages (as in
| thousands a minute) and some consumer insists on using
| there http backend to push the messages to. There system is
| down every once in a while for quite some time. We're using
| an async queueing solution, but you can't keep those
| messages forever. We sometimes have milions of messages for
| them in there queue's, which take up space... If all of our
| consumers had those problems we would have to buy loads of
| storage.. We're simply dropping messages older than x, and
| have an endpoint that they can call to retreive the 'latest
| state of things'. This way when they come back from a
| failure, they simply get the latest state, and then
| continue with updates from our end.. It's far from perfect,
| but it works really well.
|
| I know the goal for most systems is just to be 'up to date'
| Not to get the entire history. So in most cases you don't
| need to stash all the messages, you just need to be able to
| retreive the latest state of stuff...
| atombender wrote:
| This misses the problem explained in the article, which is
| that there are scenarios where events are "acked" but
| things still go wrong because of bugs.
|
| For example, you rolled out code on the receiver side that
| did the wrong thing with each message. Now there's no way
| to replay the old webhooks events in order to reinstate the
| right behaviour; there's no way to ask the producer to send
| them again.
|
| The only way around this is to store a record of every
| received message on the receiver side, too, which the
| article author thinks is an unnecessary burden compared to
| polling.
|
| Personally, I think push is an antipattern in situations
| where data needs to be kept in sync. The state about where
| the consumer is in the stream should be kept at the
| consumer side precisely so it can go back and forth.
| nine_k wrote:
| The question is: who maintains the queue of events, and pays
| for it?
|
| Certainly the event producer is in a better position to
| maintain a queue without missing events, but it also means
| they need to buffer more data in their queue system to
| accommodate for your receiver's downtime
| BasieP wrote:
| this!
| danudey wrote:
| Having helped manage a Kafka cluster, I do not want to run a
| Kafka cluster just so that Microsoft Teams can webhook me
| events now and then.
| Floegipoky wrote:
| Yeah I was scratching my head reading this article; they're
| bending so far backwards to avoid the obvious solution that I
| thought they were gearing up to pitch some competing tech.
|
| > If the sender's queue starts to experience back-pressure,
| webhook events will be delayed, and it may be very difficult
| for you to know that this slippage is occurring
|
| I've never before seen anyone try to argue that properly
| dealing with backpressure is a bad thing. The author's proposed
| model makes this situation even worse. With kafka, consumers
| can continue processing the event stream and you can continue
| to serve reads from your primary datastore. With the author's
| model the event stream lives in your primary datastore, so if
| that starts to lock up the blast radius is much larger.
| alexbouchard wrote:
| Totally, things can get very reliable if you start processing
| webhooks asynchronously. Personally I've found it pretty
| cumbersome and complicated to build the necessary
| infrastructure in the past. I've been building
| https://hookdeck.com as a simpler alternative specifically to
| ingesting incoming webhooks.
| klysm wrote:
| This seems like another instance of the never ending war of push
| vs pull
| fiddlerwoaroof wrote:
| It seems to me that you could build a protocol where normal syncs
| happen through webhooks where each webhook event refers to the
| event id of an immediately previous event. If the system
| receiving notifications doesn't have that event, it makes an API
| request for all events between the latest event it has and the
| one it just received.
| aboodman wrote:
| It the system is moving fast this is still somewhat complicated
| to implement robustly because by the time the "catchup" request
| to t0 returns, more time has passed and more events have
| happened, so you still can't resume consuming the webhooks.
|
| To be correct with such a system you have to be prepared to
| queue the incoming webhook events, do the catchup query, then
| replay the queued events.
| MrStonedOne wrote:
| only if the order of events matters.
| fiddlerwoaroof wrote:
| Yeah, if the events can be handled idempotently and this
| handling properly accounts for time, you don't have to stop
| processing webhooks while filling in the gaps.
| aboodman wrote:
| OK but, it usually matters. If the events represent
| changes to some object, those changes almost always have
| to be handled in order if you are to arrive at the same
| end state as the source.
|
| _edit:_ Also, idempotent is not the right term here.
| Idempotent just would mean the event could be handled by
| the receiver multiple times w /o changing the meaning. If
| you need the events to be applicable out of order, then
| you need them to be commutative. This is a much more
| difficult property to ensure and in practice, I am
| guessing, almost non-existent in deployed webhook APIs.
| fiddlerwoaroof wrote:
| I think that's right re. idempotency. However, I think if
| each webhook is a statement of the state of an object at
| time t (using some monotonic definition of time, like a
| Lamport clock), commutativity is trivial: compare the
| time in the webhook event to the time in your local
| database, and only update your database if the webhook
| event is newer.
| williamdclt wrote:
| That is pretty much what TCP does, except that it doesn't
| "request" missing packets: the client just acknowledges the ID
| of the last packet it was happy with.
|
| It requires a sequence though, so that you know that the client
| can know that packet it just received isn't the one it
| expected, but you could build that with a chain of IDs like
| you're proposing.
| senden9 wrote:
| The Telegram bot API supports webhooks and long running /event
| calls. Because Telegram bots often have no web UI events has also
| the plus point of allowing to block every incoming connection
| with a firewall. Also so DNS name or static IP needed!
|
| This is a huge helper for me personally.
| qubyte wrote:
| One pattern I like to use is to have a regular /events endpoint,
| which works for polling. The endpoint also checks the accept
| header though, and if it sees `text/event-stream` it'll act as an
| EventSource endpoint.
|
| EventSource is also nice because you can add cursors, so if the
| connection drops the source naturally catches up when the
| connection is re-established. While the interface is browser
| oriented, there's no reason not to use it in other contexts.
| tlarkworthy wrote:
| long polling is not good for serverless consumers. webhooks are
| great for compute on demand so we should work towards that (it's
| cheaper coz it's more effecient).
|
| You do not need storage in the producer AND in the consumer, you
| just need a queue in the producer. Yes, even that is annoying,
| but the suggested architecture will still lose data if the long
| poller is down unless there is storage in the producer... so
| nothing significant has really been solved
| kureikain wrote:
| I think with a right a strategy for retry then webhooks are
| great. On my email forwarding app https://hanami.run we offer
| both method to give user access to their email:
|
| 1. webhook with retry up to 7 days. 2. a REST api to fetch all
| data
|
| Sending webhook out properly require lots of effort, especially
| idempotent key concept to avoid duplicate data. And control
| concurency to avoid swarming the webhook endpoint.
|
| So at the end of day, both are require same amount of resources,
| either on the sender side or the receiver side.
| tabbott wrote:
| Zulip's API is built on roughly this design pattern:
|
| * https://zulip.com/api/real-time-events
|
| * https://zulip.com/api/register-queue
|
| * https://zulip.readthedocs.io/en/latest/subsystems/events-sys...
|
| We use this same long-polling based /events API interface for all
| official clients (web, mobile, terminal), our interactive bots
| ecosystem (https://zulip.com/api/running-bots), and many
| integrations (E.g. bridges with IRC/Matrix/etc.).
|
| We also offer webhooks, because some platforms like Heroku or AWS
| Lambda make it much easier to accept incoming HTTP requests than
| do longpolling, but the events system has always felt like a
| nicer programming model.
|
| (Zulip's events system was inspired by separate ~2012
| conversations I had with the Meteor and Quora founders about the
| best way to do live updates in a web application).
| Arathorn wrote:
| Matrix has the same, except we call it /sync these days rather
| than /events, and it long-polls :)
| throwaway290232 wrote:
| The premise of this post is wacky. They're trying to argue for
| how web applications should provide consistency to operations on
| remote systems. You know what that's called? Distributed
| Computing. I don't know if you know this, but Distributed
| Computing Is Hard. You can't solve it with a new interface or
| polling really fast.
|
| Webhooks are perfectly fine for what they're intended, which is
| inconsistent push-based notifications to loosely coupled web
| apps. If you require "consistency", you supplement with polling
| and queues and other junk. If you require _real_ consistency, you
| must use a distributed consensus algorithm.
| MrStonedOne wrote:
| distributed consensus/distributed computing appies for
| mutual/full-duplex/two-way systems, syncing changes one way
| from a 3rd party is not distributed computing, and not
| something you would want to throw a distributed consensus
| algorithm at
| hardwaresofton wrote:
| For developers/engineers who have never seen it, SSE is a nice
| way to get going with streaming (slightly different from long
| polling, it's server-push) easily -- there's no need to jump to
| tools like websocket/gRPC streaming if you don't need to:
|
| https://developer.mozilla.org/en-US/docs/Web/API/Server-sent...
|
| Obviously long polling will work in more places but SSE is great
| if the server side needs to do the pushing.
|
| Also for syncing stripe in particular you might want to check out
| the work done by Supabase (still experimental):
|
| https://github.com/supabase/stripe-sync-engine
| himoacs wrote:
| Ah, I hear that a lot from customers I work with. A true event-
| driven system requires both events and webhooks. You will always
| have apps that only interact via REST so you can't really use
| streaming architecture here but you can make them more real-time
| via webhooks.
|
| The article talks about issues with webhooks such as not being
| reliable if the service goes down and messages are lost. It also
| talks about developers daisy-chaining multiple services together
| to put forward a solution which is not robust.
|
| That's why you need a broker that does event distribution,
| supports multi-protocols (REST, AMQP, MQTT, WebSockets...)
| natively without any proxies and supports Webhooks. You can push
| messages to your REST clients and if they disconnect, the
| messages will pile up in a queue, ready to be consumed when the
| client reconnects.
|
| Solace PubSub+ Broker does all of this. Disclaimer: I work at
| Solace.
| bosky101 wrote:
| i think eventually system of records should be able to directly
| push to well authenticated queues.
|
| we are a data and cpu intensive API - and moving from an client-
| request flow - to pushing to an API to get a pending response,
| then hitting their queue/webhook once the data is ready.
| Eventually i think only-once semantics found in kafka, other
| stream processing will become the norm and better alternative to
| webhooks, with the right security constructs.
|
| giving write access to partners to a queue or even database
| connection to directly push data, may sound suspect today, but be
| common practice in a few years.
| graton wrote:
| Makes me think of how the OpenStack project watches for events to
| Gerrit. They open an SSH connection to the Gerrit server with the
| stream-event command. It then stays connected to the Gerrit
| server and all the events that occur show up in the stream and
| the program can take actions based on the events it sees.
|
| Gerrit stream-events doesn't solve the issue if the connection is
| dropped and events occur while disconnected.
|
| I personally (for my hobbyist use case) would prefer the
| article's /event system over webhooks as webhooks require you
| have a system that is available on the Internet to receive the
| webhook. Where having this /event system would not require that.
| calineczka wrote:
| Can anyone comment on how to do it without race conditions? I
| described the problem here:
| https://github.com/RailsEventStore/rails_event_store/issues/...
| AlfeG wrote:
| At SurveySolutions we ended up with using shared lock during
| events readout from postgres. This solution is not the best,
| but good enough for us.
| https://github.com/surveysolutions/surveysolutions/blob/5bc9...
| calineczka wrote:
| Thanks for answering, I appreciate. I found good
| documentation in https://mariadb.com/kb/en/lock-in-share-
| mode/ but I am not sure how it works in Postgres:
| https://www.postgresql.org/docs/9.1/explicit-locking.html
|
| It says "This mode protects a table against concurrent data
| changes" but it does not elaborate how. Is it similar in
| consequences to what MariaDB describes?
| AlfeG wrote:
| Basically Postgres SHARE lock will ensure that there is no
| active transaction in this table before executing query
| over table. This is a table wide lock and as I said it's
| not a best solution, as it will slow down a bit other
| events producers/consumers.
|
| In our case - active data export process, will slow down a
| bit main application while reading new events.
|
| Best solution would be using external tool to handle
| streams, like Event Store, Kafka or new RabbitMq Streams.
| But we prefer to stick with Postgres.
| adnanh wrote:
| You could use Hookdoo[0] or Hookdeck[1] as a replayable buffer /
| log :-)
|
| [0] hookdoo.com [1] hookdeck.com
| danudey wrote:
| My company has recently switched to Microsoft Teams, where
| unsupported integrations happen via webhooks. For example, if we
| wanted to be able to trigger builds in Jenkins or Gitlab, or
| acknowledge alerts via AlertManager, we'd have to set them up as
| webhooks to the appropriate service.
|
| The problem is that all of those services are internal to our
| network, and aren't accessible from the outside world. We cannot
| set up a webhook to Jenkins because Jenkins does not have a
| publicly accessible URL. We cannot set up a webhook to Gitlab, or
| to Prometheus, or to Sentry, or anything else, because those are
| all internal services.
|
| The only option there would be to create a new, public-facing
| server, set it up with a domain name and SSL certificate, expose
| it to the world, and then give it access to those services -
| which defeats the point of having those services internal and
| secure if we just create a non-internal system and give it access
| to them.
|
| Alternately, we have that new, public-facing server buffer those
| requests and have other services poll them, somehow, so that it
| cannot connect in, but now we're getting into the same situation
| as described in the article.
|
| If there were an API, I could easily create a small daemon that
| would watch for events and dispatch them accordingly, and then
| respond to them as needed; instead, my only option is to build
| some kind of Frankenstein - or to give up entirely, which is the
| more reasonable solution.
|
| Then again, this is Microsoft Teams, where creating an
| application requires an Azure account and jumping through a ton
| of hoops, so they're no stranger to stupid ideas that no one
| wants to deal with.
| oscargrouch wrote:
| I'm finishing a browser based application platform where the
| applications installed expose a RPC api, so in the end all
| applications can call others in the same local(or remote)
| node/s.
|
| The beauty of this is that you also can compose with other
| nodes and for a distributed service by calling the local
| service as a proxy and routing the requests to the other nodes
| of the same api.
|
| It took more time than i've predicted because its also expected
| to deliver UI and most of the 'HTML5' api to native
| applications (instead of Javascript), which is a massive
| platform by now (and the #1 reason why newcomers to browser
| technology cant compete, giving the feature creep tax imposed
| to them).
|
| The idea is also to distribute over a DHT so you can just serve
| your application over torrent without needing to register
| anything..
|
| The only way to get there is by empowering users and developers
| and taking some of the control from the cloud platform giants.
|
| In my point of view the only way to break the browser monopoly
| now is to create a new path forward, a branch.. its not the
| time to follow the rules, its time to break them or else the
| future doesn't look so bright in my opinion..
| bastawhiz wrote:
| > The only option there would be to create a new, public-facing
| server
|
| This is a problem with receiving any inbound data from a third
| party. At least with HTTP, it's pretty trivial to set up a
| robust reverse proxy with nginx.
| orf wrote:
| A small Lambda (or your cloud equivalent) is perfect for this
| pjgalbraith wrote:
| If you are using Teams and use Azure AD then something like
| Azure AD Application Proxy might be a good option
| https://docs.microsoft.com/en-us/azure/active-directory/app-...
| carlosf wrote:
| +1
|
| My company's internal apps use a mix of VPNs and IP fenced
| load balancers. We are migrating to app proxy.
|
| No inbound connections + access based on Azure AD identity
| with conditional access (restrict apps to Intune enabled
| corporate devices) and MFA is an absolute killer.
|
| My only complain is that connectors are not very DevOps
| friendly. Cloudflare Tunnel is much better in this area.
| graton wrote:
| You have the same sort of issue that I do.
|
| You might look into Cloudflare Tunnel (formerly Argo). It is
| free and allows you to poke a hole in your firewall to a
| specific service. If that meets your security requirements.
|
| https://www.cloudflare.com/products/tunnel/
| jffry wrote:
| I don't believe Cloudflare Tunnel is free, the free tier
| pricing page [1] lists Argo Smart Routing at "Starting at $5
| per month" ("Argo includes: Smart Routing, Tunnel, and Tiered
| Caching")
|
| [1] https://www.cloudflare.com/plans/
| graton wrote:
| Well they said free in their blog:
|
| https://blog.cloudflare.com/tunnel-for-everyone/
|
| Also in my personal testing I didn't need to pay.
| aidenn0 wrote:
| Argo Smart Routing is not free, but the tunnel is. The
| tunnel used to be only available under the Argo umbrella,
| but they changed that at some point.
| fidesomnes wrote:
| A similar idea from a hn member https://tailscale.com/
| ec109685 wrote:
| Really good point that corporate firewalls can trip you up.
| With slack it was so much easier to call into their events API
| than receive an outgoing webhook for precisely this reason.
|
| The downside was that the event api required a huge amount of
| scope, so if you weren't careful and were compromised, someone
| could use that token to scrape all messages in the system.
|
| Slack recently added socket mode for precisely this reason:
| https://api.slack.com/apis/connections/socket
| BeefWellington wrote:
| > The only option there would be to create a new, public-facing
| server, set it up with a domain name and SSL certificate,
| expose it to the world, and then give it access to those
| services - which defeats the point of having those services
| internal and secure if we just create a non-internal system and
| give it access to them.
|
| That's not the _only_ solution -- you could also develop a bot
| that will do those specific things.
|
| In the days of yore I know of at least three companies that
| were using IRC bots to similar effect long before webhooks ever
| existed.
|
| Because of that prior experience, this is how I currently
| manage a similar set of problems, albeit not on Teams in my
| current role.
| iamtheworstdev wrote:
| check out https://smee.io/
| historyloop wrote:
| I tried skimming through the article to attempt and derive the
| missing elevator pitch, but I saw them reimplement precisely the
| system they maligned at the start (push notifications with
| polling).
|
| Did anyone here understand what their value proposition even is?
| aboodman wrote:
| Did you check the website? The value prop is incredibly clear:
| they give you a Postgres database you can query with SQL rather
| than having to either (a) learn and code to a custom API or (b)
| setup your own sync system to get that Postgres db.
| historyloop wrote:
| I'm talking about the elevator pitch of their blog post, not
| of their company. Which are actually two distinct things
| (they're not describing their service in this blog post).
| leowoo91 wrote:
| It just depends which side you would like to put the
| responsibility on. Webhooks is mostly for owning it at source.
| sergiomattei wrote:
| Building polling infrastructure is substantially more complex
| than setting up a GET endpoint to handle webhooks.
|
| If you need strict consistency guarantees, sure. Otherwise, don't
| piss off your API consumers, webhooks work just fine.
| toomuchtodo wrote:
| > If you need strict consistency guarantees, sure. Otherwise,
| don't piss off your API consumers, webhooks work just fine.
|
| Ahh, but all problems are queues. What happens if your webhook
| destination is down and your source system sending queue backs
| up? What happens if your destination endpoint is up, providing
| 200s to requests, but throwing away the data quietly due to a
| mistake during a rollout? Or your webhook source quickly ramps
| to a volume that is effectively a DoS attack? (These are all
| problems I encountered in a role at a low code/no code
| product).
|
| I strongly endorse others in this thread who indicate long
| polling events is a suitable pattern. You want the best worlds
| of data durability and consistency through polling
| functionality, but also as close to real time event firing as
| possible.
|
| Of course, some APIs you have no control over, and are stuck
| building robust, chatty polling infra to support because your
| (paying) users demand access to those APIs. Such is the schlep.
| superbaconman wrote:
| Why do I need two different ways of retrieving events?
| Wouldn't the data also be available via the usual rest api?
| If events fail on the consumer end it's their responsibility
| to resync. Presumably code was written for an initial sync
| anyway no?
| crooked-v wrote:
| > Building polling infrastructure is substantially more complex
|
| That really depends on the setup you're using. There are plenty
| of server platforms out there where setting up a cron is no
| more complicated than making a GET route handler.
| sergiomattei wrote:
| I'm looking at it from the perspective of a web dev. If
| you're already building a web application, it is _almost
| always_ more work to do the polling setup.
|
| In Python, it's either a cron script or Celery beat. In PHP,
| usually a cron script.
|
| That all means more processes running outside of your web
| framework, more stuff to deal with, more complexity. Now you
| have to manage e.g. running a celery process, managing
| crontabs wherever you deploy...
|
| Even in languages like Elixir where long-running scheduled
| processes are cheap and native, you still have to write a
| GenServer in comparison to just using your web framework.
|
| I'm fine with the approach in the article. I actually like
| that they're giving the option for multiple ways of getting
| those events. Just please, please don't make /events the only
| option...
| justsomeuser wrote:
| I actually have the opposite opinion. I'm an experienced
| web developer. Setting up polling is easier for me than
| setting up an HTTP endpoint.
|
| I typically use a language with an event loop
| (async/await), so there is always something like
| `setInterval(poll, 500)`. All that code needs is a
| connection to the internet. If the server is down for 24
| hours I can start it up and it'll read the missed events. I
| can set up 10 dev environments with just an API key
| difference in the config. I can batch apply events in a
| single database transaction, ensuring consistency.
|
| But with a webhook, I need to ensure that my server can
| accept incoming connections, the external API knows that
| location. Each dev env needs to replicate this public HTTP
| server set up. I need to monitor the uptime of the server
| closely as missing events or erroring on a subset of events
| could leave my database in an inconsistent state.
| masklinn wrote:
| > There are plenty of server platforms out there where
| setting up a cron is no more complicated than making a GET
| route handler.
|
| Or way simpler if you're not building a web application in
| the first place.
| shadowgovt wrote:
| It's less about cost of building and more about cost of
| running.
|
| Polling is chatty when updates are infrequent.
| cultofmetatron wrote:
| I love webhooks!
|
| I recently had to work with an api to integrate a card reader
| terminal. The system was intended to work with an on premise pos
| so it was oriented around events sent back and forth over a
| websocket. Only one was allowed per restaurant.
|
| In the end, I had to build a distributed cluster that managed a
| pool of processes that each spun up a websocket connection and
| relayed it to our pubsub system. Luckily I built out system in
| elixir so it was pretty easy to spin up hoarde but this would
| have been extremely difficult in any other language.
|
| By contrast, a webhook is easy to scale from ANY language. Just
| make a http endpoint. Those are way easier to scale than a
| stateful process. Not being able to replay a webhook event is an
| issue with the publisher of the webhook. not a consumption issue.
| justsomeuser wrote:
| The Stripe `/events` endpoint is kind of like a Stripe-internal
| webhook with a cache of 30 days.
|
| One issue is that webhooks and HTTP clients can be pinned to a
| version, but the events listed at `/events` are whatever the
| Stripe account default version was at the time of the event
| creation.
|
| So all of your code clients polling `/events` needs to ensure it
| can handle many different versions.
|
| Another issue is that child lists are limited to 10 items, which
| means that you need to do a direct download to get list items >
| 10. This means the event list is lossy as items > 10 are never
| contained in the event stream.
|
| Stripe feature request:
|
| - An option to include all list items.
|
| - `/events` that can be version-pinned / contains events with the
| same API version.
| bno1 wrote:
| A pattern that I like to use in toy projects is to have an events
| endpoint and just send the newest event cursor through a
| notification channel (e.g. webhooks) when a new event occurs. You
| get new event quickly, you don't have to keep connections open
| for long-polling, and you can keep polling with a low period in
| case the event channel stops working for some reason.
| mbrevda1 wrote:
| Are events and webhooks mutually exclusive? How about a
| combination of both: events for consuming at leisure, webhooks
| for notification of new events. This allows instant notification
| of new events but allows for the benefits outlined in the
| article.
| alexbouchard wrote:
| This is the way to go and I'd love to see more API's with
| robust events endpoint for polling & reconciliation. Deletes
| are especially hard to reconcile with many APIs since they
| aren't queryable and you need to instance check if every ID
| still exist. Shopify I'm looking at you.
| coldacid wrote:
| I think that's what the author was getting at, after reading
| through the whole article. The idea isn't to get rid of
| webhooks, but provide an endpoint that can be used when
| webhooks won't necessarily work.
| saurik wrote:
| Yeah... I'd go so far as to argue that this is the only
| architecture that should even ever be considered, as only
| having one half of the solution is clearly wrong.
| snarkypixel wrote:
| Very similar to how I built my previous application.
|
| 1) /events for the source of truth (I.e. cursor-based logs) 2)
| websockets for "nice to have" real-time updates as a way to
| hint the clients to refetch what's new
| sb8244 wrote:
| What about supporting fast lookup of the event endpoint, so it
| can be queried more frequently?
|
| I think that a combo of webhooks / events is nice, but "what
| scope do we cut?" is an important question. Unfortunately, it
| feels like the events part is cut, when I'd argue that events
| is significantly more important.
|
| Webhooks are flashier from a PM perspective because they are
| perceived as more real-time, but polling is just as good in
| practice.
|
| Polling is also completely in your control, you will get an
| event within X seconds of it going live. That isn't true for
| webhooks, where a vendor may have delays on their outbound
| pipeline.
| jacobr1 wrote:
| The article advocate for long-polling
| pak9rabid wrote:
| Couldn't keeping a request open indefinitely open the
| system up to the potential of DoS attacks though? Correct
| me if I'm wrong, but isn't it kind of expensive to keep
| HTTP requests open for an indeterminate amount of time,
| especially if the system in question is servicing many of
| these requests concurrently?
| sb8244 wrote:
| Yea, you're right. I am reading the advocacy as "if you
| need real-time, then support long-polling."
|
| I see the value in this, but I actually disagree with the
| article in terms of that being the best solution. Long-
| polling is significantly different than polling with a
| cursor offset and returning data, so you wouldn't shoe-horn
| that into an existing endpoint.
| shvedsky wrote:
| Yes to the combination of both. I worked on architecture and
| was responsible for large-scale systems at Google. Reliable
| giant-scale systems do both event subscription and polling,
| often at the same time, with idempotency guarantees.
| j_san wrote:
| Sorry if I'm daft, could you/someone explain why one would
| want to use both at the same time for the same system?
|
| One thing that makes sense: if you go down use polling so you
| can work at your own pace. But this isn't really at the same
| time. When/why does it make sense to do both simultaneously?
| shvedsky wrote:
| There is an inherent speed / reliability tradeoff that is
| extremely difficult to solve inside one message bus. When
| you get to truly large systems with a lot of nines of
| reliability, it starts to make sense to use two systems:
|
| 1. Fast system that delivers messages very quickly but is
| not always partition-tolerant or available 2. Slower,
| partition tolerant system with high availability but also
| higher latency (i.e. a database)
|
| The author goes through this in the very first section.
| Webhook events will eventually start getting lost often
| enough for the developer to think about a backup mechanism.
|
| Long-polling works if you have a lot of memory on your
| database frontend. Most shared databases want none of your
| long-running requests to occupy their memory which is
| better used for caches.
|
| Even if your message bus has the ability to store and re-
| deliver events, you might want to limit this ability (by
| assigning a low TTL). Consider that the consumer
| microservice enters and recovers from an outage. In the
| meantime, the producer's events will accummulate in the
| message service. At the same time, the consumer often
| doesn't need to consume each individual event but rather
| some "end state" of some entity or a document. If all lost
| events were to get re-delivered, the consumers wouldn't be
| able to handle them, and would enter an outage again. This
| is where deliberately decreasing the reliability of the
| message bus and rely on polling would automatically recover
| the service.
|
| There are other reasons, of course. The author is
| absolutely correct in their statement, though: whenever a
| system is implemented using hooks / messages, its
| developers _always_ end up supplementing it with polling.
| kissgyorgy wrote:
| What's the point of implementing webhooks once you implemented
| long polling for the /events endpoint?
| luuio wrote:
| I don't think the original comment meant long polling (i.e.
| keeping the connection alive), they meant periodically call
| the endpoint to check for events.
| cwyers wrote:
| The article advocates for long polling of endpoints.
| mbrevda1 wrote:
| I'd argue against long/persistent polling. Webhooks allows
| for zero resource usage until a message needs to be
| delivered.
| zimpenfish wrote:
| > Webhooks allows for zero resource usage until a message
| needs to be delivered.
|
| Doesn't that only work in the case where the server treats
| each webhook delivery as ephemeral? If you're keeping a
| queue to allow reliable / repeatable delivery, that's
| definitely not "zero resource usage", right?
| kissgyorgy wrote:
| On the sender side, sure. On the receiver side? You have to
| have a service listening 0-24.
| bullen wrote:
| I agree that long poll is better than websockets, but even better
| than long poll is comet stream = the long connection sends chunks
| when events occur.
| polote wrote:
| The article is interesting but is misleading. What the article
| really says, is that if in average you receive a webhook every
| second or faster, it is better to poll every second an /events
| endpoint.
|
| Well the best advantage of Webhooks over polling is that you
| receive events straightaway no matter the volume of events. If
| you already know the volume and the volume is high, of course
| polling is going to be better for everyone.
| fiznool wrote:
| One of the nice things about a pull-based model (polling) is that
| you are in control of throughput. Need to process events more
| slowly? Increase the polling interval. It's impossible to achieve
| the same thing with push-based (webhooks), you are at the mercy
| of the producer's rate of webhook delivery. I had this issue a
| few years ago with a queue-as-a-service that sent jobs via
| webhook - the queue would intermittently drain extremely quickly,
| sending thousands of requests per second, which totally
| overwhelmed our poor single Heroku dyno.
|
| One big issue with the pull-based model though is with
| concurrency. If you have multiple workers polling an API endpoint
| for new data, you need to synchronise the 'last seen' ID or
| timestamp across all workers. Otherwise, worker A and worker B
| might pull the same data and you could end up with duplicates.
| There's no silver bullet here, either model requires work to
| harden against edge cases.
| pmlnr wrote:
| Basically: RSS for anything.
| atian wrote:
| > One idea for Stripe and other API platforms: support long-
| polling!
|
| It's great that we've went full circle. But make no mistake that
| this only means one thing: that servers are cheaper than ever. We
| can now afford to entertain previously extravagant ideas.
| Anunayj wrote:
| I've have had a couple issues implementing Long Polling in the
| past. At times I've had the firewall or reverse proxy drop
| client connections if it detects no data transfer. Which meant
| I had to timeout all the requests every 30 secons or so. Make a
| new request before the other one ends, and it all becomes
| messy.
|
| At this point, its honestly just easier to just have a
| websocket + events endpoint with a cusor both.
| nitwit005 wrote:
| I wouldn't recommend long polling for this reason. There are
| also some security products that have trouble with what
| appears to be a long file download.
|
| Websockets or server sent events at least signal to
| intermediaries that a longer term connection will be open.
| [deleted]
| gwbas1c wrote:
| But Long-polling doesn't scale in certain situations where you
| can have infinite webhooks.
|
| I'm sure many people reading this have many idle GitHub repos
| set up with webhooks into some kind of build server. The repo
| might see no more than one commit a week.
|
| It makes absolutely no sense for GitHub to long-poll (or
| websocket) all of these build servers.
|
| (Now what would make sense is for /events to support a way to
| flip over to a webhook when it's idle. IE, long-poll for a
| minute, then the next request sends a URL for a 1-time webhook
| called on the next event.)
| simonw wrote:
| Long polling is a lot easier to support now than it was a few
| years ago, thanks to the wide availability of async server
| frameworks - Node.js, Python ASGI etc - which make supporting
| thousands of simultaneous long-polling connections with a
| single server much less expensive.
| [deleted]
| maerF0x0 wrote:
| also the OP seems to ignore all the issues with webhooks that
| longpolls suffers or is worse at. eg: you'll lose all those tcp
| connections if your service is down -- one of the main
| complaints about webhooks...
| coldacid wrote:
| Which, if the parts discussed _before_ long-polling was
| mentioned were implemented, wouldn 't be an issue since you'd
| be starting from the first event following your last-recieved
| cursor anyway.
| historyloop wrote:
| > It's great that we've went full circle. But make no mistake
| that this only means one thing: that servers are cheaper than
| ever. We can now afford to entertain previously extravagant
| ideas.
|
| We've not come full circle. This is just one blog saying "how
| about long-polling". While also ignoring that since then we've
| gained web sockets, HTTP/2 and HTTP/3, each of which make long-
| polling pointless in three different ways.
| MrStonedOne wrote:
| >While also ignoring that since then we've gained web
| sockets, HTTP/2 and HTTP/3, each of which make long-polling
| pointless in three different ways.
|
| None of those have strong support across language frameworks
| for using them as a _client_ from a server context.
| Especially http /3.
| historyloop wrote:
| What language exactly you use that can't do web sockets,
| which is a thin layer over plain TCP sockets?
| BrianOnHN wrote:
| It's an incentives issue. Webhooks or for when the service
| provider is aiming for the bare-minimum functionality with the
| least overhead. If they have the incentive to maximize your
| consumption of the data, then they should offer both, and may
| even overnight a physical copy.
| xg15 wrote:
| What I don't quite get: This is all about server-to-server
| communication anyway, so no browsers or web platform limitations
| are involved. So why bother with HTTP at all and then add
| complexity to implement real-time event streaming on top of it?
|
| Why not simply offer e.g. a public STOMP endpoint?
| keeganj wrote:
| For infrequent events, why not both? A webhook to push the event
| and a full list at /events?
| XorNot wrote:
| This is a hill I find myself frequently fighting (and losing on):
| webhooks are terrible to maintain, because they start from the
| premise "this never breaks" and thats about where development in
| an organization stops.
|
| The only event API I ever want is notifications there's new data,
| and then an interface by which I can query all new data which has
| arrived by some sort of index marker - because this is
| fundamentally reliable. It means whatever happens to my system, I
| can reliably recover missed events, skipped events, or rebuild
| from previous events.
|
| And this is in fact exactly how something like Kafka actually
| works! Complete with first-class support for compacting queues to
| produce valid "summarized" starting points.
|
| Any streaming system essentially should never start as a
| streaming system - it should start as a slow-path pull-based
| system, and have a fast-path push system added on top of it if
| needed - because then you've built your recovery path already,
| rather then what happens way too often which is just "oh yeah,
| we'll develop that when it breaks".
| sroussey wrote:
| I agree. A simple ping with the latest ID (which is option for
| you to use to get events from last ID to newest ID). The go get
| the events, which is a likely reusing code. Polling is crap.
|
| Extra points for being able to set something like 1s between
| pings (now you see why I like the option ID for a range).
| delusional wrote:
| > Any streaming system essentially should never start as a
| streaming system - it should start as a slow-path pull-based
| system, and have a fast-path push system added on top of it if
| needed - because then you've built your recovery path already,
| rather then what happens way too often which is just "oh yeah,
| we'll develop that when it breaks".
|
| I think this is a quite interesting and important point. When
| we talk about "doing the simple thing first" too often we end
| building something that is technically simple but fickle. The
| trick to making the simple thing reliable is to figure out
| which part is the slow-path (or failure mode), and then only
| building that. Unfortunately, it often means out result ends up
| technically "boring" since all the interesting optimizations
| are what we cut out, but I think that's worth it if the end
| result is a more useful product.
|
| It's something I've been working with and thinking about for a
| while. I think it applies to a way broader scope than this
| discussion.
| rattray wrote:
| (I worked on the same team as bkrausz, elsewhere on this
| thread, albeit not concurrently).
|
| Yes, this is pretty much the right thing to do. It can be a bit
| more work for the API consumer, partly because they need to
| track state of their last-read ID, and there's more moving
| parts.
|
| If you're building a webhhook+events system like Stripe's, you
| might consider adding an option for a mostly-empty webhook
| body, which can speed things up in this use-case, but still
| allows "the easy way" of just processing the event from within
| the webhook body.
|
| (For readers thinking of implementing this, note that "query
| for new data" means hitting a dedicated /events api, not
| individual tables, which might have unpleasant load/performance
| consequences).
| paxys wrote:
| There are lots of reasons to want to immediately respond to an
| external event besides building an eventually consistent data
| syncing system. Polling an API endpoint works fine for the latter
| case, but not much else.
|
| A good platform should offer both of these and more (for example
| Slack does webhooks, REST endpoint, websocket-based streaming and
| bulk exports), and let the client pick what they want based on
| their use case.
| benlivengood wrote:
| Long-polling is the way to immediately retrieve events. It's
| more efficient and lower latency than waiting for a sender to
| initiate a TCP and TLS handshake.
| sk5t wrote:
| If the webhook events are coming at some sort of a brisk
| pace, the sender well may be able to reuse an already-open
| connection. And if they're rather infrequent, is the
| efficiency or latency likely to be a significant concern?
| lmm wrote:
| Yes, it is - latency-sensitive but infrequent events are an
| extremely common use case.
| sk5t wrote:
| In the general sense, yes, but your assertion rings false
| in my opinion when the situation presents only a choice
| between webhook or long polling.
| lmm wrote:
| I don't understand your statement? Rare but latency-
| sensitive events are a very common use case for webhooks
| or long polling.
| IshKebab wrote:
| If you're using HTTP use websockets or server-sent events,
| not long polling. Long polling is obsolete.
| gremlinsinc wrote:
| Websockets can cause issues especially if you're not
| closing sockets properly, or have too much activity on a
| small server etc... Livewire for instance accounts for this
| by just polling every 2 seconds for changes, this is much
| more performant than keeping 10000 sockets open if people
| leave open the page/app but don't actually do anything...
|
| Straight long-polling should be avoided, but intermittent
| polling is a good solution for performance when you don't
| want to use all your socket bandwidth.
| azinman2 wrote:
| My understanding is that long polling is the thing that
| will reliably work at scale. Perhaps this changed in the
| past few years, but I've asked various companies like
| PubNub why they only use long polling and the answer was
| that there are too many incompatibilities out there in the
| wild for anything but that.
| IshKebab wrote:
| Server-Sent Events are very reliable. What you might be
| thinking of is the fact that you probably shouldn't rely
| _just_ on server push. But that doesn 't mean you should
| use long polling.
|
| You should use normal short polling and Server-Sent
| Events.
|
| Also it makes no sense to say long polling is more
| reliable than SSE, because SSE is essentially a non-hacky
| implementation of long polling.
| [deleted]
| hakunin wrote:
| Someone has to maintain an always-running listener for
| `/events`. If a server does that, and triggers client calls,
| we call that webhooks. If a client does that, and triggers
| internal functions, it's what the op describes. I think that
| for APIs, `/events` should indeed be the fundamental feature,
| and "webhooks" should be a nice-to-have service on top of
| `/events`, for those who don't want to maintain a local
| subscriber.
| mikepurvis wrote:
| One nice benefit of long polling is the built in catch-up-
| after-a-break functionality: When the client initiates the
| poll, it tells the server the state it knows about
| (timestamp, sequence number, hash, whatever), and the server
| either replies right away if it's different, or waits and
| replies _once_ it 's different.
|
| With webhooks, as in the article, you only get state changes;
| you need some separate mechanism to achieve (or recover) the
| initial state.
| tshaddox wrote:
| That's true, although it's also true of any `/events`
| endpoint that doesn't go back to the beginning of time.
| Stripe's endpoint only goes back 30 days, so you still need
| to solve for the initial state unless you have launch all
| of your desired functionality at the very beginning of your
| Stripe account!
| mikepurvis wrote:
| Hopefully if it's a system like payments where you not
| only need to know state, you _also_ need to know the time
| and nature of all transitions, there 's a way to query
| all of that information.
|
| I'm thinking of simpler situations like my source host's
| CI spinner that seems to get stuck all the time due to
| missing the ping back from Jenkins about build statuses.
| In that case it really would be fine to always just say
| "I think the state is X, please answer me now or in the
| future whenever the state is other than X." I don't care
| about anything other than an up to date sync.
| andrewstuart2 wrote:
| A persistent connection has a cost. Your statement may be
| true in some circumstances but definitely not all. Namely,
| for infrequent events it is much more efficient to be
| notified than to be asking nonstop. Sure, the latency is
| lowest if the connection is already established, but for
| efficiency the answer is not cut and dry but is rather a
| tradeoff decision based on the expected patterns.
| dnautics wrote:
| Exactly. Perhaps an event happens once or twice-ish a day
| per customer, and never on the weekends.
| nmcfarl wrote:
| I've got an API where an event happens once a month (+/-
| 2 weeks) for a large percentage of our customers.
| eptcyka wrote:
| Also, there's the case of ISP's just dropping idle TCP
| connections. It can also take a while to determine that a
| TCP connection is broken.
| runeks wrote:
| What's the issue with that? This will be discovered as
| soon as the endpoint tries to send an event, right? At
| which point the client will see that the connection has
| been closed, reconnect, and receive the event.
| kelnos wrote:
| No, the _server_ will try to send an event, and the
| _server_ will notice the connection has dropped. The
| client will still have no idea until some sort of timeout
| is reached, as the client will usually not be sending any
| data over the connection, as the connection 's sole
| purpose is for the server to send events to the client.
|
| A way to fix this is to use an application-level
| keepalive (TCP keepalives are generally useless), but
| then that increases the load on the server and adds a
| scaling burden.
|
| Meanwhile, unless the event stream is stateful (more
| overhead!), the client has lost all events since the
| connection has dropped, and the client can't even be sure
| _when_ the connection _actually_ dropped.
|
| With webhooks, assuming the callback sending service has
| a generous retry policy, and the customer's receiving
| service does not return 200 unless the webhook has been
| completely processed, or persisted to storage, you won't
| lose events.
|
| I've been at Twilio for the past 10 years. We recently
| started offering an event stream service (that customers
| had been requesting for some time), but it's complicated
| to get right (on both the server and client side) and
| difficult to scale, and, frankly, webhooks have worked
| fine for most customers for a very long time.
| rad_gruchalski wrote:
| > No, the server will try to send an event, and the
| server will notice the connection has dropped. The client
| will still have no idea until some sort of timeout is
| reached, as the client will usually not be sending any
| data over the connection, as the connection's sole
| purpose is for the server to send events to the client.
|
| Exactly why mqtt has the ping packet for the client.
| wruza wrote:
| Long-polling is usually configured to reset at the both
| sides after a timeout preferred by the client-side
| (/events?t=30), long before any network effects kick in,
| e.g. 10-30 seconds. A client then simply spams requests
| in a loop, backing off only at http errors. If you have
| some crazy firewall in between, just set "t"
| appropriately.
| [deleted]
| fragmede wrote:
| yeah. a(n improperly configured) firewall is going to
| start dropping packets if it thinks a connection is idle
| for too long, so the system never sees an RST and think
| the connection's been terminated.
| jandrese wrote:
| Why for the love of God does the firewall not send the
| RST when it drops the connection?
| lmz wrote:
| Because what usually happens is the connection is just
| forgotten from the NAT table. Both sides still see it as
| connected but the middle box will no longer forward any
| packets.
| jandrese wrote:
| It doesn't just "fall off" the NAT table. Some process in
| the firewall chose that entry in the NAT table to drop at
| that moment. It could use the entries from that NAT table
| to construct RST packets to both sides of the connection.
| This should be easy and obvious.
| foxhill wrote:
| > A persistent connection has a cost.
|
| are you sure? specifically, are you sure a persistent
| connection has _more_ of a cost than repeatedly re-
| establishing a connection & TLS, etc.?
|
| in terms of energy costs alone, DNS resolution,
| establishing routes, generating cryptographic session keys,
| etc. it's definitely not as cheap.
|
| in terms of today's computation power, the "memory" costs
| of maintaining a connection are minuscule, and the
| performance "penalties" are negligible.
|
| example: lets say you have 50k event subscribers. if
| nothing happens, then, aside from a few TCP keepalives
| (which are not strictly speaking required, and can happen
| very infrequently), no traffic moves. if instead you have
| polling once every second, then that's nearly ~13-14
| connections a second, each one with at least 4 round trips
| of traffic. that's a measurable amount of load.
| FpUser wrote:
| I have the same thing in my backend. The backend has main
| business server that among the other thing has an API endpoint
| one can query for a list of events in date-from - date-to
| fashion. I dismissed the idea of webhooks right at design stage
| as to me it looked like a minefield choke full of potential
| problems.
| cryptonector wrote:
| YES PLEASE. Give us a GET on /events with support for hanging
| GETs.
|
| A hanging GET is where you get chunked encoding (which is the
| only option in HTTP/2 anyways) and possibly never-ending stream.
| I've implemented a (proprietary, for now) "tail -f" over HTTP
| that does this when Range: bytes=0- (i.e., end offset not
| specified), completing the transfer (i.e., final, empty chunk
| sent) only whenever the file is removed or renamed away.
|
| You'll want to add a heartbeat to any hanging GET /events.
| cryptonector wrote:
| My little tail-f-over-http server is for plain files, but this
| scheme works for anything. Of course, if you can output an
| indefinite stream (e.g., PG NOTIFYs, build logs, etc.) you can
| redirect that to a file and then serve up that file.
| qwertox wrote:
| I definitely don't want to see long polling. In that case I'd
| prefer a combination of /events and websockets, where websockets
| can push (or pass via a GET param) the last read event from
| /events to notify the server which is the last known event.
| airstrike wrote:
| TFA addresses this by suggesting long-polling as an option
| rather than the _only_ way to request ` /events`
|
| _> In our integration with Stripe, it would be neat if we
| could request /events with a parameter indicating we wanted to
| long-poll. Given the cursor we send, if there were new events
| Stripe would return those immediately. But if there wasn't,
| Stripe could hold the request open until new events were
| created. When the request completes, we simply re-open it and
| repeat the cycle. This would not only mean we could get events
| as fast as possible, but would also reduce overall network
| traffic._
| charles_f wrote:
| What about websockets, which implement a server -> client
| eventing channel without having to create a construct like a
| polling mechanism?
| fart32 wrote:
| Long polling doesn't scale very well. Webhook/websocket and
| events endpoint combined sounds like the sweetspot to me.
| raksoras wrote:
| One of the complexity of the polling approach on a consumer side
| is having a long running poller. This is trivial to do in Java
| apps - start a polling thread - but not so straight forward in
| case of PHP apps, for example. In that case you'd have to setup a
| cron job or a separate polling script under some sort of process
| supervisor like systemd to poll periodically/continuously.
|
| I wonder if the two approaches could be combined to simplify
| things for consumer apps at the cost of slightly more complexity
| on the producer side? Instead of POSTing the actual event data to
| webhook, the producer just uses consumer's webhook to "poke" it -
| to tell the consumer app "hey, you have new events waiting for
| you". On receiving the poke the consumer endpoint handler/PHP
| script can just turn around and do a GET to "/event" with
| anything > last downloaded event id query. That way you don't
| have to support long polling on the producer's servers and it's
| not a big problem if consumer misses couple of webhook "pokes".
| The next time it does receive a webhook "poke" successfully, it
| will download all the events and be all caught up. If real time
| notifications are not strictly required then producer side can
| even run the webhook dispatching code on a scheduled basis to
| coalesce multiple events in a single "poke" to a consumer to be
| more efficient, if desired.
| candiddevmike wrote:
| Why wouldn't service providers want to offer long polling? It
| seems a lot easier to build (no webhook registration backend) and
| no retry logic/SLAs outside of your control. SSE seems so much
| simpler.
| judge2020 wrote:
| Keeping those connections open probably isn't super cheap and
| complicates deploying rolling updates - you'd need to kill all
| connections when you update and that would require some sort of
| RPC (so that you only kill those existing connections after
| they're done sending in-flight data).
| Nullabillity wrote:
| Why bother? Just kill them and let the clients reconnect on
| their own. They'll have to handle that case anyway...
| judge2020 wrote:
| With polling you have to make sure you finish sending data
| for the current in-flight event. If you don't then the
| client doesn't receive that event, unless you also send
| them events from the past 30 seconds on first poll.
| Nullabillity wrote:
| Don't delete the events from your queue until you receive
| a client ack?
| thrower123 wrote:
| With some of the load-balancers and gateways and things in that
| space I've had to use, long-polling doesn't work at all because
| of short timeouts.
| paxys wrote:
| > It seems a lot easier to build
|
| Quite the opposite. HTTP servers and clients are essentially a
| solved problem. Massive scale-out, load balancing, retries,
| authentication, authorization, rolling deploys etc. can all be
| done out of the box by a hundred different providers. Anything
| to do with maintaining a large number of open TCP connections
| is still a massive pain on the server side.
| handrous wrote:
| Yep. Consider: you can build a damn reliable & resilient
| Webhook handler out of a few lines of PHP or Lua, ready to
| accept a fairly heavy load, zero dependencies, and only
| default-available packages for most any distro or BSD
| (anything where nginx or Apache2 with standard modules is
| available by default) and without tweaking the config at all.
| You can be live in hours, or even inside a single hour if you
| want to cowboy it up pretty hard, and despite not taking a
| lot of care, the webhook-handling part of your service
| probably won't get you woken up at night with a
| everything's-on-fire support call (what it does with the data
| might, of course). Logging? Trivial and standard. Service
| management? It's the OS' default service definition for a web
| server daemon, and that's it. Config and deployment? So tiny
| it'd be nearly no work to document it in a run-once shell
| script, if you don't have anything fancier at hand.
| Operationally, it doesn't get much simpler. "Is it working?"
| checks? You can test it with curl, from any address that's
| able & allowed to talk to it.
|
| With long-polling, now you're managing a custom daemon,
| basically. That's a big step down in reliability-by-default,
| and a bunch more work to do it right.
|
| In either case, you'll be looking at more work if you want to
| check any kind of log on the other end for missed messages,
| but that looks pretty similar for either, and not all systems
| need that level of accuracy (and if they do, they probably
| need _even more_ and this whole thing is Doing It Wrong)
| skybrian wrote:
| If the webhook just triggers a download from /events then both
| can be made idempotent. If you miss an event then you can get it
| later.
| abnercoimbre wrote:
| I like having a one-off command-line app triggered by a
| webhook. If you miss an event, invoke the app manually. If you
| pass it the same event twice it won't matter (idempotency!)
| tlarkworthy wrote:
| This article is premised on the incorrect strawman that webhooks
| are complicated _because_ the consumer has an extra persisted
| message bus.
|
| But if the producer retries and the consumer does not respond
| with 200 until it has processed the message, no consumer side
| message queue is needed, the consumer can rely on the producer to
| reach at-least-once delivery.
|
| In both cases (webhooks and /events) storage is needed producer
| side, so nothing consequential has changed, only with /events you
| need long lived TCP connections which ties up (e.g. you can't do
| this on FaaS endpoints)
|
| Functions as a service are absolutely ideal for low frequency
| webhook receivers. SO SO SO cheap.
| alexbouchard wrote:
| I partially agree but the issue with relying on the producer
| delivery is that you effectively give up control on what the
| retry logic is. If it doesn't fit your use case, too bad for
| you. While ideally every platform would provide those
| configuration I think it's unreasonable to think all platforms
| will offer excellent webhook tools & configuration. You better
| just take things under your own hands.
| coder543 wrote:
| > But if the producer retries and the consumer does not respond
| with 200 until it has processed the message, no consumer side
| message queue is needed, the consumer can rely on the producer
| to reach at-least-once delivery.
|
| Part of the point of the article was that you may deploy bad
| code which returns 200, but doesn't actually take the correct
| action with the events, and then you have lost all that data
| and have no way to get it back, which is why you have a
| consumer-side message bus to hold the webhook history, so that
| you can replay the webhooks if you made a mistake. Your comment
| does not address this at all.
|
| If the service exposes a /events page, and especially one that
| supports long polling (or SSE), then you no longer need a
| consumer-side message bus, and you might not even need webhooks
| at all.
|
| I definitely think webhooks should be offered, but I agree with
| the article that webhooks shouldn't be the only thing.
| villasv wrote:
| The article specifically exemplifies the case of the web hook
| having faulty code (introducing nulls) while still not failing
| with errors. In this case if you don't have storage on the
| consumer side, you have to ask for the producers to be merciful
| gods.
| ezekg wrote:
| In this case, you can simply replay the badly handled
| webhooks. No need for long-polling.
| einrealist wrote:
| I am in favor of doing both: provide Webhooks (Callbacks) and
| Feeds. Webhooks are great as triggers. The payload can be also
| minimal, especially if data is sensitive and authentication /
| authorization is an issue. And Feeds provide data (can be static
| / cached, served by CDNs, optimized for batch processing in
| different variants) at the consumers' pace. The combination of
| both are ideal.
| sudhirj wrote:
| I find myself thinking about the often for any kind of
| distributed system, and here's the rule of thumb: pulling events
| / polling works 100% of the time but is inefficient, and webhooks
| work 99% of the time but are more efficient.
|
| Make the even ingestion job idempotent, so it can handle multiple
| receives, then hook up a pull / poll every 5 minutes or hour or
| whatever to start at the last pulled date (very important, don't
| start at 5 minutes ago or you'll miss downtimes). Then optionally
| set up webhooks or a push model.
| z3t4 wrote:
| Something that is underestimated in messaging systems are
| incremental numbers, eg. add + 1 for each message, so the
| receiving end get 1,2,3,4,5,6 etc, and if it then get 8 it knows
| it missed the 7th message. And it will know if the messages are
| out of order. And it can pick up if it goes down by requesting
| the missing messages.
| Redsquare wrote:
| Or put simply, a sequence number!
| the_linux_lich wrote:
| Dude, just use a timestamp.
| z3t4 wrote:
| no... time is relative! (no pun intended) - sending
| messages to the space station or a satellite in orbit would
| be off, or a moving,train, or a tall building... Did you
| know there are leap-seconds, just like there are leap-year?
| and atomic clocks are not yet standard on server/PC's, what
| is standard though is a time-client that will change the
| computers clock at some random intervals depending how much
| drift it has. A sequence number is also easier to parse.
| mrkurt wrote:
| We have (almost) the opposite problem, webhooks are too
| synchronous for what we need to ship to people. We're
| experimenting with giving people a NATs endpoint to listen for
| logs and other events: https://community.fly.io/t/fly-logs-over-
| nats/1540
|
| Having an ephemeral messaging system and a ledger to reconcile
| against is a nice, simple way to provide immediacy and eventual
| consistency (where eventual could be days). It's a pattern we're
| using all throughout our infra.
| joelcollinsdc wrote:
| We have this problem too. When we send a request to create an
| entity in an external system, before we get the response back
| with the entity Id, we already have received a webhook saying
| said entity was created. Makes a basically trivial workflow
| quite confusing.
| rkalla wrote:
| FWIW, this is what CouchDB has done from Day 1 and it _always_
| seemed like one of the most magical and surprising things tool
| or platform providers would suddenly realize and then LOVE
| about using the DB.
|
| There was nothing fancy about it, you could just listen to an
| endpoint and it was a stream of the append only log of events
| occuring in the DB to the point that you could literally use it
| to feed a replicated master or slave (or backup).
|
| I imagine your use-case is a bit more nuanced, but I sure do
| love that model.
| grejdi wrote:
| Webhooks are great for producers of events, and I'd argue that
| it's too cumbersome for them to provide an '/events' endpoint
| primary because of scaling. With webhooks, they can offload
| events at their own pace.
|
| For consumers, I agree with most here that Kafka is certainly
| overkill. We've gotten away with a very simple architecture to
| have reliable event consumption. We point all webhooks to an
| (AWS) API Gateway backed by Lambdas. The Lambdas push the events
| to an SQS queue (FIFO-queue, if it needs some sort of sequence),
| and we take our time consuming the events through a very generic
| poll.
| jcrites wrote:
| > With webhooks, they can offload events at their own pace.
|
| They can't offload webhooks at their own pace if the two
| parties want reliable delivery. The server providing the
| webhook might be experiencing a prolonged outage, in which case
| the sender needs the ability to buffer the events anyway.
| masklinn wrote:
| > Webhooks are great for producers of events, and I'd argue
| that it's too cumbersome for them to provide an '/events'
| endpoint primary because of scaling. With webhooks, they can
| offload events at their own pace.
|
| TBF they could do something similar with `/events`, instead of
| pushing events to a webhooks-sending queue just push them to
| the events buffer, which could even be a circular buffer just
| to point out that the essay is completely wrong. TFA is not
| asking for /events, they're asking for a very specific kind of
| /events with a large non-drained buffer. Something which would
| only ever work for low number of events: $dayjob's github
| integration takes in several events per second.
|
| A proper event stream would be nice though, github's webhooks
| delivery system is not exactly reliable.
| deniska wrote:
| I'm building integrations with various marketplaces at a company
| I work at (fulfilled by seller kind of deal), and I can confirm,
| it's much easier for us to schedule an HTTP request once in 15
| minutes than to create a custom HTTP service responding to
| specific requests from 3rd parties.
|
| We're in the business of selling things, we're not in the
| business of building HTTP services. Our ERP-like thing is down
| for maintenance from 10pm to 11pm, so we can't use it as a
| platform for responding to webhooks.
|
| I'll hack something together in a pinch when it's the only way to
| get orders from a marketplace service, but then eventually I'll
| have to explain to my colleagues how to linux, how to HTTPS, how
| to python, how to WSGI, and all other stuff our company typically
| doesn't do, but has to do now, because this particular
| marketplace wants to POST orders to us.
| mkherlakian wrote:
| You might want to check out https://hookdeck.com (I work on
| it). We built it precisely for this use case, you shouldn't
| have to spend of bunch of time building webhook ingestion
| infrastructure.
| cgsmith wrote:
| Any way to use cname for hookdeck? Curious if we can have
| webhooks route to example.com instead of hookdeck.com
|
| Also, any SLA on uptime?
| boring_twenties wrote:
| Just curious, how would you _prefer_ to receive these
| notifications?
| deniska wrote:
| By polling an endpoint provided by the marketplace with
| parameters "since" and "to" to filter events by the time they
| happened. We typically set "since" to two days ago and "to"
| to tomorrow. We're not in the hurry, we have an hour or two
| of leeway between receiving a message and having to act on
| it. I certainly prefer the polling solution for that usecase.
| Easier to set up, easier to debug, easier to notice that
| something is wrong.
| dceddia wrote:
| Maybe there's an opportunity here for some kind of
| buffering service that would receive webhooks and present
| them as a stream of events. Or maybe something like this
| already exists?
| deniska wrote:
| That's more or less what I implemented with a bit of
| python and sqlite. It works, but it's another piece of
| infrastructure to care about in a shop full of people who
| never had to care about that kind of infrastructure. For
| example we (well, I, really) forgot to configure certbot
| to restart nginx after renewing a cert, and only noticed
| that after a marketplace notified us that they're
| temporarily pulling off our SKUs due to our HTTP service
| being misconfigured.
|
| Can this be a 3rd party service? It certainly can be, but
| it's hard to make a generic one for any kind of webhook.
| Some marketplaces expect a dynamic response, like
| replying the order number we assigned internally (I
| typically just echo back the number they gave us with
| some prefix, but it's still more smarts than just
| replying with empty 200 OK).
|
| And I've seen services which aggregate popular local
| marketplaces into API which is easier to work with, but
| they require to concede some other parts of the business
| we'd rather keep in-house, like assortment and inventory
| management.
| boring_twenties wrote:
| I was thinking the same thing, it's kind of why I asked
| the question. :)
|
| On the one hand, it seems like something too simple to
| expect people to pay for.
|
| On the other, it's so simple it wouldn't be a huge loss
| to try it out and see if they will.
| itsbits wrote:
| How easy will it be to reset cursor position by request via
| `/events`? Can it also takes input with which it can reset else
| just use the position?
| supergeek133 wrote:
| Working in IoT land, we rely quite a bit on EventHub/AMQP type
| delivery for events to say a 3rd party (e.g., Alexa/Google/Etc).
|
| That being said, I've also lobbied for a similar endpoint so I
| don't get support tickets for "missing data".
|
| Both? Both are good.
| dgudkov wrote:
| I think the author is fighting the wrong problem. Webhooks are a
| _notification_ mechanism first of all, not a data transfer
| protocol. You can view it as a control plane which can be mixed
| or not with a data plane.
|
| What they offer is a data plane and it makes sense. Although, it
| doesn't contradict the idea of webhooks, but rather complements
| it. A consumer can get notified via a webhook when new data is
| available. Whether the data itself comes with the webhook, or is
| available via an additional API request, is a matter of design.
| Personally, I like the idea of separating the control plane from
| the data plane. However, in some cases it can be an overkill.
___________________________________________________________________
(page generated 2021-07-14 23:03 UTC)