[HN Gopher] Show HN: Inngest 1.0 - Open-source durable workflows...
___________________________________________________________________
Show HN: Inngest 1.0 - Open-source durable workflows on every
platform
Hi HN! I'm Tony, one of the co-founders of Inngest
(https://inngest.com/) Inngest is an open-source durable workflow
platform that works on any cloud. Durable workflows are stateful,
long running step functions written in code, which automatically
retry on failure. It abstracts everything about queues, event
streams and state for you, letting you focus on code. Some examples
of uses: managing stateful AI chained step functions; managing
search/rag indexes and data pipelines; integrations and webhooks;
billing and payment flows. Technical details: unlike other
solutions, we put lots of effort into designing our SDK's step.run
APIs to make them extremely easy to use -- developer experience is
the most important thing for us. We had to design and build our
own queueing system to work with multi-tenancy, batching, and
debouncing, and we're iterating on this as we move to FoundationDB.
It's largely all Go in the backend, with a bunch of caching,
clickhouse, event streams, and coordination on our behalf. Workers
are shared nothing, and run based off of the queue and execution
state. We did a post last year as we iterated on our TS SDK. The
product has changed a lot since then and wanted to show the
community what's changed as we reach 1.0: *
Golang, Java, and Python SDKs with cross-language function
invocation (across clouds, too) * Multi-tenant aware
flow control (concurrency, throttling, debounce) *
Batching, grouping many events into a single function call
* Much improved dashboard, with tracing and metrics built in
* Advanced recovery tools like function replay, temporary pausing,
bulk cancellation (with optional expressions). No more dead letter
queues! * Branch deploys built in, with staging env
support out of the box * Full local testing with
production parity There's a ton on the roadmap, with
more launching next week. We're hiring systems & infra engineers,
too -- it's a fun job with lots of challenges! Wanted to say thank
you to the HN community for feedback so far! Happy Friday :)
Author : tonyhb
Score : 91 points
Date : 2024-09-20 17:33 UTC (5 hours ago)
(HTM) web link (www.inngest.com)
(TXT) w3m dump (www.inngest.com)
| arez wrote:
| Really great product, I'm using inngest since 2-3 months and it
| definitely solved our problem. We needed a scheduling, queue,
| trigger solution. The docs are fairly good and it worked after
| some fiddling with our setup. I can quickly see every morning if
| all the jobs went through, we also starting to use them for
| webhooks now. Very happy, wrote at least 10 different functions
| in typescript so far
| ddmkr_conveo wrote:
| Congrats on the launch! I am curious, how does your multitenant
| system work? And how do you ensure fairness?
| tonyhb wrote:
| Thanks! There are a few things at play. High level, it's
| important to note we provide infra for _you_ -- so our multi-
| tenancy supports many tenants for every user.
|
| In general, fairness means distributing work evenly amongst all
| accounts. This largely means partitioned queues. Firstly, every
| functions have their own queues. We have queues of queues:
| queues of functions available. And then we have queues of
| accounts. And so on. It's like nesting dolls all the way down.
|
| There are lots of really fine details to get right:
| continuations, step parallelism, `connect()` with long running
| workers all mess with fairness, as do batching, debounce,
| throttling.
|
| We wrote about it in some detail here:
| https://www.inngest.com/blog/building-the-inngest-queue-
| pt-i.... We'll probably do some deeper dives on this next year,
| too!
| tomredman wrote:
| Wow, I had no idea you weren't at 1.0 yet lol. I've been using
| Inngest for a few years(!) and it's always felt really polished
| and complete to me.
|
| Congrats on the big one-oh. Question: I've been using Rust a lot
| more lately -- does Inngest work natively with Rust? Is there an
| SDK or another way to use it?
| tomredman wrote:
| Are there plans beyond workflow? (For context I enjoy Inngest
| already :)
| PeterZaitsev wrote:
| Looking at the License - This is not Open Source, but rather
| Source Available software.
|
| Looks great but do not appreciate deceptive marketing
| mdaniel wrote:
| And it seems their misnomer is practically everywhere, not just
| in the Show HN: their website also mislabels their links as
| "Open Source" - I guess trying to capitalize on SEO
|
| SSPLv1 for anyone similarly interested
| https://github.com/inngest/inngest/blob/v1.0.0/LICENSE.md
|
| Seems they had a change of heart around 2022:
| https://github.com/inngest/inngest/pull/81 but they actually
| only started lying about the license in this go-around because
| their previous Show HN
| <https://news.ycombinator.com/item?id=36403014> not only didn't
| mislabel things but they even said "we're gonna open source in
| the future" but I guess the future isn't here yet
| tonyhb wrote:
| We'll roll out a change that releases source as GPL after 3-4
| years next week, actually. I do appreciate these comments and
| points.
| lgsilver wrote:
| And the production infra for running isn't even available,
| just a pared down "development server" via SSPL. This is a
| long way from OSS.
| mdaniel wrote:
| Well, my experience has been closer to the "more eyes make
| for shallow bugs" school of thought, so opening the source
| to contributions would actually help that process, not
| hinder it
|
| I've written _quite a lot_ of CI for projects because it 's
| something I believe in and am willing to roll up my sleeves
| to get done (as a concrete example). I believe strongly
| that being able to reference the canonical CI build helps
| contributors since they can see how it's built for
| different systems and also ensure they don't submit "works
| on my machine" patches
| lgsilver wrote:
| Agree. This is absolutely deceptive. It's too bad how the OSS
| moniker is being misused these days...
| ljm wrote:
| Used this for a short while and the dev experience was great (the
| console in particular which allowed for copying events to
| reproduce them locally). The typescript integration was good.
|
| My only issue was that the execution of an inngest function
| wasn't completely intuitive, at least in TS, and you have to
| think in terms of inngest or, more precisely, the abstraction it
| is providing. Is it an actor, a step function, an event consumer,
| a saga? or a combination of some?
|
| When you get used to it it's nice, less overhead than building
| your own actor model or your own event sources, and really good
| visibility into what is happening.
| tomredman wrote:
| > My only issue was that the execution of an inngest function
| wasn't completely intuitive, at least in TS, and you have to
| think in terms of inngest or, more precisely, the abstraction
| it is providing. Is it an actor, a step function, an event
| consumer, a saga? or a combination of some?
|
| I'm personally curious about this. I'm not saying you don't,
| but why would you need to understand what kind of abstraction
| it is beyond it just being Inngest"? I like to think I've been
| able to use it effectively without this having ever crossed my
| mind. But I also just might be dumb. Hence the curiosity!
| ljm wrote:
| Because code doesn't execute as you might think it does,
| which is why Inngest has docs describing best practices.
|
| You have to be careful about how you structure effectful code
| and how you might share data between steps, and you have to
| understand when that code might be executed, so the more you
| know about that, the better. Inngest itself, at least with
| Vercel, doesn't do its own compute.
|
| Inngest, as far as I know, does not do its own compute, it
| piggybacks on your own, so if you're not careful you can go
| hard with inngest but you'll see the impact on your hosting
| bill; especially with Vercel.
|
| Inngest is a breath of fresh air but, you know, you have to
| audit your dependencies whether they're SaaS or not. Know how
| they work, know how to debug them, know how much they'll cost
| you.
| Xiol32 wrote:
| No option to self-host?
| tonyhb wrote:
| Our launch week starts on Monday. There's something coming,
| with a full roadmap for differing DBs, multi-node setups, and
| metrics :)
| FinalDestiny wrote:
| I'm excited to try it! Sounds like a great platform
| guilhermecgs wrote:
| what is the difference from https://temporal.io/
| tonyhb wrote:
| There are similarities with "durable workflows" but, honestly,
| a completely different approach. Lots of respect for them.
|
| A few critical things:
|
| * Inngest is primarily event driven. You send an event, we run
| _one or more_ functions.
|
| * Events give you lots of benefits: batching, rate limiting,
| replay, archiving, fan-out, etc.
|
| * You also get a powerful event-based API in functions:
| `step.waitForEvent`. This lets you do a _lot_ of things: human
| in the loop flows, coordination -- and you don 't have to learn
| complex APIs, worry about state, can handle timeouts easily.
|
| * And events also let you connect things like webhooks, CDC,
| external systems to functions with basically zero overhead.
|
| * Because of this, you can pause individual functions, then
| redrive events through specific functions - as events are
| stored for you in an OLAP event store for your workspace.
|
| Fundamentally, in terms of DX:
|
| * You don't have to register individual activities and
| workflows. Steps are lambdas. It's easy to work with
|
| * Every step (activity) has an ID, which means versioning,
| replay, determinism is easier to reason about and see in our
| model.
|
| * We also run on servers as well as serverless. It's actually
| quite nice bringing state and retries to serverless functions -
| they're good for (some) bursty workloads
|
| * We also embed a bunch of flow control into the system for
| you.
|
| Overall, both replace queueing systems. I think that's a good
| thing: you shouldn't really have to think about the specifics
| of your underlying infra when you're writing application code.
| mdaniel wrote:
| Foremost, thank you for the detailed answer. It's always
| helpful to have competitive analysis from the source
|
| While the edit window on your comment is still open, you may
| want to consider removing the leading whitespace from your
| bullet points (since they're already newline delimited)
| because on HN those leading spaces make it pre-formatted and
| that means folks on mobile have to horizontally scroll to see
| your whole sentence
| sumukh1 wrote:
| This looks great. Do you have an ETA for a Ruby SDK?
| lmeyerov wrote:
| Congrats, definitely an area we're always examining tools
|
| Afaict, this seems like a more restrictively licensed & less
| capable alternative to Dagster and Prefect. However, there may be
| some specific areas it is ahead -- the multitenancy bullet point
| sounds interesting, for example. Maybe you can share a
| comparative description?
| jedberg wrote:
| Congrats on the launch! Can't wait to see what you guys launch
| next week.
| darrylcodes wrote:
| Congrats this is huge! I've been in the market looking for a tool
| like this and was curious to know how this compares against
| toolings such as Temporal (https://temporal.io/product) or
| Trigger.dev(https://trigger.dev/) ?
| tonyhb wrote:
| Answered this in depth here:
| https://news.ycombinator.com/item?id=41605220
|
| The TL;DR is both us and Temporal help with durable execution.
| We layer on events and a differing DX to make things easier,
| faster, and to also work on serverless if people need.
|
| I can't speak to Trigger, as they've changed a lot since I last
| looked. It doesn't look like they do durable execution,
| workflows, steps, or events -- it looks like they're an
| alternative to Lambda + SQS for TS only, so very different --
| more for your simple "run one thing in the background", from
| what I can gather. Take this with a grain of salt, though!
| ykhli wrote:
| Congrats! I used Inngest when I wrote a video processing pipeline
| here https://github.com/tigrisdata-community/multi-modal-
| starter-...
|
| Amazing devEx. Thanks so much for all the work and enabling a
| local mode too
| throwaway743950 wrote:
| What are open source alternatives to this? Can temporal.io do the
| same things?
___________________________________________________________________
(page generated 2024-09-20 23:00 UTC)