[HN Gopher] Building a modern durable execution engine from firs...
___________________________________________________________________
Building a modern durable execution engine from first principles
Author : whoiskatrin
Score : 49 points
Date : 2025-03-27 13:51 UTC (1 days ago)
(HTM) web link (restate.dev)
(TXT) w3m dump (restate.dev)
| oulipo wrote:
| Interesting, how does it compare to Inngest and DBOS?
| p10jkle wrote:
| Hey, I work on Restate. There are lots of differences
| throughout the architecture and the developer experience, but
| the one most relevant to this article is that Restate is itself
| a self-contained distributed stream-processing engine, which it
| uses to offer extremely low latency durable execution with
| strong consistency across AZs/regions. Other products tend to
| layer on top of other stores, which will inherit the good
| things and the bad things about those stores when it comes to
| throughput/latency/multi-region/consistency.
|
| We are putting a lot of work into high throughput, low latency,
| distributed use cases, hence some of the decisions in this
| article. We felt that this necessitated a new database.
| ALLTaken wrote:
| Hi,
|
| I'm building a distributed application based on Hypergraphs,
| because the data being processed is mostly re-executable in
| different ways.
|
| It's so refreshing to read this, I was also sitting down many
| nights and was thinking up about the same problem that you
| guys solved. I'm so glad about this!
|
| Would it be possible to plug other storage engines into
| Restate? The data-structure that needs to be persisted allows
| multiple-path execution and instant re-ordering without
| indexing requirements.
|
| I'm mostly programming in Julia and would love to see some
| little support for it too =)
|
| Great work guys!
| sewen wrote:
| Thank you for the kind words!
|
| The storage engine is pretty tightly integrated with the
| log, but the programming model allows you to attach quasi
| arbitrary state to keys.
|
| So see whether this fits your use case, would be great to
| better understand the data and structure you are working
| with. Do you have a link where we could look at this?
| bluelightning2k wrote:
| I find this type of thing very interesting technically, but
| not very interesting commercially.
|
| It would seem to me that durable execution implies long
| running jobs, but this kind of work suggests micro
| optimisation of a couple of ms. The applications inherently
| don't care about this stuff?
|
| What am I missing. Or is it just that at a big enough scale
| anything matters.
| popalchemist wrote:
| How does it compare against Trigger or Hatchet?
| dang wrote:
| One of the authors worked on Apache Flink but is too modest to
| include that interesting detail! So I'm adding it here. Hopefully
| he won't mind.
| yubblegum wrote:
| They mention it in their about page: https://restate.dev/about
| dang wrote:
| Thanks--sounds like it was more than one of them!
| sewen wrote:
| All of the Restate co-founders com from various stages of
| Apache Flink.
|
| Restate is in many ways a mirror image to Flink. Both are
| event-streaming architectures, but otherwise make a lot of
| contrary design choices.
|
| (This is not really helpful to understand what Restate does for
| you, but it is an interesting tid bit about the design.)
| Flink | Restate -------------------------------
| | analytics | transactions |
| coarse-grained | fine-grained snapshots | quorum
| replication | throughput- |
| latency-sensitive optimized |
| | app and Flink- | disaggregated code share
| process | and framework |
| Java | Rust
|
| the list goes on...
___________________________________________________________________
(page generated 2025-03-28 23:00 UTC)