[HN Gopher] Every System is a Log: Avoiding coordination in dist...
___________________________________________________________________
Every System is a Log: Avoiding coordination in distributed
applications
Author : sewen
Score : 138 points
Date : 2025-01-24 13:57 UTC (9 hours ago)
(HTM) web link (restate.dev)
(TXT) w3m dump (restate.dev)
| sewen wrote:
| A short summary:
|
| Complex distributed coordination and orchestration is at the root
| of what makes many apps brittle and prone to inconsistencies.
|
| But we can mitigate much of complexity with a neat trick,
| building on the fact that every system (database, queue, state
| machine) is effectively a log underneath the hood. By
| implementing interaction with those systems as (conditional)
| events on a shared log, we can build amazingly robust apps.
|
| If you have come across "Turning the Database Inside Out"
| (https://martin.kleppmann.com/2015/11/05/database-inside-
| out-...), you can think of this a bit like "Turning the
| Microservice Inside Out"
|
| The post also looks at how this can be used in practice, given
| that our DBs and queues aren't built like this, and how to strike
| a sweet-spot balance between this model with its great
| consistency, and maintaining healthy decoupling and separation of
| concerns.
| teddyh wrote:
| Is this summary AI generated?
| sewen wrote:
| Haha, no, but maybe all the AI-generated contents out there
| is starting to train me to write in a similar style...
| p10jkle wrote:
| I feel this happening to me too... depressing
| TeMPOraL wrote:
| Or an improvement, given that SOTA models write better
| than most people...
| whoiskatrin wrote:
| whats your take on handling log compaction to prevent unbounded
| growth, especially in systems with high write throughput?
| sewen wrote:
| Nice question! Restate is not a log that retains the raw events
| for a long time - conceptually just until they where processed
| by the handlers, DB, locking, etc.
|
| When you build stateful handlers, the state per key is in the
| internal DB, and that get's you a similar effect to log
| compaction, i.e., retain one value per key.
| trollbridge wrote:
| I have a "summarise" log entry: the current log's contents that
| will be relevant to the future are summarised. For example, if
| it's FY2023's financial transactions, we compute the final
| balances at the end of the year. We then close the log, and
| write an entry to it of "no more log entries after this are
| valid".
|
| We then copy the summary transactions to a new log, and
| compress and archive the old log.
|
| You can identify high throughput and low throughput types of
| log entries and segregate them into different log streams. For
| example, the "new customer/change customer info" stream
| probably gets way less traffic than the "customer has logged
| in" stream. The former is also harder to summarise. Put the
| hard to summarise but low volume stuff in its own log.
| EGreg wrote:
| Since we're on the subject of logs and embarassingly parallel
| distributed systems, I know someone who's also in NYC who's been
| building a project exactly along these lines. It's called
| _gossiplog_ and it uses Prolly trees to make some interesting
| results.
|
| https://www.npmjs.com/package/@canvas-js/gossiplog
|
| Joel Gustafson started this stuff at MIT and used to work at
| Protocol Labs. It's very straightforward. By any chance sewen do
| you know him?
|
| I first became aware of this guy's work when he posted
| "merklizing the key value store for fun and profit" or something
| like that. Afterwards I looked at log protocols, including SLEEP
| protocol for Dat/Hypercore/ pear and time-travel DBs that track
| diffs, including including Dolt and even Quadrable.
|
| https://news.ycombinator.com/item?id=36265429
|
| Gossiplog's README says exactly what this article says--
| everything is a log underneath and if you can sync that (using
| prolly tree techniques) people can just focus on business logic
| and get sync for free!
| sewen wrote:
| Never encountered it before, but it looks cool.
|
| I think they are trying to solve a related problem. "We can
| consolidate the work by making a generic log that has
| networking and syncing built-in. This can be used by developers
| to make automatically-decentralized apps without writing a
| single line of networking code."
|
| At a first glance, I would say that Gossiplog is a bit more low
| level, targeting developers of databases and queues, to save
| them from re-building a log every time. But then there are
| elements of sharing the log between components. Worth a deeper
| look, but seems a bit lower level abstraction.
| EGreg wrote:
| It's part of his higher-level framework called Canvas.
|
| Check this out:
| https://joelgustafson.com/posts/2024-09-30/introduction-
| to-c...
|
| And this: https://github.com/canvasxyz/canvas
| vdm wrote:
| Thank you @EGreg for sharing this.
| EGreg wrote:
| Def. I geek out on this stuff, as I am building my own
| distributed systems. I have had discussions with a lot of
| people in the space, like Leslie Lamport, Petar Maymounkov
| etc.
|
| You might like this interview:
| https://www.youtube.com/watch?v=JWrRqUkJpMQ
|
| This is what I'm working on now:
| https://intercoin.org/intercloud.pdf
| davexunit wrote:
| My takeaway from this article is that the proposed solution for
| distributed app coordination is a shared, centralized log. What
| did I miss?
| azmy wrote:
| IMHO the article is not mainly about the implementation of the
| Log, but rather leveraging on the idea of the log to build
| reliable and fault tolerant applications. The implementation of
| the log itself can be either centralised or decentralised.
| sewen wrote:
| That gist is correct - I would add that the log needs a few
| specific properties and conceptually be the shared log for
| state, communication, execution scheduling.
|
| The next step is the, how do you make this usable in
| practice...
| trollbridge wrote:
| I've been doing a similar thing, although I called it "append
| only transaction ledgers". Same idea as a log. A few principles:
|
| - The order of log entries does not matter.
|
| - Users of the log are peers. No client / server distinction.
|
| - When appending a log entry, you can send a copy of the append
| to all your peers.
|
| - You can ask your peers to refresh the latest log entries.
|
| - When creating a new entry, it is a very good idea to have a
| nonce field. (I use nano IDs for this purpose along with a
| timestamp, which is probabilistically unique.)
|
| - If you want to do database style queries of the data, load all
| the log entries into an in memory database and query away.
|
| - You can append a log entry containing a summary of all log
| entries you have so far. For example: you've been given 10 new
| customer entries. You can create a log entry of "We have 10
| customers as of this date."
|
| - When creating new entries, prepare the entry or list of entries
| in memory, allow the user to edit/revise them as a draft, then
| when they click "Save", they are in the permanent record.
|
| - To fix a mistake in an entry, create a new entry that "negates"
| that entry.
|
| A lot of parallelism / concurrency problems just go away with
| this design.
| XorNot wrote:
| How do you know summary entries are valid if order doesn't
| matter?
|
| I.e. "we have 10 customers as of this date" can become
| immediately invalid if a new entry is appended afterwards with
| a date before that summary entry (i.e. because it was on a peer
| which hadn't yet sent it)
| clayg wrote:
| IME you have to be willing to recalculate the summaries up to
| some kind of consistency window.
|
| Yes you may be changing history and you may have a business
| reason not to _address_ that revision immediately (you 've
| already billed them?) - but the system can still learn it
| made a mistake and fix it (add activity from Jan 30 evening
| that comes in late to the Feb bill?)
| withinboredom wrote:
| Realistically, you never store summaries in the log. Instead,
| you store what it took to calculate them. So you won't store
| "we have 10 customers on this date with this range" but
| instead store "we found these 10 customers on this date with
| this range". This assumes you can store infinite sized lists
| in your log, but realistically, this is never a concern if
| you can keep your time windows small enough. Then, you
| periodically do a reconciliation and log corrections (look
| for entries not summarized -- easily done via a bloom filter
| which can tell you what entries are definitely NOT in your
| set) over a longer period.
|
| For example, we had a 28-day reconciliation period at one
| company I worked at (and handled over 120 million events per
| day). If you appended an event earlier than 28 days prior, it
| was simply ignored. This very rarely happened, but allowed us
| to fix bugs with events for up to 28 days.
| Kinrany wrote:
| > The order of log entries does not matter.
|
| This is surprising, Kafka-like logs are all strictly ordered.
| cduzz wrote:
| * within a partition
| grahamj wrote:
| The lack of ordering is surprising. Without that you can't
| stream without a buffer.
| log4shell wrote:
| Calling a WAL a ledger, why? Ledger sounds fancier but why
| would it be a ledger in this case?
| hcarvalhoalves wrote:
| I believe "ledger" implies commutative property (order does
| not matter).
| log4shell wrote:
| I am not aware of any such implicit connection of ledger
| and commutative property, also couldn't find anything as my
| google-fu is letting me down. Anything I can refer to?
| Generally curious to know use of term ledger outside of
| accounting and blockchains.
|
| I have seen it used to mean WAL before, so I am taking this
| with a dose of skepticism.
| glitchc wrote:
| How do you manage log size for high-transaction systems?
| jaseemabid wrote:
| A notable example of a large-scale app built with a very similar
| architecture is ATproto/Bluesky[1].
|
| "ATProto for Distributed Systems Engineers" describes how updates
| from the users end up in their own small databases (called PDS)
| and then a replicated log. What we traditionally think of as an
| API server (called a view server in ATProto) is simply one among
| the many materializations of this log.
|
| I personally find this model of thinking about dataflow in large-
| scale apps pretty neat and easy to understand. The parallels are
| unsurprising since both the Restate blog and ATProto docs link to
| the same blog post by Martin Kleppmann.
|
| This arch seems to be working really well for Bluesky, as they
| clearly aced through multiple 10x events very recently.
|
| [1]: https://atproto.com/articles/atproto-for-distsys-engineers
| sewen wrote:
| That blog post is a great read as well. Truely, the log
| abstraction [1] and "Turning the DB inside out" [2] have been
| hugely influential.
|
| In a way this article here suggests to extend that
|
| (1) from a log that represents data (upserts, cdc, etc.) to a
| log of coordination commands (update this, acquire that log,
| journal that steo)
|
| (2) have a way to link the events related to a broader
| operation (handler execution) together
|
| (3) make the log aware of handler execution (better yet, put it
| in charge), so you can automatically fence outdated executions
|
| [1] https://engineering.linkedin.com/distributed-systems/log-
| wha...
| sewen wrote:
| [2] https://martin.kleppmann.com/2015/11/05/database-inside-
| out-...
| grahamj wrote:
| Table/log duality goes back further than Kleppmann though. An
| earlier article that really influenced me was
|
| https://engineering.linkedin.com/distributed-systems/log-wha...
| zellyn wrote:
| Martin Kleppmann was also directly involved with Bluesky as a
| consultant.
| dboreham wrote:
| This is basically CSP no?
| sewen wrote:
| I assume CSP is communicating sequential processes?
|
| Interesting analogy - in a way it is doing something CSP-like
| in a distributed app/service architecture with the all the
| different processes and components that are there. The shared
| log (or a partition of that) being a way to establish a
| sequential order.
| pjc50 wrote:
| > Having a single place (the one log) that forces a linear
| history of events as the ground truth and owns the decision of
| who can add to that ground truth, means we don't have to
| coordinate much any more.
|
| Well, yes, but then you've backed into CAP again because you only
| have _one_ log.
| clayg wrote:
| But can't any log be implemented as a CRDT? Was that not
| implied in the post? I didn't read it that close...
| ismailmaj wrote:
| CRDT is really only useful when inconsistencies could be
| acceptable in some situations and so it depends on the
| application.
|
| For something that is trying to solve the general problem of
| consistent single ground-truth log, you can't really do much
| better than Spanner.
| logsr wrote:
| the "conflict-free" in crdt is like miller high life being
| the "champagne of beers." it really means conflicts ignored
| and conflicting items discarded by established rules, which
| works well for some use cases, but for many does not.
| mrkeen wrote:
| If I've understood it, it's like using Kafka with 1 topic and 1
| partition. But it shouldn't rule out multiple brokers with a >1
| replication factor, giving you CP.
| sewen wrote:
| Yes, we are assuming a log that picks linearizability at the
| cost of availability under partitions. Like most logs do,
| including Kafka, Pulsar, RedPanda, etc.
|
| The application state is defined by the log here, and the log
| drives retries/recovery, so it doesn't much matter if the
| process that executes the app code splits off. The log would
| hydrate another one.
|
| Also the one log is at the granularity of a single key or
| handler execution. More of a logical log, than a physical log
| or even partition.
|
| In Restate, we implement a logical log-per-key, backed by a
| partitioned physical log.
| UltraSane wrote:
| couldn't the log be synchronously replicated to multiple
| servers to increase A?
| ismailmaj wrote:
| In case of mutation either the replicas will be out of sync
| (so no C), or you'll need to synchronously mutate the
| replicas (so bad A due to latency).
| UltraSane wrote:
| delaying writes by a factor longer than the max clock skew
| of the cluster is a pretty common strategy. It is what
| Google Spanner does.
| ismailmaj wrote:
| Right but it's not magic as you cannot predict networking
| delays.
|
| The max clock skew also has to pick between A and C, in
| underestimates you lose serializability, in overestimates
| you pay in write latency.
| UltraSane wrote:
| What if you use CockroachDB for your log? They do
| something pretty clever:
| https://www.cockroachlabs.com/blog/living-without-atomic-
| clo...
| qudat wrote:
| Great post! At pico we've been spending a lot of time thinking
| about logs and a distributed system that can read and respond to
| events from logs. This is being driven in part by building out
| global services and a need for centralized logs for monitoring.
|
| The end result is https://pipe.pico.sh which is an authenticated,
| networked *nix pipes over SSH. Since it relies on stdin/stdout
| via SSH it's one of the easiest pubsub systems we've used and we
| keep finding its ergonomics powerful. We have a centralized log-
| drain, metric-drain, and cache-clearing-drain all using `pipe`.
| bruce343434 wrote:
| > If everything's in one log, there's nothing to coordinate #
|
| On the contrary. Everything becomes coordinated.
|
| The entire "log" becomes a giant ass mutex lock. Good luck
| scaling it.
| sewen wrote:
| There is nothing to coordinate for the application, because,
| yes, the log coordinates everything. But not globally, on the
| level of a single event handler execution, or a single key.
|
| That has been proven to scale well - the way we implement that
| in Restate is classical shared nothing physical partitioning,
| with indexing on a key granularity.
|
| So nothing like a shared mutex unless you want to access the
| same key, which otherwise your database synchronizes, if you
| want any reasonable level of consistency.
| mrkeen wrote:
| I think the author is motte-and-baileying between:
|
| _Literally one log_ - which does indeed reduce your
| coordination headache, but is susceptible to your "giant ass
| mutex" comment, and
|
| _One log per ..._ - which brings the coordination problems
| right back into existence.
| sewen wrote:
| I can see where some of that could be written more clearly.
| To elaborate:
|
| - We mean using one log across different concerns like state
| a, communication with b, lock c. Often that is in the scope
| of a single entity (payment, user, session, etc.) and thus
| the scope for the one log is still small, and it reduces
| coordination headache for coordinating between the systems.
| You would have a lot of independent logs still, for separate
| payments.
|
| - It does _not_ mean that one should share the same log (and
| partition) for all the entities in your app, like necessarily
| funneling all users, payments, etc. through the same log.
| What would be needed if you try and do some multi-key-
| distributed transaction processing. That goes actually beyond
| the proposal here, and has some benefits of its own, but have
| a hard time scaling.
| mrkeen wrote:
| > thus the scope for the one log is still small, and it
| reduces coordination headache for coordinating between the
| systems.
|
| This does not follow.
|
| Neither the dev (nor the product seller) is in control of
| what real-world effects have a causal relationship.
|
| If my actions can be _independently processed_ from your
| actions, then we don 't need to coordinate, and micrologs
| will perform faster than a monolog.
|
| If my actions cannot be _independently processed_ from your
| actions, then we do need to coordinate, unless we use a
| monolog, which kills performance.
|
| Either way, we don't get to choose the level of
| independence. If the actions are dependent, we either
| coordinate or or mess up causality.
| kikimora wrote:
| You can use something like DynamoDb with partition per
| interaction. That would scale great.
| neuroelectron wrote:
| Exactly what I was thinking. Now what's the best mutex system
| we've built? An SQL database.
| mrkeen wrote:
| Using one-log-only for an entire system does have its upsides,
| but it will kill performance. It would be like building a CRUD
| system with a single mutex for everyone to share.
| hcarvalhoalves wrote:
| I believe you want "one log" in the logical sense. In theory,
| you could have "one log" per user or group of users, or
| whatever sharding technique makes sense for multi tenancy
| model. It can also be "one log" per bounded context - e.g. the
| entire payment pipeline in one log.
| sewen wrote:
| Yes, exactly right. One log per logical entity, here "payment
| ID".
|
| The way our open source project implements that is with a
| partitioned log and indexes at key-granularity, so it is like
| virtually a log per key.
| ianburrell wrote:
| How do you deal with one side of transaction getting lost?
| Traditional double entry accounting puts entry for each
| side of the transaction. It will cause problems if one side
| is lost or delayed.
|
| How do you handle transactions in general? It is pretty
| common to have multiple changes that need to be applied
| together. In a single log, can write the changes and then
| write a record that says the changes are committed. This is
| how databases work.
| mrkeen wrote:
| That's what the rest of us are doing in event-sourcing land,
| but TFA is arguing for something much stronger:
| If everything's in one log, there's nothing to coordinate
|
| The rest of us have to coordinate the logical log of user
| creation/deletion and the logical log of user payments, etc.
|
| Separate logical logs with no need for coordination can only
| work if they are truly independent systems - no causality
| between them.
| xnorswap wrote:
| It sounds like they have just re-discovered Distributed
| Transactions with a Distributed Transaction Coordinator.
|
| But DTs have a huge problem: What happens if the owner of the
| lock netsplits?
|
| Either the DTC waits (potentially forever?) for the owner of the
| lock to get back in touch and release the lock, or a timeout is
| applied and now the owner of the lock (who may be unaware of the
| netsplit) will be out of sync with the system.
| logsr wrote:
| they point out that they are adding conditional append to a
| log, which gives the universality of a log data structure and a
| mechanism for implementing lock-free/wait-free concurrency
| algorithms. it is a complete toolkit for building distributed
| systems because you can build anything else on top of those
| primitives.
| hiAndrewQuinn wrote:
| I am a _huge_ fan of append-only logs as a fundamental
| architectural principle. The Log [1] should be required reading
| for any CS undergraduate.
|
| [1]: https://engineering.linkedin.com/distributed-systems/log-
| wha...
| kasey_junk wrote:
| I love them so much that I've noodled with building a
| programming language optimized for using them.
|
| Things like types that encode what events are legal in a log,
| first class support for data versions, fast file read and
| writes, etc
| daxfohl wrote:
| How do you do GDPR takedowns?
| jaseemabid wrote:
| Immutable append only persistent log doesn't imply store
| everything _forever_.
|
| If you want to remove something you could add a tombstone
| record (like Cassandra) and eventually remove the original
| entry during routine maintenance operations like repacking
| into a more efficient format, archival into cold storage,
| TTL handling etc.
| UltraSane wrote:
| Best method is to encrypt data with a per user key so you
| if you delete the encryption key that customer's data
| cannot be read. You can create a new customer key at a
| fixed time interval to make deletion more granular.
| daxfohl wrote:
| Haven't formed thoughts on the content yet, but happy to see a
| company launching something non-AI for a change.
| gjtorikian wrote:
| My startup, Yetto (http://www.yetto.app) is building a better
| way for support professionals to do their job. (Shameless plug
| but we always gotta hustle.)
|
| We, too, are weighed down by how much space AI-focused
| companies are taking.
| hansonkd wrote:
| TBH looking at helpdesk software in 2025, I would expect new
| ones to be built AI first. It would be hard for me to
| consider one without at least some sort of LLMs helping with
| triage or at classifications of tickets, etc.
| gk1 wrote:
| See https://www.fixify.com/, although they are too modest
| about mentioning the AI they're using under the hood.
| sewen wrote:
| Some clarification on what "one log" means here:
|
| - It means using one log across different concerns like state a,
| communication with b, lock c. Often that is in the scope of a
| single entity (payment, user, session, etc.) and thus the scope
| for the one log is still small. You would have a lot of
| independent logs still, for separate payments.
|
| - It does _not_ mean that one should share the same log (and
| partition) for all the entities in your app, like necessarily
| funneling all users, payments, etc. through the same log. That
| goes actually beyond the proposal here - has some benefits of its
| own, but have a hard time scaling.
| magicalhippo wrote:
| Interesting read, not my area but I think I got the gist of it.
|
| In your Restate example of the "processPayment" function, how
| do you handle errors of the "accountService" call? Like, what
| if it times out or returns a server error?
|
| Do you store the error result and the caller of
| "processPayment" has to re-trigger the payment, in order to
| generate a new log?
| stsffap wrote:
| By default, failing ctx.run() calls (like the accountService
| call) will be retried indefinitely until they succeed unless
| you have configured a retry policy for them. In the case of a
| configured retry policy where you have exhausted the number
| of retry attempts, Restate will mark this call as terminally
| failed and record it in its log as such and return it to the
| caller.
| sewen wrote:
| You can catch these errors and handle them in a common
| try/catch manner, and because the results of `ctx.run` are
| recorded in the log, this is deterministic and reliable
| shikhar wrote:
| This post makes a great case for how universal logs are in data
| systems. It was strange to me that there was no log-as-service
| with the qualities that make it suitable for building higher-
| level systems like durable execution: conditional appends (as
| called out by the post!), support very large numbers of logs,
| allow pushing high throughputs with strict ordering, and just
| generally provide a simple serverless experience like object
| storage. This led to https://s2.dev/ which is now available in
| preview.
|
| It was interesting to learn how Restate links events for a key,
| with key-level logical logs multiplexed over partitioned physical
| logs. I imagine this is implemented with a leader per physical
| log, so you can consistently maintain an index. A log service
| supporting conditional appends allows such a leader to act like
| the log is local to it, despite offering replicated durability.
|
| Leadership can be an important optimization for most systems, but
| shared logs also allow for multi-writer systems pretty easily. We
| blogged about this pattern https://s2.dev/blog/kv-store
| logsr wrote:
| > log as a service
|
| very exciting. this is the future. i am working on a very
| similar concept. every database is a log at its core, so the
| log, which is the highest performance part of the system, is
| buried behind many layers of much lower performing cruft. edge
| persistence with log-per-user application patterns opens up so
| many possibilities.
| gavindean90 wrote:
| What about journalctl?
| shikhar wrote:
| This is why we didn't actually call it logs as a service, but
| streams :P I meant to refer to the log abstraction this post
| talks about, see links therein. Observability events are but
| one kind of data you may want as a stream of durable records.
| hinkley wrote:
| I just want a recognized standard format for write ahead logs.
| Start with replicating data between OLTP and OLAP databases
| with minimal glue code, and start moving other systems to a
| similar structure, like Kafka, then new things we haven't
| thought of yet.
| ianburrell wrote:
| The structure for the write head logs needs to different
| between systems. For Postgres, the WAL is a record of writes
| with new blocks. It can't be used without knowing the
| Postgres disk format. I don't think it can be used to
| construct logical changes.
|
| Using a standard format, converting things into logical data,
| would be significantly slower. It is important that WAL be
| fast because it is the bottleneck in transactions. It would
| make more sense to have a separate change streaming service.
| anarazel wrote:
| > I don't think it can be used to construct logical
| changes.
|
| It can: https://www.postgresql.org/docs/current/logicaldeco
| ding.html
|
| It's not entirely from the WAL though, some catalog
| accesses are necessary for metadata (shape and name of
| tables etc).
| xuancanh wrote:
| > It was strange to me that there was no log-as-service with
| the qualities that make it suitable for building higher-level
| systems like durable execution
|
| There are several services like that, but they are mostly kept
| behind the scene as a competitive advantage when building
| distributed systems. AWS uses it behind the scene for many
| services, as mentioned here by Marc Brooker
| https://brooker.co.za/blog/2024/04/25/memorydb.html. Facebook
| has similar systems like LogDevice https://logdevice.io/, and
| recently Delos https://research.facebook.com/publications/log-
| structured-pr...
| shikhar wrote:
| Indeed. We are trying to democratize that secret sauce. Since
| it is backed by object storage, the latencies are not what
| AWS enjoys with its internal Journal service, but we intend
| to get there with a NVMe-based tier later. In the meantime
| there is an existing large market for event streaming where a
| "truly serverless" (https://erikbern.com/2021/04/19/software-
| infrastructure-2.0-...) API has been missing.
| jamamp wrote:
| I wonder how this compares, conceptually, to Temporal? While
| Temporal doesn't talk about a single centralized log, I feel the
| output is the same: your event handlers become durable and can be
| retried without re-executing certain actions with outside
| systems. Both Restate and Temporal feel, as a developer coding
| these event handlers, like a framework where they handle a lot of
| the "has this action been performed yet?" and such for you.
|
| Though to be fair I've only read Temporal docs, and this Restate
| blog post, without much experience in either. Temporal may not
| have as much on the distributed locking (or concept of) side of
| things that Restate does, in this post.
| sewen wrote:
| Temporal is related, but I would say it is a subset of this.
|
| If you only consider appending results of steps of a handler,
| then you have something like Temporal.
|
| This here uses the log also for RPC between services, for state
| that outlives an individual handler execution (state that
| outlives a workflow, in Temporal's terms).
| erikerikson wrote:
| It's there a hosted offering? (Or plans to offer one?)
| zellyn wrote:
| sewen (et al)
|
| This is lovely and I'm itching to try it. One question:
|
| We have a use case where a location gets cut off completely from
| the internet at large. In that case, it makes sense for the local
| hardware (typically Android and/or iOS tablets or equivalent) to
| take over as a log owner: even though you're cut off, if you're
| willing to swallow the risk (and hence cost) of offline payments,
| you should be able to create orders, fulfill them, pay for them,
| close them out, send tickets to the kitchen to cook the food or
| to the warehouse to fetch the tractor, etc.
|
| Does restate include something that covers that use-case? In the
| noodling/daydreaming a colleague and I have done, we ended up
| with something very close to restate (I imagined just using
| Kafka), except that additionally many operations would have a
| CRDT nature: eg. you should _always_ be allowed to add a payment
| to an order, because presumably a real-life payment happened.
|
| I've also noodled with the idea of logs whose canonical ownership
| can be transferred. That covers cases where you start offline and
| then reconnect, but doesn't work so well for transactions that
| start out connected (and thus owned in the datacenter) and need
| to continue offline.
|
| One could also imagine ensuring that > n/2 consensus members are
| always located inside the restaurant/hardware store/etc., so if
| you go offline, you can still proceed. It might even be possible
| to recognize disconnection and then take one full vote to further
| subdivide that pool of consensus members so if one dies it
| doesn't halt progress. This feels like it would be getting very
| tricksy...
| withinboredom wrote:
| I'm actually working on a database implementation for this
| exact use-case... It's a distributed edge database and still
| quite a long ways to go --
| https://github.com/bottledcode/atlas-db if you want to give it
| a star.
|
| It's mostly based on wpaxos (wide-area consensus), spaxos,
| fpaxos and pretty neat. The repo above is a productionization
| of several proof of concepts to get there.
|
| > One could also imagine ensuring that > n/2 consensus members
| are always located inside the restaurant/hardware store/etc.,
| so if you go offline, you can still proceed.
|
| This is what annoys me to no end about RAFT. It's a great
| protocol, don't get me wrong, but its too simple for these
| types of problems. RAFT fails when it doesn't have consensus
| and because the consensus is non-deterministic, it must have an
| odd number of nodes. PAXOS, while far more complex than RAFT in
| terms of "grok", is deterministic so you don't need an odd
| number of nodes.
|
| If you throw in some flexible quorums, you can do some really
| neat stuff, like how Atlas handles a "region" (ie, areas
| connected via the internet instead of the same network)
| becoming disconnected; but I'm not ready yet. There's still a
| long way to go!
| amirjak wrote:
| How would you compare this to the actor model or to temporal?
| sewen wrote:
| Great question:
|
| The Virtual Objects in Restate are much like actors. They are
| somewhat inspired by Orleans [1], and you could call them
| virtual stateful actors. They blend with the durable execution
| for processing messages with multiple durable steps.
|
| Regarding temporal, check also this question:
| https://news.ycombinator.com/item?id=42815318
| TuringTest wrote:
| Excuse me for sounding rough, but - isn't this reinventing comp-
| sci, one step at a time?
|
| I learned about distributed incrementally -monotonic logs back at
| the late 90s, with many other ways to do guaranteed transactional
| database actions. And I'm quite certain these must have been
| invented in the 50s or 60s, as these are the problems that early
| business computer users had: banking software. These are the
| techniques that were buried in legacy COBOL routines, and needed
| to be slowly replaced by robust Java core services.
|
| I'm sure the Restate designers will have learned terribly useful
| insights in how to translate these basic principles into a
| working system with the complexities of today's hardware/software
| ecosystem.
|
| Yet it makes me wonder if young programmers are only being taught
| the "build fast-break things" mentality and there are no longer
| SW engineers able to insert these guarantees into their systems
| from the beginning, by standing on the shoulders of the ancients
| that invented our discipline, so that their lore is actually used
| in practice? Or am I just missing something new in the article
| that describes some novel twist?
| mrkeen wrote:
| I think your points are pretty spot on - most things have
| already been invented, and there's too much of a move-fast-and-
| break-things mentality.
|
| Here's a follow-up thought: to what extent did the grey-beards
| let us juniors down by steering us down a different path? A few
| instances:
|
| DB creators knew about replicated logs, but we got given DBs,
| not replicated log products.
|
| The Java creators knew about immutability: "I would use an
| immutable whenever I can." [James Gosling, 1] but it was years
| later when someone else provided us with
| pcollections/javaslang/vavr. And they're still far from
| widespread, and nowhere near the standard library.
|
| Brendan Eich supposedly wanted to put Scheme into browsers, but
| his superiors had him make JS instead.
|
| What other stuff have we been missing out on?
|
| [1] https://www.artima.com/articles/james-gosling-on-java-
| may-20...
| sewen wrote:
| This is certainly building on principles and ideas from a long
| history of computer science research.
|
| And yes, there are moment where you go "oh, we implicitly gave
| up xyz (i.e., causal order across steps) when we started
| adopting architecture pqr (microservices). But here is a
| thought on how to bring that back without breaking the benefits
| of pqr".
|
| If you want, you can think of this as one of these cases. I
| would argue that there is tremendous practical value in that (I
| found that to be the case throughout my career).
|
| And technology advances in zig zag lines. You add capability x
| but lose y on the way and later someone finds a way to have x
| and y together. That's progress.
| hinkley wrote:
| When I was in school I had an optional requirement. You had to
| take one out of 2 or 3 classes to graduate. That was compiler
| design, which was getting terrible reviews from my peers who
| were taking it the semester before me, or distributed
| computing. Might have been a third but if so it was
| unmemorable.
|
| So I took distributed computing. Which ended up being one of
| the four classes that satisfied the 80/20 rule for my college
| education.
|
| Quite recently I started asking coworkers if they took such a
| class and was shocked to learn how many not only didn't take
| it, but could not even recall it being an option at their
| school. What?
|
| I can understand it being rare in the 90's but the 00's and on
| were paying attention to horizontal scaling, and the 2020's are
| rotten with it distributed computing concerns. How... why... I
| don't understand how we got here.
| withinboredom wrote:
| So many people I work with don't "get" distributed systems
| and how they interplay and cause problems. Most people don't
| even know that the ORDER you take potentially competing
| (distributed) locks even matters -- which is super important
| if you have different teams taking the same locks in
| different services!
|
| The article is well written, but they still have a lot of
| problems to solve.
| Thaxll wrote:
| This is exactly this example from Temporal:
| https://github.com/temporal-sa/temporal-order-fulfill-demo
| paulsutter wrote:
| This is very compelling, nice work. I'm going to spend some
| quality time on this.
| Animats wrote:
| This is a basic concept in accounting. The general ledger is an
| immutable log of transactions. Other accounting documents are
| constructed from the general ledger, and can, if necessary, be
| rebuilt from it. This is the accepted way to do money-related
| things.
|
| Synchronization is called "reconcilation" in accounting
| terminology.
|
| The computer concept is that we have a current state, and changes
| to it come in. The database with the current state is
| authoritative. This is not suitable for handling money.
|
| The real question is, do you really care what happened last
| month? Last year? If yes, a log-based approach is appropriate.
| inopinatus wrote:
| I've always concurred with the Helland/Kleppman observation
| mentioned viz. that the transaction log of a typical RDBMS is
| the canonical form and all the rows & tables merely
| projections.
|
| It's curious that over those projections, we then build event
| stores for CQRS/ES systems, ledgers etc, with their own
| projections mediated by application code.
|
| But look underneath too. The journaled filesystem on which the
| database resides also has a log representation, and under that,
| a modern SSD is using an adaptive log structure to balance
| block writes.
|
| It's been a long time since we wrote an application event
| stream linearly straight to media, and although I appreciate
| the separate concerns that each of these layers addresses, I'd
| probably struggle to justify them all from first principles to
| even a slightly more Socratic version of myself.
| globular-toast wrote:
| > It's curious that over those projections, we then build
| event stores for CQRS/ES systems, ledgers etc, with their own
| projections mediated by application code.
|
| The database only supports CRUD. So while the CDC stream is
| the truth, it's very low level. We build higher-level event
| types (as in event sourcing) for the same reason we build any
| higher-level abstraction: it gives us a language in which to
| talk about business rules. Kleppmann makes this point in his
| book and it was something of an aha moment for me.
| withinboredom wrote:
| I'm sorry; but have you ever actually used a database
| before? A database supports FAR more than "only CRUD". Some
| really simple examples are CTEs, SELECT ... INTO (or INSERT
| ... SELECT for some dialects), triggers, views, etc.
| ianburrell wrote:
| The table data in database is the canonical form. You can
| delete the transaction logs, and temporarily lose some
| reliability. It is very common to delete the transaction logs
| when not needed. When databases are backed up, they either
| dump the logical data or take snapshot of the data. Then can
| take stream of transaction logs for syncing or backup until
| the next checkpoint.
|
| I'm pretty sure journalled filesystem recycle the journal.
| There are log-structured filesystem but they aren't used much
| beyond low-level flash.
| inopinatus wrote:
| Sorry, this is mistaking the operational for the
| fundamental.
|
| If a transaction log is replayed, then an identical set of
| relations will be obtained. Ergo, the log is the prime form
| of the database.
|
| It's that simple.
| magicalhippo wrote:
| At work we need to distribute daily changes to a dataset,
| so we have a series of daily deltas. If a new client is
| brought up, they need to apply all the deltas to get the
| current dataset.
|
| This is time consuming, so we optimized it by creating
| "base versions" every month. So a client only needs to
| download the latest base version and the apply the deltas
| since then...
| Animats wrote:
| Which is what accountants call "closing the books". Once
| all ledgers have been reconciled, old ledgers can be
| archived and you go forward from the last closing.
|
| Forensic accounting, incidentally, is when something went
| badly wrong and outside accountants have to go back
| through the old ledgers, and maybe old invoices and
| payments and reconstruct the books. FTX had to do that
| after the bankruptcy to find out where the money went and
| where it was supposed to go.
| isignal wrote:
| The transaction log maintained from time 0 would be
| equivalent but too expensive to store compared to the
| tables.
| inopinatus wrote:
| On the contrary, I've known plenty of sites that keep
| their logs.
|
| Often written to tape, for obvious reasons.
| fellowniusmonk wrote:
| This is why EG-Walker is so important, diamond types adoption
| and a solid TS port can't come soon enough for distributed
| systems.
___________________________________________________________________
(page generated 2025-01-24 23:00 UTC)