[HN Gopher] Databases = Frameworks for Distributed Systems
___________________________________________________________________
Databases = Frameworks for Distributed Systems
Author : laplab
Score : 83 points
Date : 2022-05-21 17:15 UTC (3 days ago)
(HTM) web link (laplab.me)
(TXT) w3m dump (laplab.me)
| ianbutler wrote:
| I was watching a video by one of Amazon's distinguished engineers
| a while ago and for the life of me I can't find it now but the
| thing that stuck with me from it is,
|
| "There are only three modes for a distributed system - it
| implements paxos/raft itself, it relies on a data store that
| implements paxos/raft or it's wrong."
|
| Obviously a bit of an over simplification since there have been
| some advancements in that area but not far off.
|
| This article introduces an approach that falls under the second
| mode and I think it's a good way to build a distributed system
| without having to re invent the wheel especially if you can just
| rely on the data store for the consensus and make your brokers
| and nodes relatively dumb.
|
| Look at k8s and etcd for a very widely used example of this
| approach as well.
|
| I'm also currently building a distributed search engine using
| etcd for the service registry, broker peer announcements, and
| worker queue and it's been a good experience so far.
| rubiquity wrote:
| > "There are only three modes for a distributed system - it
| implements paxos/raft itself, it relies on a data store that
| implements paxos/raft or it's wrong."
|
| That sounds like something Al Vermeulen would say. Though my
| recollection of the quote is more along the lines of:
|
| "There are three types of consistent distributed systems:
| paxos, broken protocols, and single points of failures."
|
| I'm guessing the talk it came from is Under The Covers: Core
| Distributed Systems Primitives that Power Our Platform[0]. That
| presentation is full of great insight into how AWS thinks about
| distributed systems. The other presenter, Swami, is the VP of
| AWS Databases now but started off getting Amazon's initial
| Paxos implementation going. I was lucky enough to work on those
| core building blocks and it was an incredible experience.
|
| 0 - https://www.youtube.com/watch?v=QVvFVwyElLY
| ianbutler wrote:
| Yup that's the one! Thank you for digging it up.
|
| That work must have been really gratifying especially since
| it underpins so much now.
| hardwaresofton wrote:
| Well in some situations you can also get away with CRDTs or
| Gossip (SWIM), but yeah in the general sense Paxos (Raft is a
| Paxos-family algo). There are lots of paxos family algos to
| choose from[0].
|
| > I'm also currently building a distributed search engine using
| etcd for the service registry, broker peer announcements, and
| worker queue and it's been a good experience so far.
|
| As a random stranger on the internet, please build an
| abstraction layer around etcd. Even if there's only one
| implementation, I've found that so many distributed projects
| that just never reach the scale that etcd is built for would
| benefit from the option of writing their own drivers (you don't
| need a full plugin system just a regular abstraction layer).
|
| One example is Kine[0] for k8s -- if k8s had a built in option
| for writing/reading from something like Postgres from the
| beginning it would have been a better project for it, IMO.
|
| If you _do_ build a plugin system though, you can pass off the
| work of maintaining the other implementations!
|
| [0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-
| all
|
| [1]: https://github.com/k3s-io/kine
| ianbutler wrote:
| I think that's a great idea, I already have the beginnings of
| a plugin system since local non distributed is a mode you can
| run in and I had to abstract etcd away to do that. Will add
| it to the list!
| hardwaresofton wrote:
| Thanks! Also I'm pretty curious where I can find your
| project...
| erikerikson wrote:
| This post is on to an important dynamic.
|
| We've scaled databases then broken them up and our applications
| are distributed databases now. The tools mentioned seem a bit
| random but... There does seem to be space for database tooling
| that melds into our applications but everyone wants to bring you
| into their bubble instead.
| scott_s wrote:
| Some of academia's preeminent database researchers have made a
| similar hypothesis: DBOS: A DBMS-oriented Operating System,
| https://vldb.org/pvldb/vol15/p21-skiadopoulos.pdf and
| https://dbos-project.github.io/
| jorangreef wrote:
| We've been designing TigerBeetle [1], a distributed financial
| accounting database, from the beginning with the ultimate goal
| that the whole architecture of: * global
| consensus layer (Viewstamped Replication) [2], and the *
| local storage engine (a high-performance storage fault-tolerant
| LSM-tree over io_uring) [3], * which can all be
| deterministically simulated and fuzzed,
|
| can eventually be used as a distributed library that gives you
| not only consensus but storage, all in one, kind of like "a
| DistSys Iron Man suit"--you put in your own state machine and
| business logic [4] inside it, and get all the replication and
| fault-tolerance around that, out of the box.
|
| TigerBeetle is based on some of the latest research around
| storage faults, how to recover safely and maximize high
| availability, and we've been heavily inspired by the
| deterministic simulation testing of FoundationDB [5].
|
| [1] https://github.com/coilhq/tigerbeetle
|
| [2]
| https://github.com/coilhq/tigerbeetle/blob/main/src/vsr/repl...
|
| [3] https://www.hytradboi.com/2022/tigerbeetles-lsm-forest
|
| [4]
| https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma...
|
| [5] https://www.youtube.com/watch?v=OJb8A6h9jQQ -- "How I Learned
| to Stop Worrying and Trust the Database"
| buildbot wrote:
| At My first programming job, as an intern we went with his
| approach, using rethinkDB amd python "microservices" that were
| all separate python processes reading from their own rethinkDB
| change feed lambda - the system was essentially built as a
| distributed reactive microservice pipeline, with rethinkDB as the
| source of truth for every other process - which existed on
| 100-1000 machines depending!
| gslin wrote:
| I would say the author just tried to redefined "distributed", and
| ignored those real distributed systems, e.g. Paxos/Raft-based
| ones.
| codelorado wrote:
| As per Wikipedia, a distributed system is: "a system whose
| components are located on different networked computers, which
| communicate and coordinate their actions by passing messages to
| one another from any system."
|
| You're describing consensus algorithms.
| quietbritishjim wrote:
| You are thinking of fully decentralised systems.
|
| A system where there is a centralised leader is still a
| distributed system so long as at least some of the processing
| takes place on other nodes.
| brewkode wrote:
| A few years back, a colleague and me built a distributed system
| toolkit in an effort to understand the blocks/pieces of a
| distributed system and also to make it easier to build one for
| our needs. This article seems to hit on the same set of notes
| that we had in mind when we started out building suuchi.
|
| https://github.com/ashwanthkumar/suuchi.
| janandonly wrote:
| This article was originally posted on dev.to, but it turns out
| that HackerNews banned this website, so I decided to create my
| own.
|
| HN "blocks" dev.to ? Why ?
| spijdar wrote:
| Dang answered that here:
| https://news.ycombinator.com/item?id=27110093
| jandrewrogers wrote:
| While databases having been converging on composability for a
| long time, the model presented here is not the correct one and
| causes problems for many important and emerging use cases.
|
| As an elementary observation, isolating the designs of storage,
| compute, and networking as separate modules is a reliable way to
| achieve poor performance and resource efficiency. Some of the
| most important scalability and performance optimizations in
| modern database architecture are schedule-driven, which requires
| tight coupling across the design of storage, compute, and
| networking. Ignoring this class of optimizations produces very
| large reductions in real-world workload performance, and feeds a
| narrative that the software is wasteful and not environmentally
| friendly.
|
| Heterogeneous and specialized resource collections with complex
| topologies are becoming more common for a diverse set of reasons.
| This breaks most horizontal resource abstractions. The practice
| of treating every thread as a complete monolithic database kernel
| connected together with protocols that are good at equilibrium
| finding lends itself to this environment very well, and the edge
| (for which database technology is broadly broken), in addition to
| being ideal for the boring data center case where the topology is
| highly regular and every node has uniform resources.
|
| Most of our distributed systems are modeled as giant distributed
| file systems, and that fits okay with the idea of horizontally
| partitioning compute, storage, and network. This is easy for
| developers to reason about because they are familiar with file
| systems. However, it is difficult to express important database-y
| operations in this model. For example, if you want to do _ad hoc_
| joins in a scale-out environment, you need mechanics like
| decentralized parallel orchestration which isn 't really a
| concept in a "distributed file system" model of "distributed".
|
| Databases have been slowly converging on latency-hiding HPC
| software architectures (which has no implication of HPC
| hardware), which look very different than a distributed file
| system, can express more powerful capabilities, and are generally
| more scalable. To the extent there is a general principle and
| framework behind databases, it is more this than traditional
| distributed systems.
___________________________________________________________________
(page generated 2022-05-24 23:02 UTC)