[HN Gopher] The end of a myth: Distributed transactions can scale
       ___________________________________________________________________
        
       The end of a myth: Distributed transactions can scale
        
       Author : luu
       Score  : 303 points
       Date   : 2023-04-11 02:33 UTC (20 hours ago)
        
 (HTM) web link (muratbuffalo.blogspot.com)
 (TXT) w3m dump (muratbuffalo.blogspot.com)
        
       | justinclift wrote:
       | > Is RDMA mature (robust/reliable) enough to use in distributed
       | transactions?
       | 
       | At least on Linux systems, RDMA has been pretty robust/reliable
       | for probably a decade, maybe more.                   > What are
       | the handicaps?
       | 
       | It works differently to TCP/IP, which everyone in IT has at least
       | passing familiarity with. So, it tends to be automatically passed
       | over unless people hit a situation where they're open to "exotic"
       | solutions.
       | 
       | That being said, there's a TCP/IP shim layer (IPoIB) available
       | which can be used by existing software to run on an IB network.
       | 
       | That shim layer though used to have a reputation for flakiness,
       | and it was (or at least used to be) measurably slower than using
       | native IB.                   > What are the reasons for slow
       | uptake on this?
       | 
       | Repeated self-inflicted foot-guns by Mellanox leadership or
       | perhaps their sales and marketing leadership is my best guess.
       | 
       | Mellanox adapters when brand new are priced fairly high for
       | network adapters, or at least they used to be. However, when a
       | network using them is upgraded to the next generation gear, a
       | substantial number of the old ones would commonly end up for
       | ~cheap resale on places like Ebay.
       | 
       | So, *nix DevOps staff ("Sysadmins" back in the day) and anyone
       | else that needed fast networking for their home labs and similar
       | would pick them up and figure out how use them.
       | 
       | Which of course meant over time an increasing group of people
       | familiar with IB, that when figuring out solutions for their work
       | places would then have enough confidence and knowledge to order
       | them.
       | 
       | Sounds like typical organic growth right?
       | 
       | <rant>
       | 
       | Except Mellanox leadership - or at least their Sales & Marketing
       | people - seemed to be _fucking horrified_ that people were buying
       | their expensive adapters cheaply on Ebay.
       | 
       | So, while the Mellanox technical people were receptive to this
       | ground swell growth in usage by "unofficial" people, and tried to
       | help out, their leadership time and time again did their level
       | best to stamp it out.
       | 
       | Including killing off one of their decent "Community" growth
       | initiatives -> just turned it into a fucking marketing channel
       | for press releases and other crap <- Ugh.
       | 
       | But also doing things like instructing their support staff to not
       | answer anyone on their forums who seemed to have purchased their
       | gear through unofficial channels, etc. Nor let anyone else do so.
       | 
       | There were many, _many_ examples of this bullshit over the years.
       | And they were always like  "Why don't we have huge adoption?"
       | 
       | Gee, I wonder? :( :( :(
       | 
       | </rant>
       | 
       | Anyway, I gave up on them and moved on a few years afterwards,
       | prior to Nvidia buying them. I still buy older Mellanox
       | ConnectX-3 (VPI) adapters off Ebay occasionally for home gear use
       | (with Linux), and they're still solid. 10/40GbE. :)
        
       | samsquire wrote:
       | Thank you for this.
       | 
       | I implemented multiversion concurrency control in Java with
       | threads but I want to raise it to multimachine.
       | 
       | The problem I have is that the read and write timestamps need to
       | be available to detect if a transaction between machines
       | conflicts.
       | 
       | How do I synchronize read and write timestamps with minimal
       | latency?
       | 
       | When a database replica receives a transaction for key X, it
       | needs (a) a timestamp that is globally accurate (b) needs to tell
       | other replicas about it so they can detect dangerous read-write
       | dependencies.
       | 
       | I feel it inherently requires serialisation of timestamps to
       | accurately determine what other nodes are doing with data.
       | 
       | Otherwise you get dangerous read-write skew.
       | 
       | (See the whitepaper "Serializable Snapshot Isolation")
       | 
       | Load balancers don't do much work, except shift traffic. I wonder
       | if a load balancer that collects timestamps and enriches requests
       | with timestamp information would be a scalable solution?
        
         | foota wrote:
         | This is an example of a system built like you're describing at
         | the end: https://www.pingcap.com/blog/how-an-open-source-
         | distributed-...
         | 
         | Also see https://tikv.org/deep-dive/distributed-
         | transaction/timestamp...
         | 
         | Cockroachdb describes their approach here:
         | https://www.cockroachlabs.com/blog/living-without-atomic-clo...
        
       | mindwok wrote:
       | If we're bypassing TCP/IP, does that mean we need to build our
       | own protocol on top of this to achieve the reliability guarantees
       | etc provided by the TCP stack?
        
         | eternalban wrote:
         | -- extracted from:
         | https://research.cs.cornell.edu/projects/Quicksilver/public_...
         | --
         | 
         |  _RDMA (remote direct memory access) is a zero-copy
         | communication standard.
         | 
         | RDMA is a user-space networking solution, accessed via queue
         | pairs: lock-free data structures shared between user code and
         | the network controller (NIC), consisting of a send queue and a
         | receive queue.
         | 
         | RDMA supports several modes of operation [such as] reliable
         | two-sided RDMA operations, which behave similarly to TCP. With
         | this mode, the sender and receiver bind their respective queue
         | pairs together, creating a session fully implemented by the NIC
         | endpoints.
         | 
         | Once a send and the matching receive are posted, the data is
         | copied directly from the sender's memory to the receiver's
         | designated location, reliably and at the full rate the hardware
         | can support.
         | 
         | A completion queue reports outcomes. End-to-end software
         | resending or acknowledgments are not needed: either the
         | hardware delivers the correct data (in FIFO order) and reports
         | success, or the connection breaks._
        
       | throwawaaarrgh wrote:
       | Wait, why aren't they just using an FPGA to offload processing?
       | How does a 100G programmable NIC with direct DMA not give them
       | the performance they need?
        
       | jzelinskie wrote:
       | The last discussion point from this article very much so rings
       | true for me, too. It's been how many years since the Spanner
       | paper and I still can't get a GCP VM with atomic clocks?
       | 
       | At least now I can provision Cloud Spanner as a managed service,
       | but is this the future of clouds keeping their services at an
       | advantage?
        
         | dangoodmanUT wrote:
         | Last I checked the VMs are hooked up to their internal atomic
         | clock time servers. They all have double digit microsecond
         | drift at worst IIRC.
        
         | deepGem wrote:
         | The atomic clock is not in the VM, it's a service from what I
         | understand similar to NTP but obviously much more localised.
         | 
         | I in fact think once you offer atomic clock as a service for
         | any public cloud, distributed transactions become a lot easier
         | to implement thereby disrupting Spanner.
        
         | withinboredom wrote:
         | Not GCP related, but I found some local (to my server) telcos
         | and reached out to their IT department and got access to use
         | their internal NTP servers. So my servers are now stratum 1,
         | connected to GPS and atomic clocks, doing skew via chrony. I'm
         | not paying a dime. I merely had to agree to keep my access low
         | and my servers peer with each other to keep their clocks within
         | a few nanoseconds of each other.
         | 
         | It's amazing what you can get, if you just ask. It's the latter
         | part you really want, which is doable with modern hardware
         | accelerated time stamping and a peering NTP system (like
         | chrony).
        
           | bjornsing wrote:
           | Time synchronization over a network with unknown latency
           | characteristics is fundamentally impossible.
        
             | dilyevsky wrote:
             | Drop the "over a network with unknown latency
             | characteristics" - it's cleaner
        
             | ikiris wrote:
             | ptp to the ToR isn't rocket science. If you need something
             | hyper precise and are a large enough customer, you can get
             | just about anything you need built in even the big clouds.
             | 
             | crazy customer needs is what made up most projects I saw or
             | worked with.
        
             | atoav wrote:
             | Really? It is now Tuesday in central europe and I guess
             | most machines1 managed to figure out that fact.
             | 
             | Most machines will also know it is 8:11.
             | 
             | The question is how much resolution do you need and are
             | your clocks accuratly synced enough for the thing you plan
             | to do.
             | 
             | You are aware that many of the physics experiments that
             | operate at the edge of what is possible use extremely
             | accurate clocks synced over national and sometimes
             | continental borders?
             | 
             | 1 let's ignore the ones that have been configured
             | incorrectly
        
               | rezonant wrote:
               | _Perfect_ time synchronization over a network with
               | unknown latency is provably impossible.
               | 
               | Fairly certain distributed physics experiments will be
               | using atomic clocks that are not synchronized over the
               | network. And/or they'll use direct satellite or GPS time
               | sources which have predictable latencies.
        
               | t0b1 wrote:
               | CERN at least seems to be using a solution that is
               | running over Ethernet[1] but with custom hardware that is
               | probably fairly expensive. They use a single time source
               | and then measure the delay between each switch/node.
               | Though, this is limited by needing to be able to run a
               | cable between each node so idk how your definition of a
               | distributed experiment fits.
               | 
               | [1]: http://white-rabbit.web.cern.ch/
        
               | sethammons wrote:
               | We've several thousands of nodes in our system. Our ops
               | people have witnessed clock skew from seconds to months
               | that are transient. Time is constantly synced via ntpd.
               | 
               | You can't trust system clocks in a distributed system to
               | ensure ordering. Some reading:
               | 
               | https://codeburst.io/why-shouldnt-you-trust-system-
               | clocks-72...
        
             | withinboredom wrote:
             | Wow. It's a wonder anyone has a correct time then. /s
             | 
             | Sarcasm aside, their NTP servers are less than 2 hops from
             | my router. I know that because of excellent tools like ping
             | and trace route.
        
               | protocolture wrote:
               | no-propagate-ttl really does clean up an ISP network.
        
               | rezonant wrote:
               | Based on your other post, it sounds like you are doing
               | NTP with your ISP and then using PTP to sync your servers
               | on the local network. Nothing about this implies that you
               | are accurate to the global clock within a microsecond,
               | let alone accurate within any number of nanoseconds.
               | 
               | With a correctly equipped and configured PTP installation
               | locally, you can expect your clocks to be synchronized on
               | the order of a very small number of _microseconds_ with
               | respect to each other, but the relationship between those
               | and the atomic global clock is something else.
               | 
               | If you are using satellite/GPS time sources, then that's
               | another matter. Why then is it relevant to have an NTP
               | hookup at your ISP?
        
               | withinboredom wrote:
               | With A ISP, not my ISP. Also, I'm syncing with a Stratum
               | 0 servers which are connected to an atomic clock and
               | several GPS clocks.
               | 
               | Also, consistency and accuracy are orthogonal to each
               | other. My servers clocks are consistent and fairly
               | accurate in regards to global time.
        
               | rezonant wrote:
               | Ah yes, I figured as much, this is why I skipped over the
               | GPS and satellite parts initially. Having a high quality
               | highly local NTP source is better than most sources, but
               | it is nowhere near as close as having your own local time
               | sources, which is what a lot of time sensitive
               | installations do.
        
           | readams wrote:
           | You are definitely not within nanoseconds using NTP
        
             | withinboredom wrote:
             | I don't think Chrony is NTP, if we are being pedantic, just
             | NTP compatible. My servers have hardware timestamping on
             | the network interface and chrony uses this.
        
               | rezonant wrote:
               | That would be PTP then
        
               | mlichvar wrote:
               | No, NTP and PTP are two different protocols. They can
               | both use hardware timestamps and reach single-digit
               | nanosecond accuracy in ideal conditions. The main
               | difference is in existing support in switches and
               | routers, which is needed to avoid the impact of
               | asymmetric delay between ports (typically tens of
               | nanoseconds per switch).
               | 
               | PTP has good support in higher-end switches and routers,
               | but it's difficult to secure and make resilient to
               | failures. It was designed for automation and control
               | networks in factories etc. NTP is a better fit for
               | computer networks, but there doesn't seem to be any
               | switches or routers with HW NTP support. If you really
               | need the best accuracy with NTP, you can find old 100Mb/s
               | hubs on ebay and create a separate network.
        
               | rezonant wrote:
               | Yes, PTP and NTP are indeed very different protocols.
               | 
               | There's no networking hardware timestamp support for NTP
               | because NTP has nothing to do with hardware timestamps.
               | 
               | PTP can be done _without_ hardware timestamps, but it was
               | designed with hardware support in mind.
               | 
               | I don't know where you got it that NTP does anything even
               | orders of magnitude close to nanoseconds:
               | 
               | > NTP can usually maintain time to within tens of
               | milliseconds over the public Internet, and can achieve
               | better than one millisecond accuracy in local area
               | networks under ideal conditions
               | 
               | https://en.m.wikipedia.org/wiki/Network_Time_Protocol
               | 
               | > The Precision Time Protocol (PTP) is a protocol used to
               | synchronize clocks throughout a computer network. On a
               | local area network, it achieves clock accuracy in the
               | sub-microsecond range, making it suitable for measurement
               | and control systems.
               | 
               | https://en.m.wikipedia.org/wiki/Precision_Time_Protocol
               | 
               | Literally neither solution comes anywhere near nanosecond
               | accuracy.
               | 
               | For reference, there are 1,000,000 nanoseconds in a
               | _millisecond_
        
               | mlichvar wrote:
               | > There's no networking hardware timestamp support for
               | NTP because NTP has nothing to do with hardware
               | timestamps.
               | 
               | Both NTP and PTP don't care (as protocols) where the
               | timestamps are coming from. That's an implementation
               | detail.
               | 
               | > NTP can usually maintain time to within tens of
               | milliseconds over the public Internet, and can achieve
               | better than one millisecond accuracy in local area
               | networks under ideal conditions
               | 
               | That was maybe 20-30 years ago, but not today. The
               | wikipedia article needs an update. If you don't hit a
               | routing asymmetry, in my experience it's usually
               | milliseconds over Internet and tens of microseconds in
               | local network if using SW timestamping. Please note that
               | NTP clients by default use long polling intervals to
               | avoid excessive load on public servers on Internet, so
               | they need to be specifically configured for better
               | performance in local networks.
               | 
               | You can find some measurements with HW timestamping here:
               | https://chrony.tuxfamily.org/examples.html
               | 
               | Note that this is for the system clock, which has to be
               | synchronized over PCIe to the hardware clock of the NIC.
               | That adds hundreds of nanoseconds of uncertainty. It
               | doesn't matter if the hardware clock is synchronized by
               | PTP or NTP.
               | 
               | If you care only about the hardware clocks, it's easy to
               | show how accurate is the synchronization by comparing
               | their PPS signals on a scope. NTP between two directly
               | connected NICs, or a with a hub, can get to single-digit
               | nanosecond accuracy. I have seen that in my testing. It's
               | just timestamps, it doesn't matter how they are
               | exchanged.
        
           | starfallg wrote:
           | In real world applications, NTP is accurate to the
           | millisecond, while PTP is accurate to the microsecond.
           | 
           | PTP is often used now in the Telco world as well as for
           | broadcasting applications.
        
         | cbsmith wrote:
         | I think the reality is making the SLA guarantees that Spanner
         | in a third party cloud is just... not very practical.
        
           | Cthulhu_ wrote:
           | That's it; there's plenty of competing products for Spanner,
           | but Spanner's USP is the host and their hardware.
        
       | pshirshov wrote:
       | From what I understand this doesn't change anything on
       | fundamental scale.
       | 
       | The particular timestamp oracle implementations still might
       | suffer from metadata bloating and its commit protocol assumes an
       | optimistic segment lock which would be a bottleneck in multiple
       | scenarios.
       | 
       | So, from what I understand this is a (good?) practical speedup
       | for some scenarios but doesn't change the fact that it's not
       | possible to make an arbitrary sytem distributed without
       | sacrificing correctness or throughput.
       | 
       | An easy reasoning: imagine that every node is one light year
       | apart from all others.
        
       | hanniabu wrote:
       | Ethereum rollups have proven that transactions can scale
        
       | jamesblonde wrote:
       | This is impressive, but won't have huge impact, IMO.
       | 
       | Way back in 2015, MySQL Cluster (NDB Cluster engine) benchmarked
       | 200m transactions/second on commodity hardware [1]. It was read-
       | committed transactions, not snapshot isolation, but still
       | impressive. NDB (or RonDB, the new DB by its author) uses a non-
       | blocking 2-phase commit protocol (failed transaction coordinators
       | are failed over) and is even open-source. Still, it hasn't had a
       | big impact outside of the Telecom and real-time gaming worlds.
       | 
       | [1] https://www.slideshare.net/frazerClement/200-million-qps-
       | on-...
        
         | vlovich123 wrote:
         | How many machines was that? This is doing it across ~35.
        
           | hardwaresofton wrote:
           | 105 machines with 28 cores each, says the presentation, and
           | over infiniband. Looks like RonDB is quite impressive.
        
         | _448 wrote:
         | Why not many are talking about RonDB? This is like Erlang of
         | the database world! People at Ericsson were really very smart
         | and ahead of their time.
         | 
         | FoundationDB, TiKV, CitusData's Postgres extension are very
         | well known, but RonDB looks like a hidden gem.
         | 
         | What are the practical issues with RonDB that it is not widely
         | known(or used?)?
        
           | giovannibonetti wrote:
           | Because RonDB is not a SQL database, it is just a KV store.
        
             | mikaelronstrom wrote:
             | RonDB is a KV store with SQL capabilities (the MySQL
             | storage engine NDB is maintained by Oracle as part of MySQL
             | NDB Cluster). Hopsworks is on top of this adding a new REST
             | API (REST + gRPC) service that is already in the github
             | tree and will ready for production usage in a few months.
        
             | doh wrote:
             | FoundationDB is not a SQL database either though.
        
             | _448 wrote:
             | Yes, but as per the document you can put a MySQL server
             | along with the Data server node and use SQL.
        
           | ergonaught wrote:
           | I don't discuss it because of its name, despite whatever its
           | technical merits may be and despite knowing that Mikael is a
           | very intelligent/capable person.
        
             | wwilim wrote:
             | Is it Ron to discuss it?
        
             | qaq wrote:
             | what is the problem with the name?
        
             | andruby wrote:
             | Why not? Is the name that bad?
        
               | ergonaught wrote:
               | I can't say whether the name is bad or not. I dislike it,
               | and I dislike it strongly enough to not discuss the
               | database, despite personally liking and respecting the
               | developer.
               | 
               | Simple and honest answer to the question and I have
               | probably the most downvotes I've ever gotten for
               | anything. Y'all are just plain broken.
        
               | Supermancho wrote:
               | You: > I don't discuss it because of its name
               | 
               | Also you: > I can't say whether the name is bad or not. I
               | dislike it, and I dislike it strongly enough to not
               | discuss the database
               | 
               | > probably the most downvotes I've ever gotten for
               | anything. Y'all are just plain broken.
               | 
               | You didn't add to the conversation, except adding your
               | own personal problems as noise. This is working as
               | intended. Hopefully this information will be helpful to
               | all of us, going forward.
        
               | ergonaught wrote:
               | Let's try this again.
               | 
               | Question is posed: What is the explanation for X?
               | 
               | Answer is provided: Here is my explanation for X.
               | 
               | You submit that this answer is "adding personal problems
               | as noise" and downvoted, so "working as intended".
               | 
               | If the relevance leap is hard for you, let's try this.
               | 
               | Question: We just launched a new product called
               | LAKJfkdshfdskjfwfdsfnbozg, but no one is talking about
               | it. Why doesn't anyone ever talk about
               | LAKJfkdshfdskjfwfdsfnbozg?
               | 
               | Answer: I don't talk about it because I don't like the
               | name.
               | 
               | Your response to this would be to downvote the answer
               | because it is a personal problem, noise, which does not
               | contribute to the conversation?
               | 
               | I hope you do not function in any capacity which requires
               | understanding of sentiment.
        
               | Supermancho wrote:
               | Let's not. GL with whatever.
        
       | vrglvrglvrgl wrote:
       | [dead]
        
       | piuantiderp wrote:
       | RDMA, sounds unsafe by design
        
         | withinboredom wrote:
         | What seems unsafe about it? Are you worried about auth? There's
         | sRDMA?
        
         | exabrial wrote:
         | Consider this: Is pcie insecure by design? Any tech could be
         | used in a stupid way, making it insecure.
        
       | kohlerm wrote:
       | That's a click bait title IMHO. Sure with some hardware changes
       | you can get different tradeoffs with regards to performance, but
       | that does not solve the distributed transactions problem
       | completely. E.g. What about availability?
        
       | [deleted]
        
       | hestefisk wrote:
       | Interesting paper. That said, I find the image at the top of the
       | article and the play on words in NAM-DB disturbing, considering
       | the many atrocities and war crimes the US committed in the
       | Vietnam War. The authors should have been more thoughtful given
       | the usefulness of the paper on pure technical merits.
        
       | fulafel wrote:
       | Is the RDMA nic offload important for scaling here in the big-O
       | sense? Does it eliminate some worse-than-linear-time operations?
        
       | Berengmark wrote:
       | Good!
        
       | juliangamble wrote:
       | I'm using Transactions on a Distributed Database with Cassandra
       | 4.0: https://medium.com/building-the-open-data-stack/an-apache-
       | ca...
       | 
       | I'm curious to see what advantages this paper offers to everyday
       | Cloud Developers over using the latest version of Cassandra.
        
       | jongjong wrote:
       | That sounds really good.
       | 
       | I think reasons for slow adoption are probably a mix of:
       | 
       | 1. Lack of developer awareness.
       | 
       | 2. Security implications (or perceived implications) of exposing
       | memory directly to a network without passing through CPU or
       | application-level access control mechanisms.
       | 
       | The second point may be prohibitive for a lot of general purpose
       | database systems which are intended to be run on shared
       | infrastructure on virtualized instances.
       | 
       | Another reason may be that a lot of production systems are CPU-
       | bound, not memory-bound. RDMA seems ideal for systems which
       | require a lot of memory. I'm thinking maybe with recent
       | advancements in AI/LLMs, it could be an interesting technology as
       | these do require a huge amount of memory relative to CPU.
        
         | jandrewrogers wrote:
         | There are several good ideas in distributed databases that are
         | effectively not deployable in cloud environments because the
         | requisite hardware features don't exist. Since cloud
         | deployability is a pervasive prerequisite for commercial
         | viability, database designs tend to overfit for the limitations
         | of cloud environments even though we know how to do much
         | better.
         | 
         | Basically, we are stuck in a local minima of making databases
         | work well in cloud environments that are not designed to enable
         | efficient distributed databases. It is wasteful and also
         | provides an arbitrage opportunity for cloud companies.
        
           | jpalomaki wrote:
           | What kind of hardware features are we missing on cloud?
        
             | withinboredom wrote:
             | - network packet timestamping via hardware
             | 
             | - this paper
             | 
             | - dedicated bandwidth (Azure gives you bandwidth based on
             | instance size)
             | 
             | - XDP on network interface
             | 
             | Probably more, but those are what I know of from running
             | into their non-existence.
        
             | deepGem wrote:
             | The atomic clock is the key to enable distributed
             | transactions and Google has a proprietary lock on their
             | atomic clocks for Spanner.
        
               | withinboredom wrote:
               | You need this for <em>global</em> consistency, but for
               | logical local (like a single entity) this is unnecessary.
        
               | deepGem wrote:
               | Yeah, you have a much better understanding of this topic
               | than me for sure :)
        
               | amitport wrote:
               | Do you know what is exactly proprietary in spanner? AFAIK
               | most (all?) of the ideas existed before in the
               | theoretical clock synchronization literature.
        
               | deepGem wrote:
               | Nothing proprietary in Spanner but I believe no other
               | vendor has an atomic clock similar to what Google has.
               | Hence they are not able to implement the paxos global
               | transaction lock which requires that all servers
               | participating in the transaction be perfectly
               | synchronized. This is what one of the comments above
               | refers to I think.
               | 
               | So while the spanner paper is open to be implemented by
               | any vendor, they don't have the proprietary advantage
               | that Google has - the atomic clock. So Yugabyte,
               | CockroachDB don't rely on atomic clocks. I tried to get
               | to the ground level basics of this, but I haven't
               | understood this matter completely yet.
        
               | amitport wrote:
               | I guess using worse clocks would mean using a (slightly?)
               | slower spanner, but I'm not sure what is the impact. In
               | any case, if a big vendor (e.g., Amazon IBM Oracle
               | Dell...) would want something on par with Google's clock
               | they probably can achieve it (though I don't know much
               | about these clocks).
        
               | di4na wrote:
               | The problem is that the slower it is the more the window
               | for error grows. And the harder it is to recover. It also
               | impose a high boundary to latency. You cannot be faster
               | than your clock error without risks.
               | 
               | Note that even Spanner had multiple downtime due to clock
               | and/or network failures. In these case, any operation lal
               | guarantees are lost. This makes it really dangerous.
        
         | jeffreygoesto wrote:
         | Do you have indications why CPU bound? I have never seen
         | Systems other than for example linear algebra or special
         | algorithms to max out a modern CPU. Almost all loads today are
         | in love way or another memory bound.
        
           | JackSlateur wrote:
           | Bah Systems are either limited by CPU (processing) or by CPU
           | (waiting for IO, especially memory)
           | 
           | Systems limited by memory as in "quantity of" are scarse
        
             | lightbendover wrote:
             | Only scarce since it's an easy distributed problem to solve
             | compared to IO.
        
         | cbsmith wrote:
         | Distributed transactions are invariably latency (usually
         | storage I/O) bound, rather than CPU or memory bound. I think
         | your #2 is a big part of the challenge with RDMA, as well as
         | the "trickiness" of the programming paradigm.
        
         | HopenHeyHi wrote:
         | The experimental setup involves using a cluster of 56 machines
         | connected by an InfiniBand FDR network
         | 
         | That bit might have something to do with it.
        
           | justinclift wrote:
           | Heh Heh Heh
           | 
           | A bulk purchase of ~60 FDR IB cards, cabling, and network
           | switches to support them sounds pretty expensive.
           | 
           | That being said, IB FDR gear is "older tech" now so the cards
           | and switches can commonly be found reasonably cheaply on
           | Ebay. The switches tend to be bloody loud though, so they're
           | not something you'd want nearby if you can help it.
        
           | necubi wrote:
           | Amazon will rent you infiband-class machines like the
           | C6in.metal (https://instances.vantage.sh/aws/ec2/c6in.metal)
           | with 200Gb/s of bandwidth. With EFA
           | (https://aws.amazon.com/hpc/efa/) you can use HPC features
           | like rDMA.
        
       | q3k wrote:
       | The important parts:
       | 
       | > The data is assumed to be randomly distributed to memory nodes
       | in the shared memory pool. Any memory node is equi-distant to any
       | compute node, and a compute node needs to reach multiple memory
       | nodes for transaction execution.
       | 
       | and also:
       | 
       | > Once a memory server fails, NAM-DB halts the complete system
       | and recover all memory servers to a consistent state from the
       | last persisted checkpoint. The recovery procedure is executed by
       | one dedicated compute server that replays the merged log for all
       | memory servers.
       | 
       | >This global stall is problematic for production, but NAM is a
       | research proof-of-concept system. Maybe using redundancy/quorums
       | would be a way to solve this, but then that could be introducing
       | challenges for consistency. Not straightforward.
       | 
       | Yeah, research and proof-of-concept sounds about right. Some
       | might even call it a toy database.
       | 
       | Also worth noting that this is a paper from 2016/2017. The world
       | of (actually!) distributed database design has significantly
       | moved on since then.
        
         | spenczar5 wrote:
         | > The world of (actually!) distributed database design has
         | significantly moved on since then.
         | 
         | What are some of the significant movements since then?
        
           | [deleted]
        
       | btown wrote:
       | The linked blog post at the top of this article -
       | https://muratbuffalo.blogspot.com/2023/01/is-scalable-oltp-i... -
       | provides graphics that give extremely useful context. And here's
       | the repo for the paper that discusses:
       | https://github.com/DataManagementLab/ScaleStore
       | 
       | The idea that one of many writer-compute-nodes can literally
       | reach into a memory buffer that is shared across machines,
       | atomically flip some lock bits and propagate some cache-coherence
       | messages, and use that to build a multi-writer distributed
       | database without needing to partition (and where any writer-
       | compute-node can handle any message, so you can just round-robin
       | a firehose of messages at them)... and that there's a chance
       | (though not yet implemented) that one could implement ACID on top
       | of this? It's absolute madness, and wildly exciting.
        
       | twhitmore wrote:
       | Haven't read the full paper, but several aspects of this sound
       | less than fully reliable.
       | 
       | It reminds me specifically of 90's multi-client LAN database
       | systems (dBase, Clipper) where clients coordinated via file
       | locks. Unreliability & hangs became a big problem for us.
       | 
       | In the summarized RDMA database, I'd be pretty concerned about
       | reliability & integrity:
       | 
       | 1) Crashed servers will leave records locked, and the system will
       | hang. 2) Question whether lock timeouts can be adjudicated
       | reliably. 3) Any errors in server behaviour can easily & widely
       | corrupt data across any other nodes. 4) Overall the RDMA
       | coordination makes me cautious. Can we really replace Paxos with
       | RDMA reliably? If not, problems squeeze out elsewhere. 5)
       | Proposed single-threaded recovery procedure sounds a hazardous
       | operational bottleneck. 6) I'm also cautious about coordination
       | requirements around recovery/ or to transact knowing that
       | recovery is not in process, unless we can show that can be
       | reliable & not add cost to the protocol.
        
       | m2f2 wrote:
       | Something like Oracle Exadata perhaps?
        
       | [deleted]
        
       | [deleted]
        
       | amitport wrote:
       | IBM IMS DB
       | 
       | Everything is direct memory access.
       | 
       | SNA protocol, No TCP. This is used since the 70s by many banks to
       | process high volume transactions
        
         | exabrial wrote:
         | Everything old is new again. The MongoDB people are still
         | convinced they "invented" NoSQL databases, but really just
         | poorly implemented an idea that
         | https://en.wikipedia.org/wiki/ADABAS had perfected while the
         | MongoDB hipsters were in diapers.
        
         | throwawayForMe2 wrote:
         | Parallel Sysplex with 32 mainframes in a cluster.
        
       | concerned_ wrote:
       | Now do it with shared nothing architecture. When you share all
       | this stuff you are twisting the meaning of "distributed
       | transaction"
        
       ___________________________________________________________________
       (page generated 2023-04-11 23:02 UTC)