[HN Gopher] FoundationDB: A distributed unbundled transactional ...
___________________________________________________________________
FoundationDB: A distributed unbundled transactional key value store
Author : mlerner
Score : 243 points
Date : 2021-10-03 21:05 UTC (1 days ago)
(HTM) web link (www.micahlerner.com)
(TXT) w3m dump (www.micahlerner.com)
| didip wrote:
| Wouldn't the "unbundled" cause a lot of headache when trying to
| deploy it in production?
| noncoml wrote:
| Isn't this the one that disappeared overnight? Good luck getting
| the people's trust again
| Operyl wrote:
| https://github.com/apple/foundationdb
|
| https://github.com/foundationdb
|
| EDIT: I forgot they were a commercial product before Apple
| open-sourced it after acquisition.
| [deleted]
| commoner wrote:
| FoundationDB was pulled from market when Apple acquired it in
| 2015.[1] Apple re-released it as open source in 2018.[2]
| Since FoundationDB is FOSS this time around, it would be
| possible to fork it in case it becomes unavailable again.
|
| [1] https://news.ycombinator.com/item?id=9259986
|
| [2] https://news.ycombinator.com/item?id=16877395
| Operyl wrote:
| I completely forgot it was initially closed source, and
| only seriously considered it when it was opensourced in
| 2018. For the sake of the GP argument, though, I'd say that
| rebuilt an immense amount of trust in this case. My bad!
| lokar wrote:
| It was a private company, was bought by Apple, and is now
| Apache 2.0 with major contributions from outside Apple.
| srott wrote:
| Perhaps you were referring to DabbleDB?
| [deleted]
| aptxkid wrote:
| I wrote a post explaining FDB as well, which got retweeted by
| @FoundationDB
| https://mobile.twitter.com/aptxk1d/status/141786976657708237...
|
| My post focuses on the correctness, proof of the correctness of
| FDB's failure recovery, some of which is missing even from the
| paper itself.
|
| Some of the paper authors reached out; and I corrected one issue
| on my blog post pointed out by one of the authors.
| fiddlerwoaroof wrote:
| One thing I've never been able to figure out with FoundationDB is
| how to deploy it and a good list of layers for various use cases
| (SQL, document store, etc.)
| davgoldin wrote:
| These two resources helped me a lot in understanding how to
| setup FoundationDB processes correctly:
|
| https://nikita.melkozerov.dev/posts/2019/06/building-a-found...
|
| https://forums.foundationdb.org/t/roles-classes-matrix/1340/...
| dilyevsky wrote:
| Anyone has real experience running this in prod vs tikv? Latter
| seems like a more familiar design but missing watches which fdb
| seems to support
| taywrobel wrote:
| Former Apple employee here. They've been using it extensively
| in production for years. Pretty much all of CloudKit is run
| using FDB for storage.
|
| Public source for that info, so I don't get accused of NDA
| violations - https://youtu.be/HLE8chgw6LI
|
| Timestamp 14:15 touches on CloudKit's extensive FDB usage.
| saagarjha wrote:
| Nobody here is going to accuse you of violating your NDA :P
| ralusek wrote:
| I might, but it's mostly because you said that I wouldn't
| davgoldin wrote:
| One of PingCap's (company behind TiKV and TiDB) founders once
| did his own FoundationDB review [1] - he concluded: "the
| performance is cool". I'm assuming at that time FDB's
| performance was superior to TiKV.
|
| [1] https://medium.com/@siddontang/benchmark-foundationdb-
| with-g...
| gregwebs wrote:
| Disclaimer: I work for PingCAP. That benchmark didn't compare
| to TiKV. We generally avoid making such comparisions not
| because our competitors are better but because we know we
| will always have difficulty fairly benchmarking. Instead we
| generally do what is shown in that blog post: note in what
| aspect of a banchmark other databases are showing strength:
| because that indicates an opportunity to learn.
|
| We don't view ourselves in competition with FoundationDB
| because most TiKV users want the ability to use TiDB with
| TiKV for SQL (and MySQL compatibility). If you know you don't
| ever need that then FDB can be a good choice (but TiKV can be
| as well).
| foobiekr wrote:
| We do, and while I love it, a word of warning: do not allow
| production FDB clusters to have any member get above 90%
| storage consumed.
| OctoberTime wrote:
| I've only ever tinkered with tikv, but one advantage that it
| currently has over fdb is its support of co-processors --the
| ability for each node in the cluster to run some piece of query
| logic locally on its own shard of data instead of having all of
| the data shipped to one node (or even all the way up to the
| application level) for processing, which is quite bandwidth
| intensive.
| endisneigh wrote:
| Once CouchDB switches to using FoundationDB I think it'll surpass
| most NoSQL stores. Can't wait. The main issue right now with
| FoundationDB is that it's very nontrivial to setup and interface
| with IMHO.
| option_greek wrote:
| Any announced date for when this switch might happen? Overall a
| good decision by Couchdb community.
| tehbeard wrote:
| I'm not sure when FDB will get easier soon, if ever.
|
| Certainly had a devil of a time trying to get it to work on
| windows.
| speedgoose wrote:
| I also look forward to CouchDB 4. But CouchDB 2 and 3 has been
| very easy to setup and use over the years. Hopefully it stays
| the same.
| jwr wrote:
| I'm migrating from RethinkDB to FoundationDB right now. One thing
| which is easy to miss when you look at FoundationDB is that it is
| more of a toolkit for building databases than a database in
| itself. So, if you're looking for a faster horse, say a "better
| PostgreSQL", or a MongoDB that doesn't lose your data, it will
| seem disappointing ("where are all the features I expected?").
|
| Where I think it really shines is when you couple it tightly with
| your application. Suddenly you get a fully transactional
| distributed database with clearly understandable semantics, and
| if you can show some flexibility in how you map your data onto
| the underlying primitives, you can get fantastic performance, as
| well as correctness.
| CodesInChaos wrote:
| FoundationDB has rather severe limitations:
|
| > Transaction size cannot exceed 10,000,000 bytes of affected
| data
|
| In my experience almost all transactions are smaller than that,
| but the there is a long tail of large ones even for the same
| operation (because some 1:n link usually has a small n, but
| occasionally has an n which is orders of magnitude larger).
|
| > FoundationDB currently does not support transactions running
| for over five seconds.
|
| Limiting the duration of write-transactions isn't too bad. But
| being able to use a longer lived readonly snapshot is very
| convenient.
|
| Do higher level databases building on FoundationDB (e.g. the SQL
| layer) typically work around some of these limitations, or do
| they leave it to the application to avoid these?
| Diggsey wrote:
| The size limit makes a lot of sense when you look at the
| transaction model. When you write to FDB, none of the writes
| are even sent to the server until you attempt to commit the
| transaction, so this limit is actually a "maximum request size
| limit" on the final commit request.
|
| AIUI, the transaction size does not include values that are
| read, and for snapshot reads does not include the keys either,
| so this 10 MB limit only really constrains transactions that
| are writing a large volume of data.
|
| For those transactions, the documentation suggests writing the
| data first (using manny smaller transactions) and then only
| updating a pointer to that data in the final transaction.
|
| I'm currently building a simple database I'm calling AgentDB on
| top of FDB. It implements a message-passing system within the
| database, with guaranteed exactly-once semantics. It's designed
| to allow business logic to be more easily expressed without
| having to worry about all the failure modes typically present
| in a distributed system.
|
| For this, I process messages in batches, with one transaction
| per batch. If a transaction fails, I retry with a smaller batch
| size, so in my case the answer would be "yes, to an extent".
| The user of AgentDB still has to ensure that processing a
| single message doesn't exceed the transaction size, but they
| don't have to worry that a batch of messages would exceed that.
| maxmcd wrote:
| The Record Layer provides mechanisms to get around the key size
| limitations and the the transaction time/size limitations.
|
| See section 4: https://www.foundationdb.org/files/record-layer-
| paper.pdf
| qaq wrote:
| If that is severe limitation how would you characterize
| DynamoDB?
| OctoberTime wrote:
| I'd love to know how Apple distributes its CloudKit data across
| all of (what I assume are) its dozens of globally distributed
| FoundationDB clusters.
|
| As of version 6.0, a single FoundationDB cluster currently only
| supports one active, writable 'master' region + one hot failover
| region. Transactions are all essentially LAN so performance
| characteristics are great. It doesn't support the slower
| TiDB/CRDB style active/active configuration where all nodes are
| writable, data is globally distributed and transactions can span
| across all nodes in the cluster.
|
| Do they have some kind of scheme whereby ACID is preserved for
| local transactions only, while far away regions are
| asynchronously replicated to with less guarantees on data
| consistency?
| [deleted]
| adamfeldman wrote:
| There were 2 FDB papers published recently - this one discusses
| some of that: https://www.foundationdb.org/files/QuiCK.pdf
| Operyl wrote:
| I suspect it's because they're not just running a single
| cluster, but many many clusters with customers distributed
| globally. They had a talk about how they do it with other
| systems some time ago that I can't find at the moment.
| foobiekr wrote:
| Maximum fdb cluster size is actually not that large, in host
| count terms, and the maximum practical size per host is
| similarly not large for the Apple scale, so they are almost
| certainly sharding objects across hundreds or thousands of
| clusters.
|
| This has nice properties for blast containment.
| drenvuk wrote:
| Keep in mind that while Foundationdb has its place and it's
| really good at what it does it's ssd only. If you're running on
| HDDs for cheaper storage you're better off with something else.
| jiggawatts wrote:
| A vaguely similar project that might be of interest is:
| https://github.com/microsoft/FASTER
|
| It's also an "unbundled" low-level component that one could use
| as the foundation for a database engine or whatever. According to
| Microsoft, FASTER is not just "fast", but significantly faster
| than even some basic in-memory data structures that ship in the
| .NET standard library!
|
| The downside is that it doesn't (yet) support some more advanced
| features like multi-server distributed mode.
|
| However, that relative simplicity may be preferred in some
| scenarios...
| pas wrote:
| Is it a LevelDB-like log structured merge tree?
|
| Their distributed checkpointing/recovery thing (built on
| FASTER) is very interesting!
|
| The graph in the upper right corner of page 9 is certainly
| impressive: https://tli2.github.io/assets/pdf/dpr-
| sigmod2021.pdf (it is indexed with number of virtual machines)
| SteadBytes wrote:
| The CMU Vaccination Database Talks series includes an excellent
| talk on FASTER from one of the authors:
| https://youtu.be/4Y0oNFud-VA
| jiggawatts wrote:
| Thanks for that link.
| chalcolithic wrote:
| If anyone can use it for anything other than session storage
| I'd love to hear about that. Their API is so limited that I
| just don't understand how can I possibly use it.
| PostThisTooFast wrote:
| It's "key/value store.'
|
| It's sad that the slash has become too hard for people.
| throwaway984393 wrote:
| In theory, their design makes sense: separating the component
| pieces makes it easier to adapt to different systems/scenarios,
| it allows you to more succinctly map out the architecture (and
| hopefully simplify interfaces), you have that whole testing
| framework you can use on the individual pieces, etc.
|
| The problem is with implementation. The more you separate
| components, the more code you have to add to allow them to
| interoperate. The more code, the more complexity. The more
| complexity, the more bugs. In addition, when you continue to
| separate components out into new failure domains that have a
| higher probability of failure ("3 components running on 1 node"
| -> "3 components running on 3 separate nodes") you increase the
| chances of problems even more. So, while the design looks nice in
| theory, and I'm sure they have some wonderful reports about how
| thorough their test framework is, in practice it might be a tire
| fire (depending on how it's actually used).
|
| Some people are going to go, "I've been using this in production
| for a year, it's rock solid!" Well, people said that about Riak,
| until they wanted to set it on fire when they found out how buggy
| the implementation was at actually handling various errors and
| edge cases, and that they each had to hire dedicated Riak
| programmers just to fix all its problems. You can make large-
| scale Riak clusters work "at scale" if you hire enough on-call to
| constantly fight fires and monkey-patch the bugs and juggle
| clusters and nodes and repair indexes, and hand-wave outages and
| flaws as due to some other issue ("the cluster was completely
| hosed and had to be recovered from backup because the disks got
| corrupted and a node went down and a network partitioned all at
| once, it's not Riak's fault").
|
| (looking at this HN thread, it seems like my assumption may be
| correct: https://news.ycombinator.com/item?id=27424605)
| ctvo wrote:
| Do you work in distributed systems at a level where you
| understand their claims, and the significance of this work vs.
| a general dislike of their high level implementation choices?
| Not implying anything with the previous statement except that
| not all software people understand strict serializability, etc.
| in distributed systems. In this case, you need some
| understanding to critique the paper.
|
| And if you do work in distributed systems at that level, please
| elaborate in detail what you're trying to refute, because it's
| all "I _feel_ this should be bad. We need to be careful in
| trusting them! ", which is unconvincing.
| sigstoat wrote:
| > looking at this HN thread, it seems like my assumption may be
| correct
|
| that has got one guy complaining about fdb, and a couple of
| others saying positive things.
|
| so in short, you've got no experience with fdb, wrote a couple
| of paragraphs of speculation and references to other systems,
| and then declared victory, hoping nobody would read the other
| thread, i guess.
|
| i've written stuff against fdb, and i've seen it in non-trivial
| production. it's not a panacea, it's a useful point in the
| design space of databases, and does pretty well there.
| j16sdiz wrote:
| If this paper is to be trusted. FDB is tested in many failure
| mode and scenceio. I expect it may fail, but should fail in
| predictable way.
| throwaway984393 wrote:
| My point is that even if it fails in a predictable way _in
| test scenarios_ , it won't necessarily in real life. The
| paper & test framework are effectively going to trick people
| into trusting that the thing will work well, rather than
| going by actual observations of how it works in varied
| production environments. For the casual observer, who cares?
| But for the people about to spend millions of dollars on this
| tech to run it in production, I hope they aren't swayed
| purely by theoreticals.
|
| Remember a decade ago when NoSQL came out, and everybody was
| hooping and hollering about how _amazing_ the concept was,
| and people like me would go "Well, wait a minute, how well
| does it actually run in production?" And people on HN would
| shout us down because the cool new toy is always awesome. And
| lo and behold, most NoSQL databases are no better in
| production than Postgres, if not a tire fire. People who have
| fought these fires before can smell the smoke miles away.
| j16sdiz wrote:
| Their test framework uncovered a few zookeeper bugs.
|
| Zookeeper bugs are very rare even in production and hard to
| find.
|
| If they can uncover zookeeper bug, i think they did some
| serious testing.
___________________________________________________________________
(page generated 2021-10-04 23:03 UTC)