[HN Gopher] FoundationDB: A distributed, unbundled, transactiona...
___________________________________________________________________
FoundationDB: A distributed, unbundled, transactional key value
store [pdf]
Author : wwilson
Score : 261 points
Date : 2021-06-07 16:37 UTC (1 days ago)
(HTM) web link (www.foundationdb.org)
(TXT) w3m dump (www.foundationdb.org)
| jwr wrote:
| I just implemented a database with changefeeds using FoundationDB
| (in Clojure), to eventually replace RethinkDB in my system. Very
| impressed so far.
| adamfeldman wrote:
| That's awesome! I'm interested specifically in using FDB with
| Clojure.
|
| Did you look at Crux as well? (DB written in Clojure, has
| primitives to build changefeeds, opencrux.com).
| krn wrote:
| Also, would it make any sense to use FoundationDB instead of
| Kafka as Document Store for Crux?
| adamfeldman wrote:
| This is actively being discussed in #crux on Clojurians
| Slack [1].
|
| It ought to show up within the archive [2] in a couple of
| days. Should already be in the Clojurians Zulip mirror as
| well.
|
| [1]: https://clojurians.slack.com/archives/CG3AM2F7V/p16231
| 025131...
|
| [2]: https://clojurians-log.clojureverse.org/crux
| jwr wrote:
| Yes, I did. But I had specific requirements, and the main one
| was that I need a fully distributed database, where one of
| the nodes can disappear for any reason at any time and things
| would just continue as if nothing happened.
| refset wrote:
| Hi, I work on the Crux team. I think "fully distributed"
| has a few possible meanings, but is it essentially a case
| of wanting something with a dead-simple clustering story?
| Or is it more about multi-region distribution &
| availability?
|
| Whilst Kafka itself is almost certainly not as simple to
| operate as FDB (although I can't speak from experience), it
| does in turn provide Crux with dead-simple clustering,
| because each Crux node acts as an isolated replica. FDB
| could be used equally instead of Kafka, but the maximum
| write throughput would then be somewhat lower. The only key
| part of the story Crux doesn't currently supply out-of-the-
| box is a load-balancing layer atop a cluster of nodes.
|
| I expect the real drawback of Crux's current design, by
| comparison, is that each node is an ~expensive full
| replica, whereas your system (built on FDB) will benefit
| from fully sharded indexes running across a cluster of
| (smaller) machines. The main trade-off then is the low-
| latency query performance of a fully local KV store.
| davgoldin wrote:
| Only great things to say about FoundationDB. We've been using it
| for about a year now. Got a tiny, live cluster of 35+ commodity
| machines (started with 3 a year ago), about 5TB capacity and
| growing. Been removing and adding servers (on live cluster)
| without a glitch. We've got another 100TB cluster in testing now.
| Of all the things, we're actually using it as a distributed file
| system.
|
| We've tried Ceph, GlusterFS, HDFS, MinIO and some others, and
| eventually decided on a custom FDB solution. It's a breeze to
| setup, and seems to eclipse others in performance [0] and
| reliability - Kyle (aphyr) the author of Jepsen series on
| distributed systems correctness, said: "haven't tested foundation
| in part because their testing appears to be waaaay more rigorous
| than mine." [1]
|
| The way we use FDB, if anyone is interested, is we simply split
| files into small chunks (per FDB data design recommendations),
| and store all file's & folder's meta data in FDB such as byte
| count, create/access/write times, permissions, and a lot more.
| Folders are handled by the builtin Directory layer [2].
|
| [0] https://apple.github.io/foundationdb/performance.html
|
| [1]
| https://web.archive.org/web/20150312112552/http://blog.found...
|
| [2] https://forums.foundationdb.org/t/whats-the-purpose-of-
| the-d...
| tpetry wrote:
| That's a really interesting solution. Can you tell us more
| about it? Operating distributed blob storage systems is kind of
| fragile with every software i have yet tried.
| davgoldin wrote:
| FDB's Directory layer provides all you need to create and
| edit nested paths. What's left to develop is a file chunking
| and assembly part, and statistics if needed.
|
| The only reason you need chunking is because FDB has very
| clearly defined limits in their documentation, and one of
| those limits is value size - it can't exceed 100kB, and
| should be kept below 10kB for best performance.
|
| For statistics like folder byte count, you could use FDB's
| atomic increment operation.
|
| Here's a good starting point (not mine):
| https://forums.foundationdb.org/t/object-store-on-
| foundation...
|
| I'll be happy to answer specific question if you got any.
| tpetry wrote:
| What kind of read/write ratio are you using? And would your
| solution work for a write-heavy workload?
|
| Kafka has limits on the message size and i need a solution
| for storing large blobs (up to 10MB) at data ingestion on
| for a very short time until the job has been processed. So
| read/write ratio will be exactly 50% and there will be a
| high write load. Is FoundationDB capable for this specific
| task? Are there some knobs tuneable for acid? Perfect acid
| requirements would not really be needed, if everything is
| fsynced every second would be completely ok.
| davgoldin wrote:
| From own experience, I'd say FDB can handle it all, we've
| got 20% read 80 write during peak hours, and reverse: 80
| read 20 write other time.
|
| Without doing your own tests, here's per core numbers,
| from which you can extrapolate (e.g. via CPU mips)
| towards your own hardware: https://apple.github.io/founda
| tiondb/performance.html#throug...
|
| FDB is ACID as is shipped, you don't need to turn knobs
| to make it such. Toughest part is to figure out
| classes/roles of the system. Here are a couple of good
| starting points:
| https://nikita.melkozerov.dev/posts/2019/06/building-a-
| found... https://forums.foundationdb.org/t/roles-classes-
| matrix/1340/...
| jamesblonde wrote:
| Interesting! We have been doing the same thing with HopsFS for
| a couple of years. Except, we only store the small files in our
| database (www.rondb.com) - RonDB is a recent fork of MySQL
| Cluster (NDBCluster). Very small files (<1KB) are stored in
| memory in RonDB, small files (typically <128KB) are stored in
| NVMe disks in RonDB, and other files in HopsFS (which now
| stores its blocks in object storage (S3, ABS).
|
| We had a paper on it as ACM Middleware and it's open-source on
| github. Are you going to publish your solution?
|
| (Discussed here on HN:
| https://news.ycombinator.com/item?id=25149154 )
| jamesblonde wrote:
| Do you support atomic rename? Atomic rename on subtrees?
| Consistent directory listings?
| jbverschoor wrote:
| It's unfortunate that they went silent for years after the Apple
| acquisition. That period was key for database adoption. I have
| the feeling everybody kind of settled for pgsql.
| threeseed wrote:
| > I have the feeling everybody kind of settled for pgsql.
|
| That's probably because of spending time on this echo chamber.
|
| In reality everyone has likely been staying with the same
| databases they know and love but just moved to the cloud. It's
| why now AWS for example offers such a wide variety of databases
| e.g. MySQL, PostgreSQL, SQL Server, Oracle, MongoDB, Cassandra,
| Redis.
| eloff wrote:
| Those are two completely non overlapping use cases. If you can
| use pgsql for your problem, you have no business trying to use
| a distributed key value store instead. That would be at least
| as dumb as driving screws with a hammer.
| cwp wrote:
| Yeah, but there are quite a few efforts out there to extend
| PG into a distributed DB of one flavor or another. Some
| examples are YugabyteDB, CockroachDB, Aurora and Citus. It's
| a reasonable approach, but it's also reasonable to come at it
| from the other direction - build a SQL engine on top of a
| solid distributed key-value store. Contrafactuals are always
| dicey, but FDB vanishing behind the Apple wall of silence
| sure didn't help.
| eloff wrote:
| > Some examples are YugabyteDB, CockroachDB, Aurora and
| Citus.
|
| Of those the first two are not PG, they just share the wire
| protocol and try to be compatible at a SQL level. Aurora is
| not really distributed, it's replicated for availability
| and durability six ways at the block storage level. Citus
| is distributed as I understand it though.
|
| > but it's also reasonable to come at it from the other
| direction - build a SQL engine on top of a solid
| distributed key-value store.
|
| Sure, that's possible. I'm not talking about the wisdom of
| building your own relational database as the end goal, just
| that a distributed key value database and a SQL database
| don't have overlapping problem sets.
| cwp wrote:
| Sure quibble about the details. The point is, these are
| all attempts to make PG more scalable. Around the time
| these projects got started Foundation looked like
| abandonware. If it hadn't, it's possible that "how can we
| have really scalable SQL databases?" might have had
| Foundation as part of the answer.
| jen20 wrote:
| FoundationDB was proprietary software before the
| acquisition, and not open source, so I'd say close to "no
| chance".
| qaq wrote:
| Yugobyte is slightly altered PG "Frontend" with custom
| storage backend
| jen20 wrote:
| CockroachDB is exactly this - a SQL engine on top of a
| distributed key value store. It is not an extension of
| Postgres itself, it just speaks the protocol and implements
| many of the features.
| dinedal wrote:
| Does anyone know of the"sqlite connector" mentioned in this post
| ?
|
| https://opensourceconnections.com/blog/2013/05/06/does-found...
|
| It would be really cool to find it, if it's still out there.
| eyelovewe wrote:
| CouchDB 4 is built upon Foundation FWIW
| jbverschoor wrote:
| Didn't know, very happy to hear
| z77dj3kl wrote:
| This is a really good document:
|
| https://apple.github.io/foundationdb/data-modeling.html
|
| I have been studying these key-value stores with efficient range
| iteration lately (such as LevelDB, RocksDB, BigTable,
| FoundationDB, etc). This is a great reference on how to make such
| a simple abstraction do a lot of useful things.
| rubyn00bie wrote:
| Here's one of my favorite articles on FoundationDB, where it
| (FDB) passes Jepsen first try:
| https://web.archive.org/web/20150312112556/http://blog.found...
|
| > I ran FoundationDB Key-Value Store through every nemesis in
| Jepsen - including those that found failures in other databases -
| and FoundationDB passed all of them with flying colors.
|
| FoundationDB is one of the coolest pieces of technology I've used
| in the past decade. The tuple keyspace is incredibly useful, so
| are the multi-key transactions. I've physically killed the power
| on an FDB node and FDB cluster; multiple times (heh, home
| servers)... and _every_ time the cluster or node just comes back.
| gregwebs wrote:
| That's great that you are doing your own resiliency testing.
|
| Having someone other than those officially on the Jepsen
| project run the Jepsen test is a good start. However, many
| databases have claimed to run the Jepsen tests themselves and
| pass, but when there is an actual paid engagement for a
| distributed database there are always issues that are found.
| That's generally true even for unpaid official runs as well
| although Zookeeper did pass existing tests. Every database is
| different and the paid engagement will design specific tests
| designed to break the database in question.
| [deleted]
| kendallgclark wrote:
| This was the FDB team's stock demo in the early days. It's a
| killer move.
| jFriedensreich wrote:
| I am pretty sure that the new cloudant transaction/storage engine
| is also based on foundationDB, which powers a lot of things
| behind the scenes at ibm. And couchdb 4 with foundationDB storage
| engine is hopefully not too far out either. Lets see how long
| this whole transition takes, but i am still hopeful that the
| mindshare and motivation of apple, snowflake, ibm and apache
| community will lead to something great.
| jorangreef wrote:
| Markus Pilman from Snowflake did an awesome talk on
| FoundationDB's testing at CMU's Quarantine Tech Talks (2020), How
| I Learned to Stop Worrying and Trust the Database:
|
| https://www.youtube.com/watch?v=OJb8A6h9jQQ
| sgk284 wrote:
| Here's another excellent talk at Strangeloop on FoundationDB's
| simulation testing by Will Wilson in 2014:
| https://www.youtube.com/watch?v=4fFDFbi3toc
| jtdev wrote:
| I'd love to see a good primer on data models and scenarios that
| are well suited to FDB.
| selljamhere wrote:
| Their docs might be a good place to start.
| https://apple.github.io/foundationdb/developer-guide.html#da...
| sigstoat wrote:
| this is limited by your creativity and willingness to make
| tradeoffs.
|
| the only really general statement i can think of is that the
| "larger"/"longer" your transactions are, the harder a time
| you'll have getting it to cooperate with FDB. "small"/"fast"
| transactions will be easier to fit into its model.
|
| (to likely replies: this isn't an absolute, see all the quotes.
| yes things like redwood will alleviate some of this, but not
| all.)
| vvern wrote:
| IIRC fdb is fully optimistic concurrency control. It doesn't
| do any locking. If you have workloads which are highly
| contended, you'll need to do something in the layer above to
| coordinate. Otherwise, performance will be unbearable.
|
| This may be out-dated, please let me know if the story has
| evolved here.
| sigstoat wrote:
| if your transactions are conflicting heavily with each
| other, yeah, you'll have a bad time. and if everything
| synchronizes on some small set of keys, you'll have a
| really bad time. monitoring the transaction conflict rate
| on your cluster is important.
| georgelyon wrote:
| FDB is an awesome and unique piece of software (I attribute quite
| a bit of Snowflake's success to FDB). I've also had the pleasure
| of meeting some folks from the original team and they are true
| engineers. Does anyone know if/when Redwood (the new storage
| engine) has landed / will land?
| victor106 wrote:
| > I attribute quite a bit of Snowflake's success to FDB
|
| How so?
| foobiekr wrote:
| Snowflake is the biggest deployment of fdb in the world after
| iCloud.
| kendallgclark wrote:
| Founders are building a distributed systems simulation product
| now called Antithesis. My data fabric startup, Stardog, is a
| happy Antithesis early adopter customer. It's helping us
| reproduce and fix non-deterministic bugs deterministically.
| Good stuff.
| maxmcd wrote:
| Fascinating: https://antithesis.com/
| acjohnson55 wrote:
| Am I right that this is like a distributed form of something like
| LevelDB or RocksDB, which would be the underlying storage engine
| for a full database product?
|
| And/or would it be comparable to DynamoDB?
| gigatexal wrote:
| Didn't couchbase move to FDB for their underlying engine?
| one2three4 wrote:
| >> In its newest release, CouchDB [2] (arguably the firstNoSQL
| system) is being re-built as a layer on top of FoundationDB.
|
| That is impressive. Like a framework for implementing noSQL DBs.
| twoodfin wrote:
| Did they ever implement a SQL layer? They seemed like one of the
| only NoSQL products with the architecture to make it plausible to
| do so.
| mping wrote:
| There was an SQL layer but performance was sub-par IIRC. There
| was also a blog post somewhere explaining why it's probably not
| a good idea to build an SQL layer on top of a KV store, devil
| in details, etc.
|
| Edit: found it
| https://www.voltdb.com/blog/2015/04/foundationdbs-lesson-fas...
|
| Not sure what to think of it, I'm not a DB expert by any means
| but the post sounds plausible enough and the SQL layer is
| discontinued AFAIK. I guess with each new abstraction layer you
| leave some perf on the table.
| funny_falcon wrote:
| iirc, they did. But it wasn't opensourced.
| strangattractor wrote:
| With little or no admin and monitoring tools.
| polskibus wrote:
| What is the backup / restore story in FoundationDB? How does it
| compare to postgresql?
| ex3ndr wrote:
| Much much better. Single line backup/restore and Disaster
| Recovery mode that syncs second DC and able too switch on the
| fly with barely any configs (except one file).
| e12e wrote:
| This seems like a good place to ask - are there any new and
| exiting FOSS "application" worth checking out? I recall from the
| initial publication of the source - there was references to a
| great sql layer? I don't know if a FOSS work-a-like ever
| materialized? Other things I'd hoped for was a network
| filesystem/blob layer, like maybe s3/nfs/webdavfs compatible?
| What are people building on top of foundationdb today?
|
| Ed: i suppose various document/db applications - like IMAP might
| be a good fit too?
| jFriedensreich wrote:
| large unstructured blobs and large files are among the things
| not well suited to foundationdb and couchdb 4 actually reduced
| supported blob size in the transition to foundationdb. it looks
| like object/blob storage systems are at the moment rather
| seperating more from key/value and document storage than
| growing together. but this is a good thing because the
| tradeoffs are very different and it allows each system to focus
| on what it does best. blob stores will hopefully move even more
| to content addressing and merkle dag similar to git and ipfs.
| mirker wrote:
| Can you elaborate on what requirements these blob systems
| should have?
|
| My understanding is object stores are typically "flat" by
| design to scale well (in contrast to a tree structure found
| in filenames).
|
| For content addressing, are people using the keys in
| sophisticated ways or are the values being indexed? Any
| reason to push this complexity into the storage layer as
| opposed to composing the functionality?
| jFriedensreich wrote:
| well there is often no hierarchy as in folders on an old
| school filesystem but if the system uses chunking, the
| organization of blob chunks is very important for the
| performance and scaling characteristics. The chunking
| algorithm needs to be performant but also lead to sensible
| chunk size and count and in addition can also do data based
| boundaries so chunks can be reused even if blob data
| changes at the start of the data. This can be different
| depending on your specific application (eg. the read/write
| ratio and average file sizes) and requirements for optimal
| use of the underlying filesystem, that's one reason why no
| de facto standard chunker has been established so far.
| There are many tradeoffs for key organization too. Do you
| need more sophisticated range queries or only single keys?
| How balanced is growing and shrinking of your data
| structure vs performance? What is the clustering story? How
| do you handle rebalancing/cleanup/pruning? Is your primary
| key organization content hashes like in ipfs or more
| arbitrary strings as in s3/minio? Is your metadata/
| secondary keys system completely integrated or more
| independent?
|
| Thats exactly what your last questions points to. If you
| are lets say dropbox and have probably a super
| sophisticated key value store setup i can imagine you would
| want your content addressable layer to be as simple and
| narrowly optimized as possible and develop and optimize the
| indexing, metadata and key queries system nearly completely
| separately. If you are working on some system that also
| should scale down to run on individual machines like ipfs,
| git annex or minio before their focus on kubernetes you
| want a system that can run as a single daemon but also
| where users can reason about the whole system as an
| integrated concept.
| mirker wrote:
| I see. Some of these, like ipfs, are more general purpose
| systems (basically communication protocols) than I was
| thinking of.
| agency wrote:
| I'm curious about this as well. Is anyone working on building
| text search on top of FDB? It's kind of astounding to me that
| last time I checked Elasticsearch was still essentially the
| only game in town.
| jFriedensreich wrote:
| its pretty hard to catch up with lucene, there is just so
| much work, features and brainpower in there at this point. as
| many features of foundationdb such as the transaction
| guarantees and reliability are not super important for
| fulltext search i cannot imagine any company even apple or
| ibm being able to justify that gigantic investment, instead
| im sure nearly any soluion willcontinue to use lucene under
| the hood for the forseeable future.
| bpicolo wrote:
| Tantivy is giving it a good go: https://github.com/tantivy-
| search/tantivy
|
| The good thing about Lucerne existing is you're allowed to
| also use their good ideas.
| jng wrote:
| Lucene is Java, right? There should be space for a native
| implementation, like ScyllaDB is doing to Cassandra (and
| DynamoDB, though the gap is not of the same shape in the
| last case). Or am I missing something?
|
| I used ElasticSearch and run one cluster in production in
| the past and found it horrible. Maybe I'm missing
| something, since they're so successful even as a public
| company...
| jFriedensreich wrote:
| yes its java and also yes there is definitely a space for
| a native implementation, its not that i don't want that
| to happen, on the contrary. but the reality is that its
| really hard to do and i dont see only getting rid of java
| to currently motivate a relevant player or large enough
| dev community.
| jFriedensreich wrote:
| just out of interest, what was horrible about es?
| vosper wrote:
| There's a lot to Lucene, and there's also a lot to
| ElasticSearch. And I think they're fairly tightly
| coupled.
|
| But I do think that a well-funded and skilled startup
| team could take a run at ES. They're a monopoly in their
| niche. There has to be money in disrupting them.
| jFriedensreich wrote:
| im sure its possible in theory, but without a permissive
| license it would be not relevant to most applications
| that are interesting. now try convincing a venture backed
| startup to build a lucene alternative and license it
| permissively after seeing what happend with amazon and
| es...
| sigstoat wrote:
| peruse the fdb forum. they produce document and record layers
| now. there are community layers of varying quality for a
| network block device, a filesystem, and a few other things.
| AtlasBarfed wrote:
| They got acquihired by apple, didn't they? Was. Fdb ever oss'd?
|
| Is it CP or AP? Comments seem to imply AP
| ssgao wrote:
| FoundationDB is Apache 2.0
| https://github.com/apple/foundationdb/blob/master/LICENSE
|
| It is CP per https://apple.github.io/foundationdb/cap-
| theorem.html
| kendallgclark wrote:
| It wasn't an acquihire. Apple paid a lot of $$ for FDB.
| [deleted]
| grouphugs wrote:
| was slightly excited about this, but the whole apple thing just
| ruined that
| ryanworl wrote:
| Two quotes from the paper that I think will motivate people to
| read it:
|
| "Rigorous correctness testing via simulation makes FDB extremely
| reliable. In the past several years, CloudKit [59] has deployed
| FDB for more than 0.5M disk years without a single data
| corruption event. Additionally, we constantly perform data
| consistency checks by comparing replicas of data records and
| making sure they are the same. To this date, no inconsistent data
| replicas have ever been found in our production clusters."
|
| "For example, early versions of FDB depended on Apache Zookeeper
| for coordination, which was deleted after real-world fault
| injection found two independent bugs in Zookeeper (circa 2010)
| and was replaced by a de novo Paxos implementation written in
| Flow. No production bugs have ever been reported since."
| jeffbee wrote:
| Ehhhh, doesn't align with my experience. I think FDB is
| actually really poorly tested. When I was evaluating it for
| replacement of the metadata key-value store at a major, public
| web services company we found that injecting faults into
| virtual NVMe devices on individual replicas would cause corrupt
| results returned to clients. We also found that it would just
| crash-loop on Linux systems with huge pages, because although
| someone from the project had written a huge-page-aware C++
| allocator "for performance", evidently nobody had ever actually
| tried to use it, including the author.
|
| It's also really, really weird that their non-scalable
| architecture hits a brick wall at 25 machines. Ignoring the
| correctness flaws, it only works if you can either design
| around that limit by sharding, and never off cross-shard
| transactions, or if you can assure yourself that your use case
| will never outgrow half a rack of equipment.
| fnordpiglet wrote:
| Can you fix a point in time? Software evolves and I think a
| point I saw is that it wasn't well tested then they changed
| once production workloads told them it needs to change.
| bpicolo wrote:
| What were the strong contenders?
| rbranson wrote:
| Were there other distributed databases that did pass the
| fault injection testing?
| jeffbee wrote:
| There weren't any, which is why that particular shop
| elected to roll their own distributed system on top of
| rocks.
|
| In general I think people who think they want to do
| FoundationDB owe themselves a serious contemplation of the
| cost/benefit of using Cloud Spanner instead. Obviously you
| cannot do your own fault injection testing of Spanner, but
| it does have end-to-end checksums.
| sigstoat wrote:
| > There weren't any, which is why that particular shop
| elected to roll their own distributed system on top of
| rocks.
|
| that's nuts. rocks could've been added as a storage
| engine to fdb far more easily.
| ryanworl wrote:
| This is currently in progress right now.
|
| https://github.com/apple/foundationdb/blob/e7d7b39f12afa8
| ea2...
| jeffbee wrote:
| For the record, I said the same thing. But it's a
| management problem because on the one hand you have a
| known open project with demonstrable flaws, and on the
| other you have your own in-house developers and you will
| tend to discount the bugs they haven't written yet.
|
| But, also for the same record, thinking you can implement
| a reliable, globally-replicated key-value store on top of
| FoundationDB that is cheaper and better than Cloud
| Spanner may be evidence of the same cognitive bias.
| sigstoat wrote:
| > But, also for the same record, thinking you can
| implement a reliable, globally-replicated key-value store
| on top of FoundationDB that is cheaper and better than
| Cloud Spanner may be evidence of the same cognitive bias.
|
| man, good thing nobody made any claim like that.
| sandinmyjoints wrote:
| What is the Flow referred to here?
| oconnor663 wrote:
| It's an async/await framework for C++. I'm not sure what the
| best source on this is, but here's a discussion:
| https://forums.foundationdb.org/t/why-was-flow-
| developed/171...
|
| My understanding is that FDB relies heavily on deterministic
| simulations for testing, and that their async/await model is
| a big part of how they make sure they cover different
| possible interleavings in a deterministic way.
| jorangreef wrote:
| Thanks for the quotes, I've been wanting to read this paper for
| some time. Great to see they went through the consensus
| literature and made a decision to go with Active Disk Paxos,
| instead of stopping short and not fully understanding the
| consensus they're building on. The consensus and replication
| protocol is such a huge part of building a distributed
| database.
| fizwhiz wrote:
| > de novo Paxos implementation written in Flow
|
| That's... brave. Flow is a DSL built on top of C++?
| alistairw wrote:
| Yeah it's their own language on top of c++ to help them with
| testing distributed systems with deterministic simulation.
|
| Their talk from a while ago about it was something that
| really blew me away at the time [0]
|
| [0] https://www.youtube.com/watch?v=4fFDFbi3toc
| Meai wrote:
| Personally I don't understand how you can call a database robust
| if it can't scale down nodes after you scaled them up once. What
| am I supposed to do if I ever deploy to 50 nodes and then it
| turns out that I only need 5. Shut the business down? Pay to run
| database servers forever that I don't even need anymore? Also the
| database configuration has a lot of gotchas and is very opaque.
| You might be waiting for 30sec for your CLI to connect to your
| _localhost_ cluster of two processes and you have no idea what is
| happening or why it is taking that long. It just never felt so
| safe and robust as people claim it to be. I don 't know, these
| were just my findings on the brief tests I did with it.
|
| Also you better get familiar with a whole bunch of hidden "knobs"
| that are apparently configurable and very important somewhere and
| then get printed out into xml logs but of course there is no log
| viewer so you have to write your own. Maybe this isn't a problem
| for large companies but I'm providing feedback as a single user
| here.
|
| I also don't understand how people can praise the c++ DSL. They
| should rewrite that into standard c++ coroutines as soon as
| possible so their entire build and dev environment isn't so hard
| to understand. As a user of open source software I generally like
| to be able to debug through the projects I use and figure out
| problems I have. It's much harder when a project uses their own
| custom language. I certainly tried to set it all up correctly but
| there always seemed to be some problems in regards to
| Intellisense within the IDE.
| __float wrote:
| https://apple.github.io/foundationdb/administration.html#rem...
|
| What is this process describing, and how does it differ from
| what you were trying to do?
| Meai wrote:
| They are using the term "machine" and "process" in multiple
| conflicting meanings I think. I mean maybe they improved this
| since I used it 1.5 years ago but I kind of doubt it.
|
| If I remember correctly there is a definite problem if you
| remove one of the processes that end up guaranteeing your
| configured redundancy mode. So then suddenly your entire
| cluster is inoperable. Yes really, I think it doesn't even
| properly respond to cli commands anymore and shows nodes as
| simply missing. Oh and suddenly those long wait times for
| your cli commands are really starting to bother you... I
| don't really want to talk more about it because it's been a
| while but I just wanted to make the point that the user
| experience was quite bad.
| monstrado wrote:
| Have nothing but praise for FoundationDB. It has been by far the
| most rock solid distributed database I have ever had the pleasure
| of using. I used to manage HBase clusters, and the fact that I
| have never once had to worry about manually splitting "regions"
| is such a boon for administration...let alone JVM GC tuning.
|
| We run several FDB clusters using 3-DC replication and have never
| once lost data. I remember when we wanted to replace all of the
| FDB hardware (one cluster) in AWS, and so we just doubled the
| cluster size, waited for data shuffling to calm down, and just
| started axing the original hardware. We did this all while
| performing over 100K production TPS.
|
| One thing that makes the above seamless for all existing
| connections is that clients automatically update their "cluster
| file" in the event that new coordinators join or are reassigned.
| That alone is amazing...as you don't have to track down every
| single client and change / re-roll with new connection
| parameters.
|
| Anyway, I talk this database up every chance I get. Keep up the
| awesome work.
|
| - A very happy user.
| maxpert wrote:
| How would it compare to say something like hosted Redis, or if
| you wanna be more fancy ElasticSearch. I have been looking into
| FDB for pretty long time and have been looking for a perfect
| opportunity to use it. Would be helpful if you can describe
| your usage scenario (kind of data you are storing).
| monstrado wrote:
| The key takeaways of FoundationDB is that it is a strongly
| consistent KeyValue store that preserves ordering
| (lexicographical). Although you might consider FDB's APIs to
| be quite primitive (get/set/scan), the payoff is how it
| seamlessly handles multi-key transactions without requiring
| you to write or manage some client side two-phase-commit
| process.
|
| Given these primitives, you or other engineers can write
| higher level APIs on top (e.g. sql, search, etc). In fact, we
| make extensive use of their RecordLayer [1] library which
| provides a strongly consistent schema based write process
| using Protobuf. This includes on-write-consistent indexes.
| Apple has also open-sourced a MongoDB API [2] compliant
| interface that allows you to get all the consistency
| guarantees from FDB, but with an API interaction of MongoDB.
|
| The beauty of FDB is that the primitives are done in such a
| rock solid fashion, you can write higher level APIs without
| having to worry about the really hard stuff (transactions,
| failures, config errors, testing, simulation, etc). Another
| example is that CouchDB is switching their back-end to use
| FDB for their 4.0 release.
|
| Given that the database sorts data lexicographical...to us,
| it became a natural fit for an online (always-mutable) time-
| series database.
|
| If you want more insight in how we use it, I go over it in
| some detail towards the end of my keynote [3] from last
| August.
|
| [1] https://www.foundationdb.org/blog/announcing-record-
| layer/
|
| [2] https://github.com/FoundationDB/fdb-document-layer
|
| [3] https://www.youtube.com/watch?v=93b--lTq2ng
___________________________________________________________________
(page generated 2021-06-08 23:03 UTC)