[HN Gopher] Skipping the boring parts of building a database usi...
___________________________________________________________________
Skipping the boring parts of building a database using FoundationDB
Author : ovaistariq
Score : 38 points
Date : 2022-09-21 21:25 UTC (1 hours ago)
(HTM) web link (blog.tigrisdata.com)
(TXT) w3m dump (blog.tigrisdata.com)
| ovaistariq wrote:
| Building distributed database systems correctly from the ground
| up is a notoriously hard problem. We have seen this firsthand
| building Docstore at Uber.
|
| This is one of the most confusing aspects of the modern data
| infrastructure industry, why does every new system have to
| completely rebuild (not even reinvent!) the wheel? Vendors are
| spending so much time rebuilding existing solutions, they end up
| not solving the actual end users' problems, although ostensibly
| that's why they decided to create a new data platform in the
| first place!
|
| In this post we talk about our approach to building Tigris - the
| open source developer data platform. We talk about why we chose
| to build on top of FoundationDB, one of the most reliable
| distributed KV store with an amazing correctness story. We also
| go into detail about our experience using it.
| metadat wrote:
| Are you Himank or Yevgeniy?
|
| Good luck with your project, FDB fucked it's users back in 2015
| when it abruptly closed shop and went closed source. Hopefully
| some good can come of it yet.
| ovaistariq wrote:
| It has significant usage across large companies such as
| Apple, Snowflake, Epic Games, VMWare, etc. I don't see it
| going closed source. Besides that, here Tigris is taking the
| responsibility for the product as it is abstracting
| FoundationDB from the end-user.
| endisneigh wrote:
| I can assure you from FDB will not be closed sourced again.
| But I agree that it was pretty bad for its [open source]
| users that it was made closed source. Terrible move for
| adoption.
| endisneigh wrote:
| It's a shame that FoundationDB went closed source when it did,
| since that was the key period (imo) for database exploration -
| the early 2010s. If it would've been more popularized then I
| imagine most people would be using it now.
|
| The nice thing about FDB is that after 3 plus nodes, you can
| simply add nodes using your cloud provider of choice and it
| scales pretty nicely while still giving your high availability
| and fault tolerance.
|
| It's pretty funny to me though to see this - I've been spending a
| few days building a simple database on top of FDB that supports
| indexes, secondary indexes and schema migrations backed by json-
| schema (very, very similar to this, totally independently!)
|
| To get into a little bit, it's not super difficult if you use
| FDB. FDB is a very bare key value store. It's incredibly low
| level. You don't even get a notion of collections. You have to
| implement everything yourself. what it does give you, however, is
| a giant hash map that will guarantee that items are in sorted
| order.
|
| so to build what I was describing it's easy:
|
| a collection can be a tuple to map: (your-app,
| your-collection, _id, your-model-id-number) => json
|
| e.g. (hn-app, users, _id, 1) => { _id: 1,
| username: endisneigh } (hn-app, users, _id, 2) => { _id:
| 1, username: reader }
|
| an index can be something like: (your-app, your-
| collection, your-field-to-index, index-value, _id, your-model-id-
| number) => json
|
| e.g. (hn-app, users, username, endisneigh _id,
| 1) => { _id: 1, username: endisneigh } (hn-app, users,
| username, reader, _id, 1) => { _id: 1, username: reader}
|
| Because FDB gives you transactions, you maintain the index by
| populating the keys according to the pattern above on your
| create* and update* operations.
|
| To do something like a schema migration, FDB gives you a
| get_range operation that you can use to find all keys that have a
| prefix. So what you'd do is store a value indicating that you're
| doing a migration into the database, iterate through the keys in
| a batch (so it's all a single transaction), update the value in
| the db saying what the last key you've migrated is, and continue
| until you've done all of the keys.
|
| A lot of stuff is pretty trivial once you assume the underlying
| semantics are solved. I've seen some interesting projects
| involving things like using FDB as a virtual file system for
| SQLite, but the problem with that is FDBs primitives are actually
| flexible, and so there are optimizations you can make if you
| built it using those primitives from the beginning, as opposed to
| using FDB as simple a key value store without taking advantage of
| the transactions.
|
| -------
|
| On another note, one idea I've had (feel free to steal) is to
| reimplement IndexedDB using FoundationDB. IndexedDB is also a key
| value store which supports transactions, like FDB. Obviously IDB
| is not networked.
|
| The idea is that if you can semantically map IDB with FDB, then
| you could use FDB as a store for IDB (scoped to the user, of
| course). And then any app that uses IDB for its storage (like an
| offline app) could use FDB as the backing without having to use a
| different set of data structures to actually represent the
| storage.
| typingmonkey wrote:
| > reimplement IndexedDB using FoundationDB. IndexedDB is also a
| key value..
|
| I did sth pretty similar last month: https://rxdb.info/rx-
| storage-foundationdb.html
|
| It supports indexes, mongoDB queries etc. to store and query
| JSON documents via RxDB on top of FoundationDB.
| endisneigh wrote:
| beautiful! I'll check it out. heh, I knew it was a good idea,
| I'm glad someone else thought so as well
| ovaistariq wrote:
| We love FDB for all the reasons you mentioned and decided to
| build on top of it. It is quite flexible and provides a low
| enough primitive to be used as a building block for systems
| like Tigris.
|
| However, if you are an application developer looking for a
| ready-made solution that you can plug-in as your application's
| backend, then FDB does require heavy lifting. For example, you
| would have to implement auth mechanism, query layer, schema
| management, and indexing. This is where Tigris comes into play.
|
| You have a very interesting idea about backing IndexedDB APIs
| with FDB.
| endisneigh wrote:
| Oh absolutely! I love that you're taking this on. I noticed
| there was no documentation on schema migrations - I assume
| you just haven't added it, but I assume it's available or on
| the roadmap?
|
| Once you get all of your core functionality completed, you
| should definitely look at the IndexedDB APIs with FDB. I see
| you're considering FDB as a service. You could definitely
| compete with Firebase if you had some admin primitives around
| ACLs and you reimplemented the IDB APIs with FDB.
|
| For instance, you and I both are on two computers obviously.
| We could each have a Tigris instance. If your app is down, we
| fallback to the regular IDB api and everything is saved. You
| could save entire transactions that aren't persisted to FDB
| and replay them when FDB comes back up.
|
| More interestingly, as the admin, you could use all of the
| IDB tooling like LevelDB, PouchDB, absurdsql, etc and only
| concern yourself about the user (you and I) and things like
| how many keys they can save on the free plan, premium, etc.
| ovaistariq wrote:
| Schema migrations are supported but there are some
| restrictions. Here are some docs, we will be adding more
| details: https://docs.tigrisdata.com/overview/datamodel
|
| Assuming you have declared your schema as shown here
| https://docs.tigrisdata.com/typescript/getting-started You
| can evolve it by updating your type definitions, deploy the
| new version of application. Once `createOrUpdateCollection`
| is called, it will update the schema.
|
| --
|
| The IDB idea sounds very cool, let me dig into it more.
___________________________________________________________________
(page generated 2022-09-21 23:01 UTC)