[HN Gopher] Build your own fast, persistent KV store
       ___________________________________________________________________
        
       Build your own fast, persistent KV store
        
       Author : dineshgowda24
       Score  : 77 points
       Date   : 2023-02-14 18:27 UTC (4 hours ago)
        
 (HTM) web link (dinesh.wiki)
 (TXT) w3m dump (dinesh.wiki)
        
       | vitiral wrote:
       | How are the keys indexed and looked up? How does it handle disk
       | fragmentation? This has always seemed to me to be one of the
       | harder problems with databases.
        
       | robbs wrote:
       | Building your own database is a great exercise. There's lots of
       | topics to explore, like distributed computing, how indexes work,
       | caching, parsing, etc and you can pick and choose what to
       | implement.
        
         | ignoramous wrote:
         | Strictly speaking, building a database is more system
         | engineering than distributed computing, but folks skip right
         | past that when they opt to use "boltdb" or "rocksdb" or
         | "innodb" and so on.
        
       | samsquire wrote:
       | I wrote a simple dynamodb style database with a python dictionary
       | and a Google pygtrie ("trie" data structure)
       | 
       | It's still a toy but I kept adding features.
       | 
       | I then kept working on it and added distribution with consistent
       | hashing, rudimentary SQL joins, Cypher graph database queries and
       | document storage. You can even query documents with SQL.
       | 
       | I didn't get around to changing the graph storage to be
       | multimodal.
       | 
       | It takes very little code to write something with lots of
       | features.
       | 
       | https://GitHub.com/samsquire/hash-db
       | 
       | There's an AVL tree that farleyknight wrote and a btree that I
       | wrote that need to be integrated into it.
        
       | abadger9 wrote:
       | Thank YOU! I have literally been looking for something like this
       | to try to build a rocksdb-like project for fun. I've been waiting
       | to pull the trigger on YC's code crafters
       | (https://codecrafters.io/) which have you build your own redis
       | and mysql-lite compatible db. Looking forward to going through
       | this.
        
         | dineshgowda24 wrote:
         | Hey, thank you. If you are blocked on anything, check out the
         | GitHub repo I have added to the article. It has all the working
         | code.
         | 
         | TIL about code crafters. Looks promising; I will check it out.
        
         | itsmemattchung wrote:
         | This might interest you as well:
         | https://github.com/emichael/dslabs
         | 
         | That distributed systems lab is what Georgia Tech's Distributed
         | System lab[0] is based on, at least when I took the course back
         | in 2021
         | 
         | [0] - https://omscs.gatech.edu/cs-7210-distributed-computing
        
         | packetslave wrote:
         | Building a _distributed_ RocksDB-based system is also a fun
         | project... and practical, too! Large swaths of Facebook
         | services are built on top of ZippyDB, which is basically
         | "distributed RocksDB as a service"
        
           | LAC-Tech wrote:
           | I'd be curious to hear more about this. KV stores as a basis
           | for distributed databases are really interesting.
        
             | yawgmoth wrote:
             | https://www.igvita.com/2012/02/06/sstable-and-log-
             | structured...
             | 
             | May be of interest to you
        
       | codr7 wrote:
       | I've built several versions of a log based db with composite keys
       | over the years, the most complete version so far in Common Lisp:
       | 
       | https://github.com/codr7/whirlog
       | 
       | I've found that reinventing wheels is a great way to learn, even
       | if you never use them.
        
       | bachmeier wrote:
       | Not sure why "toy" was added to the title. It's not part of the
       | title of the post, nor should it be.
        
         | dineshgowda24 wrote:
         | Noted. Updated the title.
        
       | itsmemattchung wrote:
       | If building a distributed KV store interests you, you might want
       | to check out Georgia Tech's OMSCS Distributed Computing course:
       | https://omscs.gatech.edu/cs-7210-distributed-computing . While
       | rewarding, this class was hell and still gives me nightmares....
       | 
       | By the way, the course is built on top of Distributed System's
       | lab: https://github.com/emichael/dslabs
        
       | megiddo wrote:
       | [flagged]
        
         | simscitizen wrote:
         | Depends on whether the author is pedantic about the use of GB
         | vs. GiB.
        
         | JustSomeNobody wrote:
         | Read the rest and see. good grief.
        
       ___________________________________________________________________
       (page generated 2023-02-14 23:00 UTC)