[HN Gopher] Show HN: A Database Written in Golang
___________________________________________________________________
Show HN: A Database Written in Golang
Recently created a minimal persistent relational database in Go.
Main focus was on implementing & understanding working the of
database, storage management & transaction handling. Use of B+ Tree
for storage engine(support for indexing), managing a Free List (for
reusing nodes), Support for transactions, Concurrent Reads. Still
have many things to add & fix like query processing being one of
the main & fixing some bugs Repo link -
https://github.com/Sahilb315/AtomixDB Would love to hear your
thoughts
Author : Sahil121
Score : 267 points
Date : 2025-02-26 14:28 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| whalesalad wrote:
| How do you connect to it and actually use it? Does it behave like
| a SQL database? Row-based? Column-based? Good for analytical
| workloads? Document store? Redis/memcached database? What are
| it's strengths or weaknesses?
|
| Cool accomplishment in and of itself but hard for anyone here to
| really give you any criticism or feedback without understanding
| where it excels and how to work with it.
| zsoltkacsandi wrote:
| > Main focus was on implementing & understanding working the of
| database
|
| I think this clearly describes what was the goal.
| whalesalad wrote:
| Yep... but there are dozens of different types of databases
| out there. There is no way to look at this codebase and give
| any kind of feedback one way or the other without that
| understanding, particularly with zero usage examples. Which
| is what the OP is asking about.
| jasonlotito wrote:
| > There is no way to look at this codebase and give any
| kind of feedback
|
| Less than a minute and I know how to use it. It's not
| complicated. The source code is available and fairly easy.
| If you can't figure out how to use it in a trivial amount
| of time, you aren't going to be able to offer anything of
| value. When did HN go from being about interesting stuff to
| making bold, ignorant statements like "There is no way to
| look at this codebase and give any kind of feedback one way
| or the other without that understanding."
|
| You should know better.
| maxmcd wrote:
| It does not seem to speak SQL: $ git clone
| git@github.com:Sahilb315/AtomixDB.git Cloning into
| 'AtomixDB'... $ cd AtomixDB/ $ go run .
| Welcome to AtomixDB Available Commands:
| CREATE - Create a new table INSERT - Add
| a record to a table DELETE - Delete a record
| from a table GET - Retrieve a record from a
| table UPDATE - Update a record in a table
| BEGIN - Begin new transaction COMMIT -
| Commit transaction ABORT - Rollback
| transaction EXIT - Exit the program
| > create Enter table name: foo Enter column
| names (comma-separated): bar,baz Enter column types
| (comma-separated as numbers): 1,2 Enter indexes
| (format: col1+col2,col3, ... or leave empty): Table
| 'foo' created successfully. > get Enter table
| name: foo Select query type: 1. Index
| lookup (primary/secondary key) 2. Range query
| 3. Column filter Enter choice (1, 2 or 3):
|
| It looks like records are stored in rows:
| https://github.com/Sahilb315/AtomixDB/blob/64c95afa8e574595c...
|
| I do find the source to be well organized and quite readable.
| Especially if you runs the commands in the cli and then trace
| how they are each implemented.
| Sahil121 wrote:
| yes man it is currently a sql db, have to work on adding
| query language thanks for checking it out
| Sahil121 wrote:
| it is kind of a sql db but currently does not have the query
| lang (sql lang) & data is stored in tables similar to other
| relational DBs. Still have a lot of stuff to add & fix Will add
| more detail to it
| pighive wrote:
| Nice, did you follow a course or any resources to build it?
| Please share, I have a similar goal. Thank you.
| rebelmackerel wrote:
| I took a quick glance at the code, I believe it may build upon
| "Build Your Own Database From Scratch in Go" [0]. The first
| part of the book is available for free on the author's website,
| along with information on how to purchase the full book (which
| includes source code).
|
| I share the same goal and am working through the material after
| working through Codecrafters' "Build your own SQLite" [1]. Good
| luck!
|
| I apologize in advance for mistakes (formatting, et cetera). I
| just registered this account to point you toward resources I
| found helpful.
|
| [0] https://build-your-own.org/database/
|
| [1] https://app.codecrafters.io/courses/sqlite/overview
| croskcool wrote:
| Does something like this exist for java, don't want to take
| the burden of a new language.
| rebelmackerel wrote:
| I don't think these things really fit the bill, but this is
| closest I could find. All the codecrafters stuff is focused
| on reading from a sqlite DB. A great start, but you're not
| implementing B+ trees.
|
| Hope this gives you some options.
|
| https://github.com/Arminas42/build_your_own_database
|
| https://github.com/search?q=codecrafters+sqlite-
| java+languag...
| lionkor wrote:
| it's not that hard, you should try it :)
| mbreese wrote:
| This looks like a good exploratory project!
|
| One thing I'd add to the readme is an example of how you'd use
| the database in an example application. From the docs, it's clear
| that this isn't a sql database (yet?), so it would be good to
| have an example to see how to use the database.
|
| It might also be nice to have a description of what happens when
| you insert or get a record, so others can learn from the code
| too. Or can you comment here about what your favorite part of the
| code is? What did you figure out that you didn't know before? If
| you're using the project to learn about databases, what have you
| learned so far?
| Sahil121 wrote:
| thanks man yes currently it is not a fully sql database (have
| to add query support) My fav part of code is the data retrieval
| code because had some issues there & got to know lot about it
|
| Learnt about B+ Trees, transactions, managing concurrent reads
| & data presistence
| littlemerman wrote:
| You might be interested in this golang db from mit's database
| systems course:
|
| https://github.com/MIT-DB-Class/go-db-2024
| https://dsg.csail.mit.edu/6.5830/
| Sahil121 wrote:
| will surely checkout thanks man
| eatonphil wrote:
| If folks would like to see more examples of databases built to
| teach oneself, they get shared on the /r/databasedevelopment
| subreddit not infrequently.
|
| Some recent ones:
|
| https://www.reddit.com/r/databasedevelopment/comments/1hyig8...
|
| https://www.reddit.com/r/databasedevelopment/comments/1ha5cc...
|
| https://www.reddit.com/r/databasedevelopment/comments/1dqgms...
|
| https://www.reddit.com/r/databasedevelopment/comments/1ix5dz...
|
| https://www.reddit.com/r/databasedevelopment/comments/18knod...
|
| https://www.reddit.com/r/databasedevelopment/comments/1h3w70...
|
| https://www.reddit.com/r/databasedevelopment/comments/1gk18n...
|
| https://www.reddit.com/r/databasedevelopment/comments/1bemf9...
|
| https://www.reddit.com/r/databasedevelopment/comments/1iw6cx...
| danhau wrote:
| Nice, thanks! I didn't know there was a subreddit for that.
| seafoamteal wrote:
| Hi Phil, I thought I'd find you here! Love your blog!
| eatonphil wrote:
| Glad to hear it. :)
| dangoodmanUT wrote:
| Here's my list (probably not all of them)
|
| https://github.com/FireScroll/FireScroll/
|
| https://github.com/danthegoodman1/DurableStreams (brand new)
|
| https://github.com/danthegoodman1/ObjectKV
|
| https://github.com/danthegoodman1/WriteAhead
|
| https://github.com/danthegoodman1/icedb (most popular)
|
| https://github.com/danthegoodman1/Percolators (kind of a DB on
| top of a DB, technically just transactions tho)
|
| there are others that might not fit exactly what you're looking
| for
| faizhalde wrote:
| excellent, would love to hear more about resources you used while
| implementing
| Sahil121 wrote:
| followed this book - https://build-your-own.org/database/
| twalla wrote:
| Someone asked about resources and I ran into this while
| evaluating embedded db options for a golang project - it's a
| collection of db components implemented in golang:
|
| https://github.com/thomasjungblut/go-sstables
| tjungblut wrote:
| thank you for referring, feel free to ask any questions you may
| have
| voodooEntity wrote:
| I didnt look into it yet but i already wanted to say - cuz of
| curiosity i build my own graph database in golang :) and i
| learned alot. so i absolutly understand why you did it and what
| experiences you probably made on the way :D
|
| congratz !
| Sahil121 wrote:
| thanks man
| varelaseb wrote:
| Is this published anywhere?
| voodooEntity wrote:
| Kinda. So first i went for a standalone application back than
| to use it as storage/database. This part is archived now. it
| was released under:
| https://github.com/voodooEntity/slingshotdb
|
| At this point i think its important to mention that its
| nativly an in-memory graph database/storage.
|
| At some point (4-5 years ago) i decided to change the way i
| handle it. Since im using the storage in many private
| projects, and there was no real traction (while also
| maintaining a database is quite some work) i decided i gonne
| change the way i handle it away from a standalone database to
| a library that works as an in-memory graph storage.
|
| Also i added a custom query language including builder (the
| queries are 100% json compatible so you can in theory build
| queries from whatever language and transmit it via json.
|
| The library is released under :
| https://github.com/voodooEntity/gits
|
| Currently im completly reworking the query parser but since
| this is a privat project and im the only maintainer it may
| take some to to finish it. The update will takle some smaller
| bugs in the query parser that can occur on very complex
| nested queries and also will optimize query speeds. I cant
| provide a date when the update will occure i only can say it
| will (because i need it myself).
|
| Finally said: The strength of gits is to be very fast (hence
| in memory handling) and easy to use from golang. Its
| threadsafe by default.
| bob1029 wrote:
| If you are wanting proper SQL command support, you could copy the
| SQLite parser approach. Properly parsing all valid command texts
| is not a problem that I would find compelling unless I was being
| compensated for it.
|
| https://www.sqlite.org/lemon.html
|
| You could probably use something like participle, but you'd have
| to translate the grammar.
|
| https://github.com/alecthomas/participle
| Sahil121 wrote:
| yes man have to work on adding the sql cmd support thanks for
| the links
| qaq wrote:
| You can use https://github.com/cockroachdb/cockroachdb-parser
| which is basically PGSQL compatible parser implemented in
| golang
| mingodad wrote:
| I do have a big collection of LALR(1) grammars to
| test/study/develop/document here
| https://mingodad.github.io/parsertl-playground/playground/
| including sqlite, tidb, vites, postgresql, mysql, ...
| alecco wrote:
| Cool!
|
| You watched CMU's intro to Database sytems, right? It's really
| good and thorough. It will save you of some common pitfalls and
| can help you navigate the trade-offs.
|
| https://www.youtube.com/watch?v=otE2WvX3XdQ&list=PLSE8ODhjZX...
| Sahil121 wrote:
| sure man
| hobs wrote:
| Looks fun, but the intro requirement is "Knows C++" - that's
| kinda a non starter right?
| alecco wrote:
| I'm sure that's for the homework.
| BiraIgnacio wrote:
| Oh thanks for the link, this is really great!
| Tostino wrote:
| I really have liked the current Optimizer series. So much
| amazing info.
|
| I hope some company that heavily relies on Postgres sponsors a
| project to replace the optimizer with a more modern
| implementation. I feel like for any large applications with
| olap style queries (even if on the OLTP database) will benefit
| incredibly from that work. SQL Servers optimizer was light-
| years ahead of Postgres when I worked with it last over a
| decade ago.
| alecco wrote:
| Sadly, most of those corporations would rather spend millions
| in proprietary solutions than give back 100k to the
| community. I've seen it first hand many times. They only give
| money if they absolutely have to.
| skrtskrt wrote:
| I also recommend "Database Internals" to people. An invaluable
| book, the logical next read after "Designing Data-Intensive
| Applications". It's the best and most approachable resource on
| distributed systems and database consistency / performance
| tradeoffs I have found.
| fipar wrote:
| Congratulations on a great exploratory project!
|
| It takes me back to my school years. I never got as far as you
| (not by any long stretch actually), but I did enjoy creating the
| storage layer of a database from scratch. To actually have to
| deal with, instead of just think of, all the edge cases, is quite
| the transformative experience.
|
| As a humble suggestion, since it seems your goal is to understand
| how relational databases work and not necessarily to write a new
| database that will compete with others, maybe don't make it an
| SQL one? We've got enough of those, and not enough of the others.
| Would be nice to have a new relational DB using Tutorial D as its
| language for example.
|
| Keep hacking!
| Sahil121 wrote:
| thanks for the suggestion man, will definitely think about this
| snissn wrote:
| to also share a reference - here's an on disk hashmap that uses
| mmap that I made in golang: https://github.com/snissn/gomap
| tapirl wrote:
| Cool!
|
| btw, README says Go 1.17+ is required, but go.mod says 1.23+.
| hsnice16 wrote:
| This is so nice!
|
| I created an email verifier project in Go to check MX records,
| SMTP server status, and existence in the disposable list.
|
| Project - https://github.com/hsnice16/email-verifier
| mdwhatcott wrote:
| Best part: a go.mod with 0 dependencies!
| OccamsMirror wrote:
| Now _that_ is impressive. Gotta love zero dependency projects.
| dveeden2 wrote:
| Other databases written in Go:
|
| - TiDB by PingCAP
|
| - Vitess by PlanetScale
|
| Both are basically only the SQL part as TiDB uses TiKV (written
| in Rust) and Vitess uses MySQL.
|
| For those who want to implement a database in Go but without
| having to implement a network protocol there is go-mysql, which
| allows you to do this: https://github.com/go-mysql-org/go-
| mysql/blob/master/cmd/go-... As demonstration I created a
| networked SQLite: https://github.com/dveeden/go-mysql-
| examples/blob/main/go-my...
|
| Both TiDB and Vitess have parsers that can be used outside
| standalone. So if you only want to implement your own on disk
| format, this can help.
|
| Note that I'm working for PingCAP on TiDB and I'm also a co-
| maintainer for go-mysql.
| raggi wrote:
| Vitess came from YouTube
| CBLT wrote:
| Didn't the Vitess team found planetscale?
| harshitgangal wrote:
| Yes! The founders of PlanetScale were the co-creators of
| Vitess at YouTube, where it was built to handle MySQL
| scalability. PlanetScale builds on Vitess but offers a
| managed, developer-friendly experience.
| aqueueaqueue wrote:
| We use Tidb at work at scale. Great product! Was looking at the
| source today to understand an error code.
| 0x3331 wrote:
| Looks like a start. Keep it up, keep learning. Definitely check
| out https://www.youtube.com/c/CMUDatabaseGroup
|
| Great lectures on relational databases and more.
___________________________________________________________________
(page generated 2025-02-27 23:01 UTC)