[HN Gopher] A MySQL compatible database engine written in pure Go
___________________________________________________________________
A MySQL compatible database engine written in pure Go
Author : mliezun
Score : 125 points
Date : 2024-04-09 19:50 UTC (3 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| jddj wrote:
| I always found the idea behind dolt to be very enticing.
|
| Not enticing enough to build a business around, due to it being
| that bit too different and the persistence layer being that bit
| too important. But the sort of thing that I'd love it if the
| mainstream DBs would adopt.
|
| I didn't realise the engine was written in Go, and honestly the
| first place my mind wonders is to performance.
| zachmu wrote:
| We haven't benchmarked the in-memory database implementation
| bundled in go-mysql-server in a while, but I would be surprised
| if it's any slower than MySQL, considering that Dolt runs on
| the same engine and is ~2x slower than MySQL including disk-
| access.
|
| https://docs.dolthub.com/sql-reference/benchmarks/latency
| jchanimal wrote:
| If you like the idea of the Dolt prolly trees[1], I'm building
| a database[2] that uses them for indexing, (eventually)
| allowing for shared index updates across actors. Our core uses
| open-source JavaScript[3], but there are a few other
| implementations including RhizomeDB in Rust[4]. I'm excited
| about the research in this area.
|
| [1] https://docs.dolthub.com/architecture/storage-
| engine/prolly-...
|
| [2] https://fireproof.storage
|
| [3] https://github.com/mikeal/prolly-trees
|
| [4] https://jzhao.xyz/thoughts/Prolly-Trees
| timsehn wrote:
| Version Control is not the type of thing "mainstream DBs would
| adopt".
|
| We needed to build a custom storage engine to make querying and
| diffing work at scale:
|
| https://docs.dolthub.com/architecture/storage-engine
|
| It based on the work of Noms including the data structure they
| invented, Prolly Trees.
|
| https://docs.dolthub.com/architecture/storage-engine/prolly-...
| jbverschoor wrote:
| This seems to be a wire-protocol proxy for mysql -> SQL.
|
| The default proxied database is dolt. I'm guessing this is
| extracted from dolt itself as that claims to be wire-compatible
| with mysql. Which all makes total sense.
| zachmu wrote:
| Not a proxy in the traditional sense, no. go-mysql-server is a
| set of libraries that implement a SQL query engine and server
| in the abstract. When provided with a compatible database
| implementation using the provided interfaces, it becomes a
| MySQL compatible database server. Dolt [1] is the most complete
| implementation, but the in-memory database implementation the
| package ships with is suitable for testing.
|
| We didn't extract go-mysql-server from Dolt. We found it
| sitting around as abandonware, adopted it, and used it to build
| Dolt's SQL engine on top of the existing storage engine and
| command line [2]. We decided to keep it a separate package, and
| implementation agnostic, in the hopes of getting contributions
| from other people building their own database implementations
| on top of it.
|
| [1] https://github.com/dolthub/dolt [2]
| https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-se...
| karmakaze wrote:
| Compatible has many aspects. I'd be interested in the replication
| protocols.
| kamikaz1k wrote:
| shouldn't these projects have a perf comparison table? there was
| a post a couple days ago about the an in-memory Postgres, but
| same problem on the perf.
|
| if someone is considering running it, they're probably
| considering it against the actual thing. and I would think the
| main decision criteria is: _how much faster tho?_
| zachmu wrote:
| This is a reasonable point, we'll run some benchmarks and
| publish them.
|
| We expect that it's faster than MySQL for small scale. Dolt is
| only 2x slower than MySQL and that includes disk access.
|
| https://docs.dolthub.com/sql-reference/benchmarks/latency
| sgammon wrote:
| Isn't that........ Vitess?
| zachmu wrote:
| Vitess (a fork of it anyway) provides the parser and the
| server. The query engine is all custom go code.
| pizza234 wrote:
| The compatibility (and functionality in general) is severely
| limited, not usable in production:
|
| > No transaction support. Statements like START TRANSACTION,
| ROLLBACK, and COMMIT are no-ops.
|
| > Non-performant index implementation. Indexed lookups and joins
| perform full table scans on the underlying tables.
|
| I actually wonder if they support triggers, stored procedures
| etc.
| zgk7iqea wrote:
| Only for the in-memory implementation. It is also specifically
| stated that you shouldn't use the in-memory stub in production
| zachmu wrote:
| Yes, triggers and stored procedures are supported. Concurrency
| is the only real limitation in terms of functionality.
|
| The bundled in-memory database implementation is mostly for use
| in testing, for people who run against mysql in prod and want a
| fast compatible go library to test against.
|
| For a production-ready database that uses this engine, see
| Dolt:
|
| https://github.com/dolthub/dolt
| didip wrote:
| tidb has been around for a while, it is distributed, written in
| Go and Rust, and MySQL compatible.
| https://github.com/pingcap/tidb
|
| Somewhat relatedly, StarRocks is also MySQL compatible, written
| in Java and C++, but it's tackling OLAP use-cases.
| https://github.com/StarRocks/starrocks
|
| But maybe this project is tackling a different angle. Vitess
| MySQL library is kind of hard to use. Maybe this can be used to
| build ORM-like abstraction layer?
| verdverm wrote:
| Dolt supports git like semantics, so you can commit, pull,
| merge, etc...
| neximo64 wrote:
| Is there anything like this for postgres?
| perplexa wrote:
| cockroachdb might be close:
| https://github.com/cockroachdb/cockroach
| malkia wrote:
| Is this for integration/smoke testing?
| timsehn wrote:
| Most direct users of go-mysql-server use it to test Golang <>
| MySQL interactions without needing a running server.
|
| We here at DoltHub use it to provide SQL to Dolt.
| zachmu wrote:
| Hi, this is my project :)
|
| For us this package is most important as the query engine that
| powers Dolt:
|
| https://github.com/dolthub/dolt
|
| We aren't the original authors but have contributed the vast
| majority of its code at this point. Here's the origin story if
| you're interested:
|
| https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-se...
| geenat wrote:
| What's the replication story currently like?
| zachmu wrote:
| The vanilla package can replicate to or from MySQL via binlog
| replication. But since it's memory only, that's probably not
| what you want. You probably want to supply the library a
| backend with persistence, not the built-in memory-only one
|
| Dolt can do the same two directions of MySQL binlog
| replication, and also has its own native replication options:
|
| https://docs.dolthub.com/sql-reference/server/replication
| geenat wrote:
| Interesting!
|
| > If you have an existing MySQL or MariaDB server, you can
| configure Dolt as a read-replica. As the Dolt read-replica
| consumes data changes from the primary server, it creates
| Dolt commits, giving you a read-replica with a versioned
| history of your data changes.
|
| This is really cool.
| maxloh wrote:
| I know it is a matter of choice, but why was MySQL chosen instead
| of PostgreSQL? The latter seems to be more popular on Hacker
| News.
| tobinfekkes wrote:
| Typically, things that are more popular on Hacker News are not
| most popular with the rest of the world.
| timsehn wrote:
| > Why is Dolt MySQL flavored?
|
| TLDR; Because go-mysql-server existed.
|
| https://www.dolthub.com/blog/2022-03-28-have-postgres-want-d...
|
| We have a Postgres version of Dolt in the works called
| Doltgres.
|
| https://github.com/dolthub/doltgresql
|
| We might have a go-postgres-server package factored out
| eventually.
| amelius wrote:
| Performance comparison against the original?
| geenat wrote:
| With Vitess likely consolidating its runtimes (vtgate, vtctl,
| vttablet, etc) into a single unified binary:
| https://github.com/vitessio/vitess/issues/7471#issuecomment-...
|
| ... it would be a wild future if Vitess replaced the underlying
| MySQL engine with this (assuming the performance is good enough
| for Vitess).
___________________________________________________________________
(page generated 2024-04-09 23:00 UTC)