https://github.com/erikgrinaker/toydb Skip to content Sign up * Why GitHub? Features - + Mobile - + Actions - + Codespaces - + Packages - + Security - + Code review - + Issues - + Integrations - + GitHub Sponsors - + Customer stories- * Team * Enterprise * Explore + Explore GitHub - Learn and contribute + Topics - + Collections - + Trending - + Learning Lab - + Open source guides - Connect with others + The ReadME Project - + Events - + Community forum - + GitHub Education - + GitHub Stars program - * Marketplace * Pricing Plans - + Compare plans - + Contact Sales - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} erikgrinaker / toydb * Notifications * Star 1.8k * Fork 126 Distributed SQL database in Rust, written as a learning project Apache-2.0 License 1.8k stars 126 forks Star Notifications * Code * Issues 4 * Pull requests 1 * Actions * Security * Insights More * Code * Issues * Pull requests * Actions * Security * Insights master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 1 branch 0 tags Code Clone HTTPS GitHub CLI [https://github.com/e] Use Git or checkout with SVN using the web URL. [gh repo clone erikgr] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching Xcode If nothing happens, download Xcode and try again. Go back Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @erikgrinaker erikgrinaker drone: use rust 1.53. ... c0f9ab5 Jun 18, 2021 drone: use rust 1.53. c0f9ab5 Git stats * 400 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time clusters clusters: disable fsync May 15, 2020 config make storage backends configurable May 23, 2020 docs docs: fix typo Jul 12, 2020 src Update crates to the latest versions (#47) Jun 18, 2021 tests Update crates to the latest versions (#47) Jun 18, 2021 .dockerignore docs: add SQL usage examples. Jun 13, 2020 .drone.yml drone: use rust 1.53. Jun 18, 2021 .gitignore storage: add in-memory B+tree key-value store (fixes #3) May 22, 2020 Cargo.lock Update crates to the latest versions (#47) Jun 18, 2021 Cargo.toml Update crates to the latest versions (#47) Jun 18, 2021 Dockerfile Dockerfile: bump rust and alpine images. Jun 18, 2021 LICENSE Added license Sep 22, 2019 README.md README: prevent logo link Jul 12, 2020 rustfmt.toml Added Clippy and rustfmt May 5, 2019 View code toyDB Documentation Usage Architecture Tests Performance Credits README.md [toydb] toyDB Build Status Distributed SQL database in Rust, written as a learning project. Most components are built from scratch, including: * Raft-based distributed consensus engine for linearizable state machine replication. * ACID-compliant transaction engine with MVCC-based snapshot isolation. * Pluggable storage engine with B+tree and log-structured backends. * Iterator-based query engine with heuristic optimization and time-travel support. * SQL interface including projections, filters, joins, aggregates, and transactions. toyDB is not suitable for real-world use, but may be of interest to others learning about database internals. Documentation * Architecture guide: a guide to toyDB's architecture and implementation. * SQL examples: comprehensive examples of toyDB's SQL features. * SQL reference: detailed reference documentation for toyDB's SQL dialect. * References: books and other research material used while building toyDB. Usage With a Rust compiler installed, a local five-node cluster can be started on localhost ports 9601 to 9605: $ (cd clusters/local && ./run.sh) A command-line client can be built and used with the node on localhost port 9605: $ cargo run --release --bin toysql Connected to toyDB node "toydb-e". Enter !help for instructions. toydb> CREATE TABLE movies (id INTEGER PRIMARY KEY, title VARCHAR NOT NULL); toydb> INSERT INTO movies VALUES (1, 'Sicario'), (2, 'Stalker'), (3, 'Her'); toydb> SELECT * FROM movies; 1|Sicario 2|Stalker 3|Her toyDB supports most common SQL features, including joins, aggregates, and ACID transactions. Architecture toyDB architecture toyDB's architecture is fairly typical for distributed SQL databases: a transactional key/value store managed by a Raft cluster with a SQL query engine on top. See the architecture guide for more details. Tests toyDB has decent test coverage, with about a thousand tests of core functionality. These consist of in-code unit-tests for many low-level components, golden master integration tests of the SQL engine under tests/sql, and a basic set of end-to-end cluster tests under tests/. Jepsen tests, or similar system-wide correctness and reliability tests, are desirable but not yet implemented. Execute cargo test to run all tests, or check out the latest CI run. Performance Performance is not a primary goal of toyDB, but it has a bank simulation as a basic gauge of throughput and correctness. This creates a set of customers and accounts, and spawns several concurrent workers that make random transfers between them, retrying serialization failures and verifying invariants: $ cargo run --release --bin bank Created 100 customers (1000 accounts) in 0.123s Verified that total balance is 100000 with no negative balances Thread 0 transferred 18 from 92 (0911) to 100 (0994) in 0.007s (1 attempts) Thread 1 transferred 84 from 61 (0601) to 85 (0843) in 0.007s (1 attempts) Thread 3 transferred 15 from 40 (0393) to 62 (0614) in 0.007s (1 attempts) [...] Thread 6 transferred 48 from 78 (0777) to 52 (0513) in 0.004s (1 attempts) Thread 3 transferred 57 from 93 (0921) to 19 (0188) in 0.065s (2 attempts) Thread 4 transferred 70 from 35 (0347) to 49 (0484) in 0.068s (2 attempts) Ran 1000 transactions in 0.937s (1067.691/s) Verified that total balance is 100000 with no negative balances The informal target was 100 transactions per second, and these results exceed that by an order of magnitude. For an unoptimized implementation, this is certainly "good enough". However, this is with a single node and fsync disabled - the table below shows results for other configurations, revealing clear potential for improvement: sync: false sync: true 1 node 1067 txn/s 38 txn/s 5 nodes 417 txn/s 19 txn/s Note that each transaction consists of six statements, including joins, not just a single update: BEGIN; -- Find the sender account with the highest balance SELECT a.id, a.balance FROM account a JOIN customer c ON a.customer_id = c.id WHERE c.id = {sender} ORDER BY a.balance DESC LIMIT 1; -- Find the receiver account with the lowest balance SELECT a.id, a.balance FROM account a JOIN customer c ON a.customer_id = c.id WHERE c.id = {receiver} ORDER BY a.balance ASC LIMIT 1; -- Transfer a random amount within the sender's balance to the receiver UPDATE account SET balance = balance - {amount} WHERE id = {source}; UPDATE account SET balance = balance + {amount} WHERE id = {destination}; COMMIT; Credits toyDB logo is courtesy of @jonasmerlin. About Distributed SQL database in Rust, written as a learning project Topics rust sql database raft mvcc distributed Resources Readme License Apache-2.0 License Releases No releases published Packages 0 No packages published Contributors 5 * @erikgrinaker * @iamazy * @chenyukang * @messense * @jonasmerlin Languages * Rust 99.9% * Other 0.1% * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.