https://github.com/benbjohnson/litestream#open-source-not-open-contribution Skip to content Sign up * Why GitHub? Features - + Mobile - + Actions - + Codespaces - + Packages - + Security - + Code review - + Project management - + Integrations - + GitHub Sponsors - + Customer stories - + Security - * Team * Enterprise * Explore + Explore GitHub - Learn & contribute + Topics - + Collections - + Trending - + Learning Lab - + Open source guides - Connect with others + The ReadME Project - + Events - + Community forum - + GitHub Education - + GitHub Stars program - * Marketplace * Pricing Plans - + Compare plans - + Contact Sales - + Nonprofit - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} benbjohnson / litestream * Watch 12 * Star 766 * Fork 10 Streaming S3 replication for SQLite. GPL-3.0 License 766 stars 10 forks Star Watch * Code * Issues 12 * Pull requests 0 * Discussions * Actions * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Security * Insights main 2 branches 9 tags Go to file Code Clone HTTPS GitHub CLI [https://github.com/b] Use Git or checkout with SVN using the web URL. [gh repo clone benbjo] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching Xcode If nothing happens, download Xcode and try again. Go back Launching Visual Studio If nothing happens, download the GitHub extension for Visual Studio and try again. Go back Latest commit @benbjohnson benbjohnson CONTRIBUTING ... 778451f Jan 28, 2021 CONTRIBUTING 778451f Git stats * 150 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github CONTRIBUTING Jan 28, 2021 cmd/litestream Allow replica URLs for generations command Jan 27, 2021 etc Add signed homebrew install Jan 24, 2021 internal Add validation interval Jan 15, 2021 s3 Allow replica URL to be used for commands Jan 26, 2021 .gitignore Add .deb packaging Jan 16, 2021 LICENSE GPL v3 Nov 6, 2020 Makefile Add signed homebrew install Jan 24, 2021 README.md CONTRIBUTING Jan 28, 2021 db.go Fix restore logging. Jan 27, 2021 db_test.go cleanup Jan 18, 2021 go.mod Switch from gzip to lz4 Jan 18, 2021 go.sum Switch from gzip to lz4 Jan 18, 2021 litestream.go Allow replica URL to be used for commands Jan 26, 2021 litestream_test.go cleanup Jan 18, 2021 litestream_unix.go Use database owner/group Jan 11, 2021 litestream_windows.go Use database owner/group Jan 11, 2021 replica.go Allow replica URL to be used for commands Jan 26, 2021 replica_test.go Fix unit tests Jan 14, 2021 View code README.md Litestream GitHub release (latest by date) Status GitHub test Litestream is a standalone streaming replication tool for SQLite. It runs as a background process and safely replicates changes incrementally to another file or S3. Litestream only communicates with SQLite through the SQLite API so it will not corrupt your database. If you need support or have ideas for improving Litestream, please visit the GitHub Discussions to chat. If you find this project interesting, please consider starring the project on GitHub. Installation Mac OS (Homebrew) To install from homebrew, run the following command: $ brew install benbjohnson/litestream/litestream Linux (Debian) You can download the .deb file from the Releases page page and then run the following: $ sudo dpkg -i litestream-v0.3.0-linux-amd64.deb Once installed, you'll need to enable & start the service: $ sudo systemctl enable litestream $ sudo systemctl start litestream Release binaries You can also download the release binary for your system from the releases page and run it as a standalone application. Building from source Download and install the Go toolchain and then run: $ go install ./cmd/litestream The litestream binary should be in your $GOPATH/bin folder. Quick Start Litestream provides a configuration file that can be used for production deployments but you can also specify a single database and replica on the command line when trying it out. First, you'll need to create an S3 bucket that we'll call "mybkt" in this example. You'll also need to set your AWS credentials: $ export AWS_ACCESS_KEY_ID=AKIAxxxxxxxxxxxxxxxx $ export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx Next, you can run the litestream replicate command with the path to the database you want to backup and the URL of your replica destination: $ litestream replicate /path/to/db s3://mybkt/db If you make changes to your local database, those changes will be replicated to S3 every 10 seconds. From another terminal window, you can restore your database from your S3 replica: $ litestream restore -o /path/to/restored/db s3://mybkt/db Voila! Your database should be restored to the last replicated state that was sent to S3. You can adjust your replication frequency and other options by using a configuration-based approach specified below. Configuration A configuration-based install gives you more replication options. By default, the config file lives at /etc/litestream.yml but you can pass in a different path to any litestream command using the -config PATH flag. You can also set the LITESTREAM_CONFIG environment variable to specify a new path. The configuration specifies one or more dbs and a list of one or more replica locations for each db. Below are some common configurations: Replicate to S3 This will replicate the database at /path/to/db to the "/db" path inside the S3 bucket named "mybkt". access-key-id: AKIAxxxxxxxxxxxxxxxx secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx dbs: - path: /path/to/db replicas: - url: s3://mybkt/db Replicate to another file path This will replicate the database at /path/to/db to a directory named /path/to/replica. dbs: - path: /path/to/db replicas: - path: /path/to/replica Retention period By default, replicas will retain a snapshot & subsequent WAL changes for 24 hours. When the snapshot age exceeds the retention threshold, a new snapshot is taken and uploaded and the previous snapshot and WAL files are removed. You can configure this setting per-replica. Times are parsed using Go's duration so time units of hours (h), minutes (m), and seconds (s) are allowed but days, weeks, months, and years are not. db: - path: /path/to/db replicas: - url: s3://mybkt/db retention: 1h # 1 hour retention Monitoring replication You can also enable a Prometheus metrics endpoint to monitor replication by specifying a bind address with the addr field: addr: ":9090" This will make metrics available at: http://localhost:9090/metrics Other configuration options These are some additional configuration options available on replicas: * type--Specify the type of replica ("file" or "s3"). Derived from "path". * name--Specify an optional name for the replica if you are using multiple replicas. * path--File path to the replica location. * url--URL to the replica location. * retention-check-interval--Time between retention enforcement checks. Defaults to 1h. * validation-interval--Interval between periodic checks to ensure restored backup matches current database. Disabled by default. These replica options are only available for S3 replicas: * bucket--S3 bucket name. Derived from "path". * region--S3 bucket region. Looked up on startup if unspecified. * sync-interval--Replication sync frequency. Usage Replication Once your configuration is saved, you'll need to begin replication. If you installed the .deb file then run: $ sudo systemctl restart litestream To run litestream on its own, run: # Replicate using the /etc/litestream.yml configuration. $ litestream replicate # Replicate using a different configuration path. $ litestream replicate -config /path/to/litestream.yml The litestream command will initialize and then wait indefinitely for changes. You should see your destination replica path is now populated with a generations directory. Inside there should be a 16-character hex generation directory and inside there should be snapshots & WAL files. As you make changes to your source database, changes will be copied over to your replica incrementally. Restoring a backup Litestream can restore a previous snapshot and replay all replicated WAL files. By default, it will restore up to the latest WAL file but you can also perform point-in-time restores. A database can only be restored to a path that does not exist so you don't need to worry about accidentally overwriting your current database. # Restore database to original path. $ litestream restore /path/to/db # Restore database to a new location. $ litestream restore -o /path/to/restored/db /path/to/db # Restore from a replica URL. $ litestream restore -o /path/to/restored/db s3://mybkt/db # Restore database to a specific point-in-time. $ litestream restore -timestamp 2020-01-01T00:00:00Z /path/to/db Point-in-time restores only have the resolution of the timestamp of the WAL file itself. By default, Litestream will start a new WAL file every minute so point-in-time restores are only accurate to the minute. How it works SQLite provides a WAL (write-ahead log) journaling mode which writes pages to a -wal file before eventually being copied over to the original database file. This copying process is known as checkpointing. The WAL file works as a circular buffer so when the WAL reaches a certain size then it restarts from the beginning. Litestream works by taking over the checkpointing process and controlling when it is restarted to ensure that it copies every new page. Checkpointing is only allowed when there are no read transactions so Litestream maintains a long-running read transaction against each database until it is ready to checkpoint. The SQLite WAL file is copied to a separate location called the shadow WAL which ensures that it will not be overwritten by SQLite. This shadow WAL acts as a temporary buffer so that replicas can replicate to their destination (e.g. another file path or to S3). The shadow WAL files are removed once they have been fully replicated. You can find the shadow directory as a hidden directory next to your database file. If you database file is named /var/lib/my.db then the shadow directory will be /var/lib/.my.db-litestream. Litestream groups a snapshot and all subsequent WAL changes into "generations". A generation is started on initial replication of a database and a new generation will be started if litestream detects that the WAL replication is no longer contiguous. This can occur if the litestream process is stopped and another process is allowed to checkpoint the WAL. Open-source, not open-contribution Similar to SQLite, Litestream is open source but closed to contributions. This keeps the code base free of proprietary or licensed code but it also helps me continue to maintain and build Litestream. As the author of BoltDB, I found that accepting and maintaining third party patches contributed to my burn out and I eventually archived the project. Writing databases & low-level replication tools involves nuance and simple one line changes can have profound and unexpected changes in correctness and performance. Small contributions typically required hours of my time to properly test and validate them. I am grateful for community involvement, bug reports, & feature requests. I do not wish to come off as anything but welcoming, however, I've made the decision to keep this project closed to contributions for my own mental health and long term viability of the project. About Streaming S3 replication for SQLite. Topics replication sqlite s3 Resources Readme License GPL-3.0 License Releases 9 v0.3.1 Latest Jan 27, 2021 + 8 releases Packages 0 No packages published Languages * Go 99.5% * Other 0.5% * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.