[HN Gopher] Litestream: Revamped
___________________________________________________________________
Litestream: Revamped
Author : usrme
Score : 428 points
Date : 2025-05-20 19:58 UTC (1 days ago)
(HTM) web link (fly.io)
(TXT) w3m dump (fly.io)
| maxmcd wrote:
| Looks like the code is here:
| https://github.com/benbjohnson/litestream/tree/v0.5
|
| Really nice to see this, I wrote this comment almost 2 years ago
| when I was a little miffed about trying to use litestream and
| litefs: https://news.ycombinator.com/item?id=37614193
|
| I think this solves most of the issues? You can now freely run
| litestream on your DB and not worry about issues with multiple
| writers? I wonder how the handoff is handled.
|
| The read replica FUSE layer sounds like a real nice thing to
| have.
|
| edit: Ah, it works like this:
| https://github.com/benbjohnson/litestream/pull/617
|
| > When another Litestream process starts up and sees an existing
| lease, it will continually retry the lease acquisition every
| second until it succeeds. This low retry interval allows for
| rolling restarts to come online quickly.
|
| Sounds workable!
| gavinray wrote:
| Just a heads-up, the link in the "Litestream is fully open
| source" callout is malformed and leads to:
|
| https://http//litestream.io/
| tptacek wrote:
| Thanks!
| srameshc wrote:
| I have been following Ben for a long time but I never knew LiteFS
| was based on his work. I somehow settled eventually for rqlite
| for self managed distributed.
|
| https://github.com/rqlite/rqlite
| digianarchist wrote:
| Similar approach to LiteFS but rqlite embeds raft into the
| project instead of depending on consul.
|
| https://youtu.be/8XbxQ1Epi5w?si=puJFLKoVs3OeYrhS
| tptacek wrote:
| I don't think they're similar at all. LiteFS uses Consul to
| elect a leader for a single-write-leader multiple-replica
| configuration, the same way you'd do with Postgres. rqlite
| (as I understood it last time I looked) runs Raft directly;
| it gets quorums for every write.
|
| One isn't better than the other. But LiteFS isn't a
| "distributed SQLite" in the sense you'd think of with rqlite.
| It's a system for getting read-only replicas, the same way
| you've been able to do with log shipping on n-tier databases
| for decades.
| apitman wrote:
| rqlite also requires you to use special client libraries,
| whereas litefs is transparent to the program.
| jasonthorsness wrote:
| What a coincidence, I was just researching Litestream today! I
| use Sqlite on my VPS and was thinking about adding this.
|
| Am I understanding correctly that I will be able to restore a
| database to any point-in-time that is while the litestream
| process is running? Because auto-checkpointing could consume the
| WAL while it isn't running?
|
| So for an extreme example if the process crashed for an hour
| between 2:00 and 3:00, I could restore to 1:55 or 3:05 but the
| information required to restore between 2:00 and 3:00 is lost?
| benbjohnson wrote:
| Litestream saves WAL segments to a given time granularity. By
| default, it ships off WAL changes every second so you should be
| able to restore to any given second in your history (within
| your retention period).
| dolmen wrote:
| Do you have DST handling issues?
|
| I'm asking because switching from winter time to summer time in
| Europe happened on March 30th with local time jumping from 2:00
| to 3:00.
| fra wrote:
| Litestream has seen very little development lately and I was
| worried it was dead. Very glad to see Ben Johnson is continuing
| to push the project forward with some exciting new plans.
| avtar wrote:
| That's the conclusion I reached a couple months ago when I was
| evaluating similar tools. The last Litestream release was
| issued in 2023 and the official Docker image is over a year
| old. In the end it seemed like a safer bet to accept some
| inconvenient tradeoffs and just create backups more frequently.
| tptacek wrote:
| Ben also wrote BoltDB, which was untouched (archived, even)
| for years despite a thriving community. Sometimes things are
| just done!
| Zekio wrote:
| seems to have active commits from 2 weeks ago, just not on the
| main branch
| noroot wrote:
| I love the idea of litestream and litefs and do use it for some
| smaller projects, but have also been worried it was abandoned.
| The line is quite thin between "done" and "not maintained".
|
| There clearly still is some untapped potential in this space,
| so I am glad benbjohnson is exploring and developing these
| solutions.
|
| Great that the new release will offer the ability to replicate
| multiple database files.
|
| > Modern object stores like S3 and Tigris solve this problem
| for us: they now offer conditional write support
|
| I hope this won't be a hard requirement, since some S3
| compatible storage do not have this feature (yet). I also do
| use the SFTP storage option currently.
| simonw wrote:
| This post is like they read my mind and implemented _everything_
| I wanted from a new Litestream. So exciting.
| psanford wrote:
| This looks great! A few years ago I wrote a sqlite vfs for using
| dynamodb as a backing store[0] called DonutDB. With the recent
| addition of CAS to S3, I was thinking about making a new version
| of DonutDB backed by S3. I'm really glad lightstream supports
| this so I don't have to!
|
| I can't wait to try this out.
|
| [0]: https://github.com/psanford/donutdb
| hamandcheese wrote:
| > With the recent addition of CAS to S3
|
| Do you have a reference for this? I assume by CAS you mean
| content addressable storage? I googled but can't find any AWS
| docs on this.
| xyzzy_plugh wrote:
| Compare And Swap
| gcr wrote:
| The TL;DR is that Amazon S3 now supports "conditional
| writes" which are guaranteed to fail if the file was
| written by some other writer. This is implemented by
| sending the ETag of an object's expected version alongside
| the write request.
|
| Litestream now depends on this functionality to handle
| multiple writers. Think of optimistic locking.
|
| https://aws.amazon.com/about-aws/whats-
| new/2024/11/amazon-s3...
| hamandcheese wrote:
| Thanks both! In the contexts I work in CAS always means
| content-addressable-storage, my mistake.
| rawkode wrote:
| Amazing to see and hear about the progress. Always a pleasure
| when Ben works on something and shares it. Keep it up!
| nico wrote:
| Very cool idea, I wonder if that works better than their Postgres
| instances
|
| Recently, I deployed a little side project using a small postgres
| vm on fly.io After a couple of days, and only having about 500kb
| of data stored in that db, the postgres vm went into an
| unrecoverable fail loop, saying it ran out of memory, restarting,
| then immediately running out of memory again, so on and so forth
|
| It took about 3-4hrs to recover the data jumping through a lot of
| hoops to be able to access the data, copy it to another volume
| and finally download it
|
| I would've reached for support, but it seems like the only option
| available is just posting on their forum. I saw a couple of
| related posts, all with unsatisfactory answers unfortunately
|
| To be fair, it was incredibly easy to get up and running with
| them. On the other hand, almost all the time I saved by that
| quick start, was wasted recovering the failing db, all the while
| my site was down
|
| Ironically, I originally developed the project using sqlite, but
| then switched to postgres to deploy
| yellow_lead wrote:
| It's strange to me that they still haven't offered a managed
| Postgres product. Other providers like Render or even Heroku
| seem to have realized that this is a core part of PaaS that
| customers want. Instead they focused on GPUs and LiteStream.
| When I evaluated different PaaS for the startup I work at, I
| had to go with Render. I couldn't even give Fly.io a try since
| I knew we needed Postgres.
| biorach wrote:
| https://fly.io/docs/mpg/overview/
| nico wrote:
| It think they are in beta. I wished they had a managed Redis
| though
|
| For Postgres I ended up going with Neon (neon.tech), very
| happy with them so far. Super easy to setup and get up and
| running, also love being able to just easily see the data
| from their web interface
| tptacek wrote:
| We're rolling out Managed Postgres, very slowly.
| yellow_lead wrote:
| Looking forward to it!
| sergiotapia wrote:
| Try Railway - nothing but good experiences with these dudes.
| Fairly priced and great dev UX.
| internet_points wrote:
| is that an alternative to supabase?
| sergiotapia wrote:
| it's an alternative between
| render/railway/northflank/fly. part of the new-gen paas.
| tptacek wrote:
| This post has nothing to do with Fly.io's platform offerings.
| Litestream is completely uncoupled from Fly.io. Ben started it
| before he got here.
| sosodev wrote:
| Clearly it does have something to do with fly.io considering
| fly is and has been pushing for litefs/stream as the ideal
| database solution for fly users. It seems reasonable that
| readers would compare it to other fly offerings.
| tptacek wrote:
| We have... never done that? Like ever? LiteFS is
| interesting for some read-heavy use cases, especially for
| people who are doing especially edge-deployed things, but
| most people who use databases here use Postgres. We
| actually had a managed LiteFS product --- LiteFS Cloud ---
| and we sunset it, like over a year ago. We have a large
| team working on Managed Postgres. We do not have a big
| SQLite team.
|
| People sometimes have a hard time with the idea that we
| write about things _because they are interesting to us_ ,
| and for no other reason. That's also 60-70% of why Ben does
| what he does on Litestream.
| sosodev wrote:
| I'm sorry. I think that I, and probably others, have
| misinterpreted it. Between Ben's writings on the fly blog
| and litefs cloud it seemed like that was the case. I
| didn't realize it had been discontinued.
| tptacek wrote:
| Neither LiteFS nor Litestream (obviously) have been
| discontinued. They're both open source projects, and were
| both carefully designed not to depend on Fly.io to work.
| mixmastamyk wrote:
| What happened to the supabase integration? Seems to have
| fizzled as well.
| apitman wrote:
| You might try paying the $29/mo. I've found the email support
| to be great.
| norman784 wrote:
| That's one of the reasons I don't use their Postgres instances
| and instead go with a service with a dedicated database
| service, but for deploying backend apps it's pretty good.
| ignoramous wrote:
| We have a sneaking suspicion that the robots that write LLM code
| are going to like SQLite too. We think what coding agents like
| Phoenix.new want is a way to try out code on live data, screw it
| up, and then rollback both the code and the state.
|
| Prescient.
|
| Agents would of course work well if they can go back in time to
| checkpoints and branch from there, exploring solutions parallely
| as needed.
|
| Anyone who has experience with building workflows (Amazon SWF,
| Temporal, and the like) knows how difficult it is to maintain
| determinism in face of retries & re-drives in multi-tier setups
| (especially, those involving databases).
|
| Replit recently announced their Agent's integration with Neon's
| _time travel_ feature [0] for exactly the purpose outlined in
| TFA. Unlike Fly.io though, Replit is built on GCP and other 3p
| providers like Neon and it is unclear if both GCP & Databricks
| won't go all Oracle on them.
|
| [0] https://blog.replit.com/safe-vibe-coding
| mixmastamyk wrote:
| For something rather new there seems to be too many choices
| already. Please pick a strategy under one name, good defaults,
| and a couple of config options.
| j0e1 wrote:
| This is exciting! Especially glad that Litestream is still
| maintained. Is there a use-case for Litestream for more than
| backup? I am a fan of offline-first but it would be cool to have
| a way to synchronize on-device SQLite instances to a single
| central instance.
| benbjohnson wrote:
| Backups & read replicas are the primary use cases. If you're
| interested in local-first, you can check out projects like cr-
| sqlite[1].
|
| [1]: https://github.com/vlcn-io/cr-sqlite
| j0e1 wrote:
| Thank you Ben for the great work on this project! Also, I
| remember looking into this one and: https://github.com/vlcn-
| io/cr-sqlite/issues/444
| dankebitte wrote:
| > read replicas
|
| Can this be done with _only_ Litestream, or is LiteVFS still
| in development? I looked into this last year but was put off
| by LiteFS 's stated write performance penalty due to FUSE
| [1]; it's still marked as WIP [2] and hasn't seen updates for
| over a year.
|
| [1] https://fly.io/docs/litefs/faq/#what-are-the-tradeoffs-
| of-us...
|
| [2] https://github.com/superfly/litevfs
| caleblloyd wrote:
| Is the backend pluggable? Could it be configured to write to any
| key value store with support for optimistic concurrency control?
| benbjohnson wrote:
| We don't support plug-ins at the moment but there's several
| backends at the moment (S3, Azure Blob Storage, Google Cloud
| Storage, SFTP, etc)
| nodesocket wrote:
| Is there a migration guide from stable to the branch 0.5? I'm
| running Litestream as a Docker sidecar alongside my Python app
| container and it's been great and a nice comfort knowing my
| SQLite db is backed up to S3.
| bradgessler wrote:
| I wish Fly would polish the developer experience on top of
| SQLite. They're close, but it's missing:
|
| 1. A built-in UI and CLI that manages SQLite from a volume.
| Getting the initial database on a Fly Machine requires more work
| than it should.
|
| 2. `fly console` doesn't work with SQLite because it spins up a
| separate machine, which isn't connected to the same volume where
| the SQLite data resides. Instead you have to know to run `fly ssh
| console --pty`, which effectively SSH's into the machine with the
| database.
|
| The problem in general with SQLite web apps is they tend to be
| small apps, so you need a _lot_ of them to make a decent amount
| of money hosting them.
| adenta wrote:
| Brad, what's your take on Rails 8 w/ SQLite? Are you
| gravitating towards it these days over Postgres?
| bradgessler wrote:
| Yep! I just migrated a Fly PG cluster database to SQLite
| because I over-provisioned DB resources and got tired of
| dealing with the occasional node crashing.
|
| TBH I wish they had their managed PG cluster running because
| it would have made it easier to downsize, but I'm happy with
| SQLite.
|
| I used SQLite for another project that I knew was going to
| max out at 100 concurrent users and it worked great. The best
| moment was when a user reported a production error I couldn't
| recreate locally, so I downloaded the database and recreated
| it with the latest production data on my laptop. You couldn't
| do that with a high-compliance app, but that's not most apps.
|
| I'm hesitant to outright say "SQLite and Rails is
| great"because you have to know your app will run on one node.
| If you know that then it's fantastic.
| hobo_mark wrote:
| If you wanted to use litestream to replicate many databases
| (ideally, one or more per user), which is one of the use cases
| described here (and elsewhere), how do you tell litestream to add
| new databases dynamically? The configuration file is static and I
| haven't found an API to tell it to track a new db at runtime.
| mrkurt wrote:
| I would expect this problem to get solved. It's tricky to
| detect new sqlites, but not impossible.
|
| In the meantime, it's pretty straightforward to use as a
| library.
| oliwary wrote:
| Fantastic to see it's getting updated! I am a big fan of
| litestream, have been using it for a while together with
| pocketbase. It's like a cheat code for a cheap, reliable and safe
| backend.
| thewisenerd wrote:
| ben, thanks for litestream!
|
| we're using it on production for a write-heavy interal use-case
| (~12GB compressed) for more than a year now; and it's costing us
| a couple hundred pennies per month (azure).
|
| excited to try the new changes when they land.
| neom wrote:
| Is Litestream on a path to subsume LiteFS's capabilities? Re:
| PITR, would this be used to facilitate automated A/B testing of
| AI-generated code changes against live data subsets? I can
| imagine a lot of cool stuff in that direction. This is really
| cool Ben!
| yowmamasita wrote:
| tangent: in modern SQLite, are writes still serialized? That's my
| main concern when choosing a tech stack for an app that might
| have thousands of writes happening on peak periods
| simonw wrote:
| Yes they are, but if you benchmark thousands of writes a second
| you'll likely find that SQLite does just fine.
|
| You might start running into problems at tens or hundreds of
| thousands of writes a second, though even then you may be OK on
| the right hardware.
| soamv wrote:
| Yes, but: https://sqlite.org/wal.html
| markusw wrote:
| I wrote about this back in 2022, still applicable today:
| https://www.golang.dk/articles/benchmarking-sqlite-performan...
| malkia wrote:
| So fossil (which is built on top of sqlite) + this = SCM?
| hiAndrewQuinn wrote:
| I'm still waiting for someone to make the GitHub for fossil.
| Bonus points if it's called Paleontology
| diggan wrote:
| Is that needed to make Fossil useful? Since it's basically
| GitHub but all in Git, it's all works P2P without the need of
| a centralized service.
|
| I guess for discovery it kind of makes sense, but wouldn't
| really be "GitHub for Fossil" but more like a search
| engine/portal.
| m3sta wrote:
| Is there anything like Livestream that can be just pip installed?
| rads wrote:
| What will be required from users of the existing Litestream
| version to upgrade to the new one? Is it a matter of bumping the
| version when it comes out or is there more to it?
| dastbe wrote:
| asking since ben does take a look here...
|
| will revamped litestream have a solution for ACKing only when
| transactions have durably committed to storage?
| mythz wrote:
| Awesome stuff, this resolves my #1 feature request of being able
| to replicate an entire directory of SQLite *.db's from a single
| Litestream process - happy it's finally here.
|
| Should make replicating Multi tenant per-user SQLite databases a
| lot more appealing.
| oulipo wrote:
| I still don't really understand the real "advantages" of such
| an architecture, over, say, a centralized Postgres server,
| which can process just as much data no?
| tptacek wrote:
| The network round trips of queries to a database server add
| up, so much that they influence query design (though we don't
| much think about that anymore, because n-tier database
| designs are so prevalent that everyone writes queries that
| way). The advantage to SQLite is that queries are incredibly
| fast.
| JSR_FDED wrote:
| Will the new litestream work with object stores that don't
| provided conditional writes?
| wiradikusuma wrote:
| For Fly.io employees here: Can I finally replace my Postgre with
| this a'la Cloudflare D1 (which is also Sqlite based)?
| bambax wrote:
| Very cool!
|
| There may be a typo here:
|
| > _The most straightforward way around this problem is to make
| sure only one instance of Litestream can_ replication _to a given
| destination._
|
| Can replicate? Or can do replications?
| ChocolateGod wrote:
| > It will be able to fetch and cache pages directly from
| S3-compatible object storage.
|
| Does this mean your SQLite database size is no longer restricted
| by your local disk capacity?
| bdcravens wrote:
| Looking at the LiteVFS repo, it appears so, with some
| limitations.
|
| "LiteVFS is a Virtual Filesystem extension for SQLite that uses
| LiteFS Cloud as a backing store."
|
| Limitations
|
| - Databases with journal_mode=wal cannot be modified via
| LiteVFS (but can be read)
|
| - Databases with auto-vacuum cannon be opened via LiteVFS at
| all
|
| - VACUUM is not supported
|
| https://github.com/superfly/litevfs
| ChocolateGod wrote:
| A SQLite database that supports read-replicas and can offload
| cold data to object storage would be super useful.
| ncruces wrote:
| > Databases with journal_mode=wal cannot be modified via
| LiteVFS (but can be read)
|
| Without modifying SQLite (what the Turso guys did), the WAL
| index is hard (but not impossible) to share across a network
| boundary. I'm guessing that's the _why_ here. There 's a hack
| that I'm pretty confident works, but I'm not sure how it
| behaves under latency (sure enough that I use it for Windows,
| and it hasn't caused issues running mptest thousands of times
| in CI over months).
|
| Leaving it here, maybe Ben's interested.
|
| https://github.com/ncruces/go-
| sqlite3/blob/c780ef16e277274e7...
| Skinney wrote:
| If I'm deploying a new version of my app, the typical managed
| solution will spawn a new server instance with that new version,
| and once a health check has succeeded a couple of times it will
| reroute trafic to this new instance and kill the old one.
|
| Previously this would be problematic, as the new instance might
| miss changes made by the old server. Is this fixed by these new
| changes?
| maxmcd wrote:
| I don't think this is trivially fixed since you can still only
| have one writer that is holding the lease.
|
| Your new service will come up, but it won't be able to get the
| write lease until the previous server shuts down. Now you have
| tools to detect this, stop one writer, and start the other, but
| the service will likely have to experience some kind of
| requests queueing or downtime.
| gwking wrote:
| It seems to me that you need to think about your server as the
| production database, rather than a web server instance that can
| be trivially spawned by a management layer.
|
| When I deploy a new version of my python/sqlite web app, I do
| not replace the whole machine. I just upgrade the python
| package and restart the systemd service.
|
| If I wanted to reduce downtime I could probably figure out a
| transition using SO_REUSEPORT. During the transition the old
| and new processes would be using the db concurrently, so the
| app would have to respect that case. If part of the upgrade
| requires a db schema change then I'm not sure how you could
| avoid some downtime. But I don't know if it is possible with
| traditional dbs either.
| wg0 wrote:
| > Now that we've switched to LTX, this isn't a problem any more.
| It should thus be possible to replicate /data/*.db, even if
| there's hundreds or thousands of databases in that directory.
|
| That was the show stopper. Now multi tenant with per tenant
| database whee (in theory) each user can roll back to certain
| point in time or at least completely download their database and
| take away for whatever they want to do with it is going to be
| possible.
| Nelkins wrote:
| Does anybody have a list of which S3-compatible object storage
| providers support Compare-And-Swap?
| wim wrote:
| Very cool! This is so clever and makes deploying it so simple. I
| just wasn't able to use it yet because we'd have (many) thousands
| of SQLite DBs to backup. I quickly hacked something together
| using fanotify + SQLite's Backup API to have some copies at
| least, but I'm going to try to switch to Litestream if this
| amount of files would be supported by the wildcard replication.
___________________________________________________________________
(page generated 2025-05-21 23:01 UTC)