[HN Gopher] Litestream: Revamped
___________________________________________________________________
Litestream: Revamped
Author : usrme
Score : 164 points
Date : 2025-05-20 19:58 UTC (3 hours ago)
(HTM) web link (fly.io)
(TXT) w3m dump (fly.io)
| maxmcd wrote:
| Looks like the code is here:
| https://github.com/benbjohnson/litestream/tree/v0.5
|
| Really nice to see this, I wrote this comment almost 2 years ago
| when I was a little miffed about trying to use litestream and
| litefs: https://news.ycombinator.com/item?id=37614193
|
| I think this solves most of the issues? You can now freely run
| litestream on your DB and not worry about issues with multiple
| writers? I wonder how the handoff is handled.
|
| The read replica FUSE layer sounds like a real nice thing to
| have.
|
| edit: Ah, it works like this:
| https://github.com/benbjohnson/litestream/pull/617
|
| > When another Litestream process starts up and sees an existing
| lease, it will continually retry the lease acquisition every
| second until it succeeds. This low retry interval allows for
| rolling restarts to come online quickly.
|
| Sounds workable!
| gavinray wrote:
| Just a heads-up, the link in the "Litestream is fully open
| source" callout is malformed and leads to:
|
| https://http//litestream.io/
| tptacek wrote:
| Thanks!
| srameshc wrote:
| I have been following Ben for a long time but I never knew LiteFS
| was based on his work. I somehow settled eventually for rqlite
| for self managed distributed.
|
| https://github.com/rqlite/rqlite
| digianarchist wrote:
| Similar approach to LiteFS but rqlite embeds raft into the
| project instead of depending on consul.
|
| https://youtu.be/8XbxQ1Epi5w?si=puJFLKoVs3OeYrhS
| jasonthorsness wrote:
| What a coincidence, I was just researching Litestream today! I
| use Sqlite on my VPS and was thinking about adding this.
|
| Am I understanding correctly that I will be able to restore a
| database to any point-in-time that is while the litestream
| process is running? Because auto-checkpointing could consume the
| WAL while it isn't running?
|
| So for an extreme example if the process crashed for an hour
| between 2:00 and 3:00, I could restore to 1:55 or 3:05 but the
| information required to restore between 2:00 and 3:00 is lost?
| benbjohnson wrote:
| Litestream saves WAL segments to a given time granularity. By
| default, it ships off WAL changes every second so you should be
| able to restore to any given second in your history (within
| your retention period).
| fra wrote:
| Litestream has seen very little development lately and I was
| worried it was dead. Very glad to see Ben Johnson is continuing
| to push the project forward with some exciting new plans.
| simonw wrote:
| This post is like they read my mind and implemented _everything_
| I wanted from a new Litestream. So exciting.
| psanford wrote:
| This looks great! A few years ago I wrote a sqlite vfs for using
| dynamodb as a backing store[0] called DonutDB. With the recent
| addition of CAS to S3, I was thinking about making a new version
| of DonutDB backed by S3. I'm really glad lightstream supports
| this so I don't have to!
|
| I can't wait to try this out.
|
| [0]: https://github.com/psanford/donutdb
| rawkode wrote:
| Amazing to see and hear about the progress. Always a pleasure
| when Ben works on something and shares it. Keep it up!
| nico wrote:
| Very cool idea, I wonder if that works better than their Postgres
| instances
|
| Recently, I deployed a little side project using a small postgres
| vm on fly.io After a couple of days, and only having about 500kb
| of data stored in that db, the postgres vm went into an
| unrecoverable fail loop, saying it ran out of memory, restarting,
| then immediately running out of memory again, so on and so forth
|
| It took about 3-4hrs to recover the data jumping through a lot of
| hoops to be able to access the data, copy it to another volume
| and finally download it
|
| I would've reached for support, but it seems like the only option
| available is just posting on their forum. I saw a couple of
| related posts, all with unsatisfactory answers unfortunately
|
| To be fair, it was incredibly easy to get up and running with
| them. On the other hand, almost all the time I saved by that
| quick start, was wasted recovering the failing db, all the while
| my site was down
|
| Ironically, I originally developed the project using sqlite, but
| then switched to postgres to deploy
| yellow_lead wrote:
| It's strange to me that they still haven't offered a managed
| Postgres product. Other providers like Render or even Heroku
| seem to have realized that this is a core part of PaaS that
| customers want. Instead they focused on GPUs and LiteStream.
| When I evaluated different PaaS for the startup I work at, I
| had to go with Render. I couldn't even give Fly.io a try since
| I knew we needed Postgres.
| biorach wrote:
| https://fly.io/docs/mpg/overview/
| nico wrote:
| It think they are in beta. I wished they had a managed Redis
| though
|
| For Postgres I ended up going with Neon (neon.tech), very
| happy with them so far. Super easy to setup and get up and
| running, also love being able to just easily see the data
| from their web interface
| tptacek wrote:
| We're rolling out Managed Postgres, very slowly.
| sergiotapia wrote:
| Try Railway - nothing but good experiences with these dudes.
| Fairly priced and great dev UX.
| tptacek wrote:
| This post has nothing to do with Fly.io's platform offerings.
| Litestream is completely uncoupled from Fly.io. Ben started it
| before he got here.
| ignoramous wrote:
| We have a sneaking suspicion that the robots that write LLM code
| are going to like SQLite too. We think what coding agents like
| Phoenix.new want is a way to try out code on live data, screw it
| up, and then rollback both the code and the state.
|
| Prescient.
|
| Agents would of course work well if they can go back in time to
| checkpoints and branch from there, exploring solutions parallely
| as needed.
|
| Anyone who has experience with building workflows (Amazon SWF,
| Temporal, and the like) knows how difficult it is to maintain
| determinism in face of retries & re-drives in multi-tier setups
| (especially, those involving databases).
|
| Replit recently announced their Agent's integration with Neon's
| _time travel_ feature [0] for exactly the purpose outlined in
| TFA. Unlike Fly.io though, Replit is built on GCP and other 3p
| providers like Neon and it is unclear if both GCP & Databricks
| won't go all Oracle on them.
|
| [0] https://blog.replit.com/safe-vibe-coding
| mixmastamyk wrote:
| For something rather new there seems to be too many choices
| already. Please pick a strategy under one name, good defaults,
| and a couple of config options.
| j0e1 wrote:
| This is exciting! Especially glad that Litestream is still
| maintained. Is there a use-case for Litestream for more than
| backup? I am a fan of offline-first but it would be cool to have
| a way to synchronize on-device SQLite instances to a single
| central instance.
| benbjohnson wrote:
| Backups & read replicas are the primary use cases. If you're
| interested in local-first, you can check out projects like cr-
| sqlite[1].
|
| [1]: https://github.com/vlcn-io/cr-sqlite
| j0e1 wrote:
| Thank you Ben for the great work on this project! Also, I
| remember looking into this one and: https://github.com/vlcn-
| io/cr-sqlite/issues/444
| dankebitte wrote:
| > read replicas
|
| Can this be done with _only_ Litestream, or is LiteVFS still
| in development? I looked into this last year but was put off
| by LiteFS 's stated write performance penalty due to FUSE
| [1]; it's still marked as WIP [2] and hasn't seen updates for
| over a year.
|
| [1] https://fly.io/docs/litefs/faq/#what-are-the-tradeoffs-
| of-us...
|
| [2] https://github.com/superfly/litevfs
| caleblloyd wrote:
| Is the backend pluggable? Could it be configured to write to any
| key value store with support for optimistic concurrency control?
| benbjohnson wrote:
| We don't support plug-ins at the moment but there's several
| backends at the moment (S3, Azure Blob Storage, Google Cloud
| Storage, SFTP, etc)
| nodesocket wrote:
| Is there a migration guide from stable to the branch 0.5? I'm
| running Litestream as a Docker sidecar alongside my Python app
| container and it's been great and a nice comfort knowing my
| SQLite db is backed up to S3.
| bradgessler wrote:
| I wish Fly would polish the developer experience on top of
| SQLite. They're close, but it's missing:
|
| 1. A built-in UI and CLI that manages SQLite from a volume.
| Getting the initial database on a Fly Machine requires more work
| than it should.
|
| 2. `fly console` doesn't work with SQLite because it spins up a
| separate machine, which isn't connected to the same volume where
| the SQLite data resides. Instead you have to know to run `fly ssh
| console --pty`, which effectively SSH's into the machine with the
| database.
|
| The problem in general with SQLite web apps is they tend to be
| small apps, so you need a _lot_ of them to make a decent amount
| of money hosting them.
| hobo_mark wrote:
| If you wanted to use litestream to replicate many databases
| (ideally, one or more per user), which is one of the use cases
| described here (and elsewhere), how do you tell litestream to add
| new databases dynamically? The configuration file is static and I
| haven't found an API to tell it to track a new db at runtime.
| oliwary wrote:
| Fantastic to see it's getting updated! I am a big fan of
| litestream, have been using it for a while together with
| pocketbase. It's like a cheat code for a cheap, reliable and safe
| backend.
| thewisenerd wrote:
| ben, thanks for litestream!
|
| we're using it on production for a write-heavy interal use-case
| (~12GB compressed) for more than a year now; and it's costing us
| a couple hundred pennies per month (azure).
|
| excited to try the new changes when they land.
| neom wrote:
| Is Litestream on a path to subsume LiteFS's capabilities? Re:
| PITR, would this be used to facilitate automated A/B testing of
| AI-generated code changes against live data subsets? I can
| imagine a lot of cool stuff in that direction. This is really
| cool Ben!
___________________________________________________________________
(page generated 2025-05-20 23:00 UTC)