[HN Gopher] Litestream Eliminated My Database Server for $0.03/M...
___________________________________________________________________
Litestream Eliminated My Database Server for $0.03/Month
Author : mtlynch
Score : 41 points
Date : 2021-04-29 13:59 UTC (9 hours ago)
(HTM) web link (mtlynch.io)
(TXT) w3m dump (mtlynch.io)
| based2 wrote:
| https://dqlite.io/ High-availability SQLite
| benbjohnson wrote:
| dqlite and rqlite are both great projects but target different
| use cases. Those run Raft consensus which requires at least 3
| nodes for HA whereas Litestream is meant to run on a single
| node and trades off with lower durability guarantees.
| atat7024 wrote:
| Could you elaborate on the durability?
|
| Would a multi tenant SaaS with databases on the much smaller
| size compared to what you all talk about do well to use this,
| with proper backups?
| beagle3 wrote:
| Is it possible to use dqlite/rqlite for high availability in
| the same datacenter, while shipping logs using litestream to
| a different data center for diaster recovery? Or are they
| incompatible?
|
| p.s. Thanks for litestream!
| otoolep wrote:
| rqlite author here. https://github.com/rqlite/rqlite/
|
| Yes, there is no reason why that wouldn't work. rqlite
| supports on-disk mode so you could run litestream alongside
| an rqlite node and backup the underlying SQLite database to
| your favourite cloud provider using litestream.
|
| The only downside is that rqlite performance is very
| sensitive to the number of writes to disk, and that's why
| rqlite uses an in-memory SQLite database by default, and
| lets the Raft log persist to disk. In exchange you can be
| sure that your writes are persisted to disk when the API
| acks your request. Perhaps if rqlite used a RAM-based
| filesystem for everything, and you combined it with
| litestream you could get much higher-performance from
| rqlite, with backup (with only a tiny window for data loss)
| if you use litestream. It's not entirely trivial, however,
| due the difference in data consistency models. For example,
| which SQLite database should be updated if the Leader in
| the cluster changes? It gets complicated.
|
| Ben has written a great program here. I wish I had his
| ideas! It's got me thinking. :-)
| tdeck wrote:
| Interesting setup! Does this mean you've got to download your
| entire DB on every deploy?
| mtlynch wrote:
| Thanks for reading!
|
| Yep, you do have to download the DB on every deploy. It's
| negligible for my deploys because my database is a few MB, but
| if it got to be in the GB range, I'd have to find some way
| around that.
| tdeck wrote:
| Is it possible to lose records written between the start of
| the deploy/DB download and the switchover to the new image?
| benbjohnson wrote:
| Litestream will flush outstanding changes on a clean
| shutdown so you shouldn't lose any data when deploying.
| mtlynch wrote:
| Author here.
|
| I discovered Litestream here on HN after benbjohnson shared it
| here a few months back.[0] I wasn't interested at first because I
| don't ever use SQLite, but it stuck with me. I eventually
| realized it's a great way to achieve vendor flexibility in
| hosting apps, so I used Litestream to build a pastebin-style
| service for my app and wanted to share what I learned.
|
| Happy to answer any questions or hear any feedback about this
| post.
|
| [0] https://news.ycombinator.com/item?id=26103776
| amzans wrote:
| Just wanted to say thank you for writing this Michael. You've
| been putting lots of great content on your blog! Really
| enjoying the real world examples you include.
| mtlynch wrote:
| That's great to hear. Thanks for reading and for the kind
| words!
| jfengel wrote:
| Question from a newbie to cloud stuff: what is the performance
| like for a database server that's separated from your host like
| that?
|
| In terms of latency alone it sounds like it risks adding tens
| or even hundreds of milliseconds to even a fairly trivial
| request: out of my datacenter, across the wire, into theirs,
| grab the data, and then reverse the path. And perhaps a charge
| for data going out each way.
|
| Am I wrong about that? Am I overestimating the travel time, or
| perhaps they end up being located in the same datacenter so
| it's all high speed?
| webmobdev wrote:
| what is the performance like for a database server that's
| separated from your host like that?
|
| It's not separated. As this diagram shows -
| https://mtlynch.io/litestream/diagram.jpg - the SQLite
| database is on the same server as the web application.
| Additionally, OP's setup seems to use Litestream to backup
| the SQLite database to some cloud storage.
| benbjohnson wrote:
| It's typical to see at least millisecond latency for a
| database server that's located in the same data center (e.g.
| hosting your own Postgres) and tens or hundreds of
| milliseconds for cloud database services that run outside
| your data center.
|
| The OP's approach of SQLite & Litestream removes that latency
| entirely since the database is colocated with the
| application. Litestream will stream backups to an external
| service (e.g. S3) and that can have tens or hundreds of
| milliseconds of latency but that won't affect the
| application's performance since it is asynchronous
| replication.
| jfengel wrote:
| Ah, I had misread how that worked. That makes a ton of
| sense. Thank you.
| benbjohnson wrote:
| Litestream author here. Great explanation of how everything
| works! That's awesome that's only running at $0.03/month. :)
| mtlynch wrote:
| Hey Ben! Thanks again for your work on Litestream and for
| your help with this post. I'm excited to see Litestream
| continue growing in popularity.
| orf wrote:
| > It's also important to note that Litestream can't resolve
| conflicts between multiple database writes, so each database can
| have only one application server with write access.
|
| It's cool, don't get me wrong, but given the above constraint
| isn't it just: while True: sleep(1)
| upload_file("sqlite.db")
|
| with extra bells and whistles?
| mtlynch wrote:
| Thanks for reading!
|
| Litestream is doing more magic than it seems, so it's much
| better than a naive while loop.
|
| Litestream is only streaming the _changes_ in the database,
| whereas the naive implementation would have to upload the
| entire database every second.
|
| Also, SQLite has a write-ahead log, so you're not guaranteed
| that all the data is always in the main .db file. Litestream
| watches the write-ahead log files as well and immediately
| streams those changes to cloud storage. A system that only
| watches the main .db file would have long time windows where
| data loss is possible.
___________________________________________________________________
(page generated 2021-04-29 23:03 UTC)