[HN Gopher] Turso SQLite Offline Sync Public Beta
___________________________________________________________________
Turso SQLite Offline Sync Public Beta
Author : charlieirish
Score : 215 points
Date : 2025-03-31 15:10 UTC (1 days ago)
(HTM) web link (turso.tech)
(TXT) w3m dump (turso.tech)
| billconan wrote:
| This sounds great, but I have some questions regarding data
| integrity and security.
|
| If I build an offline first app using Turso, will my client
| directly exchange data with the database, without a layer of
| backend APIs to guarantee data integrity and security? For
| example, certain db write is only permitted for certain users,
| but when the db API is exposed, will that cause problems? A
| concrete example would be a forum where only moderators can
| remove users and posts. Say if I build an offline first forum,
| can a hacker hack the database on the filesystem and utilize the
| syncing feature to propagate the hacked data to the server?
| justanotheratom wrote:
| That is a very crucial question. I am also interested in the
| answer.
|
| Perhaps they have RLS type policies that are only modifiable on
| the server.
| tracker1 wrote:
| I'm pretty sure you'll have to write parts of your app against
| your own APIs that represent the owner of the db for a group.
|
| With Turso, you would want a model that had, for example a db
| per user and one per group. With the turso model you want to
| think something closer to sharding by hand for secure write
| user or group.
|
| I could be wrong on this though. That's just my rough
| understanding.
| thisislife2 wrote:
| I'd have thought that in this day and age every developer would
| know by now the importance of _sanitizing user input_ before a
| web application accepts it? Your doubt has given me some pause
| ...
| wahnfrieden wrote:
| No need to give a rude, condescending and unhelpful answer -
| there will always be people learning
| setr wrote:
| If the database is local, your web app database access is
| local. It can be modified and changed by the user, unlike
| code hosted on the web server, and _any sanitization can thus
| be bypassed_.
|
| Meaning the user has effectively direct access to the
| underlying local database. Which, if blindly and totally
| synced, gives the user effectively direct access to the
| central database.
|
| I'd have thought that in this day and age every developer
| would know by now the importance of _not trusting frontend
| validation_ in a web application? _your doubt_ has given me
| some pause.
| krashidov wrote:
| This is my problem with these local first libraries. What
| happens if there's some data that needs to live in a db that's
| separate from the replicated sqlite db?
|
| What I would really love is a sync engine library that is
| agnostic of your database.
|
| Haven't really seen one yet.
| vekker wrote:
| Exactly. So many local first libs don't cover this that it
| makes me wonder if the applications I am typically working on
| are so fundamentally different from what the local-first devs
| are normally building?
|
| Most apps have user data that needs to be (partially or
| fully) shielded from other users. Yet, most local-first libs
| neglect to explain how to implement this with their
| libraries, or sometimes it's an obscure page or footnote
| somewhere in their docs, as if this is just an
| afterthought...
| ochiba wrote:
| It's definitely quite a hard engineering problem to solve,
| if you try to cover a wide range of use cases, and layer on
| top of that things like permissions/authorization and
| scalability
| franciscop wrote:
| The blog post doesn't even touch on write conflicts, which is
| the main reason I opened it (I was curious on how they solved
| them), so not surprised there's no many details about security
| etc.
| refulgentis wrote:
| You raise an interesting point, that along with the replies,
| compels me to note that _all of this stuff is bespoke_ , and
| things that sound simple like "I just want a good syncing
| library" are intractable in practice.
|
| Ex. if I'm doing a document-based app, users can have at it,
| corrupt their own data all they want.
|
| I honestly cannot wrap my mind around discussions re: SQLite x
| web dev, perhaps because I've been in mobile dev: but I don't
| even know what it'd mean to have an "offline-first forum" that
| syncs state: it's a global object with shared state rendered on
| the client.
|
| When you set aside the implications introduced by using a hack
| scenario, a simpler question emerges: How would my clients sync
| the whole forum back to the cloud? Generally, my inclination is
| to handwave about users being able to make posts and have it
| "just work", after all, can't Turo help with simple scenarios
| like a posts table that has a date column? That makes it
| virtually conflict free...but my experience is "virtually"
| bites you, hard.
| nightowl_games wrote:
| Honestly this is so simple and core to the idea that I
| literally just assume it's handled.
| ochiba wrote:
| I am not sure about Turso but I've seen a few different
| approaches to this with other sync engine architectures:
|
| 1. At a database level: Using something like RLS in Postgres
|
| 2. At a backend level: The sync engine processes write
| operations via the backend API, where custom validation and
| authorization logic can be applied.
|
| 3. At a sync engine level: If the sync engine processes the
| write operations, there can be some kind of authorization layer
| similar to RLS enforced by the sync engine on the backend.
| aboodman wrote:
| Yes, this is a central issue in sync. For most applications,
| sync engines just aren't useful without some solution. Of
| course you need to validate inputs, support fine-grained
| permissions, etc., as developers have done with web apps for
| eons.
|
| In Replicache, we addressed this by making your application
| server responsible for writes:
|
| https://doc.replicache.dev/concepts/how-it-works
|
| By doing this, your server can implement any validation it
| wants. It can also interact with external systems, do
| notifications, etc. Anything you can do with a traditional API.
|
| In our new sync engine, Zero (https://zerosync.dev), we're
| adding this same ability soon (like this week) under the name
| custom mutators:
|
| https://bugs.rocicorp.dev/issue/3045
|
| This has been a hard project, but is really critical to use
| sync engines for anything serious.
| isaachinman wrote:
| Happy user of Replicache. You and the team got it right.
| anovick wrote:
| Is Turso open-source? is libSQL open-source?
| cultofmetatron wrote:
| turso is a SAAS built on libsql. libsql itself is opensource
| canadiantim wrote:
| Mind you they're migrating from libsql to Turso (formerly
| called Limbo), a rust-based rewrite of sqlite entirely
| adamrezich wrote:
| Also important to note that, by all appearances, they seem
| to intend to Embrace Extend Extinguish sqlite3 with said
| rewrite.
| jakelazaroff wrote:
| That's a tall order; by SQLite's own estimates, SQLite is
| used more than all other databases combined and may even
| be the most widely deployed software component, ever:
| https://www.sqlite.org/mostdeployed.html
| adamrezich wrote:
| Yes, which is why I believe the entire effort to be
| pretty silly.
| rustc wrote:
| > they seem to intend to Embrace Extend Extinguish
|
| What makes you think that? Limbo is MIT licensed and I
| don't see any CLA on [1].
|
| [1]: https://github.com/tursodatabase/limbo/blob/main/CON
| TRIBUTIN...
| adamrezich wrote:
| It is very interesting that there is no "code of conduct"
| in the Limbo project, given that there is one in libsql
| [0]--I wonder if they realized that playing the whole
| "the SQLite Code of Ethics is bad and wrong, actually"
| card [1] wasn't turning out to be as popular of a move as
| they thought it would be. Or, at the very least, they
| realized that if you're going to shit on SQLite for
| having a "code of ethics", then it's better to _not_
| provide a "code of conduct" rather than to provide one
| (even though the entire impetus for SQLite writing their
| Code of Ethics was some projects required they have some
| sort of "code of conduct" in order to use SQLite...)
|
| I know it's super lame to regurgitate LLM output in
| comments, but I found it interesting that Grok was able
| to infer [2] what I was talking about when I made this
| accusation on X some months ago, and it absolutely nailed
| basically everything I had in mind:
|
| ---
|
| Q: Why might Adam think Limbo is an example of "EEE" so
| "brazen" as to resemble Microsoft in the 90s?
|
| A: Adam Rezich's comment suggests that Pekka Enberg's
| project, Limbo, could be seen as an example of "Embrace,
| Extend, Extinguish" (EEE) tactics, which were notably
| used by Microsoft in the 1990s. Here's a breakdown of why
| Adam might think this:
|
| 1. Embrace: Limbo starts by embracing SQLite, a widely
| used, open-source database engine. By aiming for full
| compatibility with SQLite, Limbo acknowledges and adopts
| the existing technology, similar to how Microsoft would
| integrate or support existing standards or technologies
| in their products.
|
| 2. Extend: Limbo extends SQLite by rewriting it in Rust,
| adding new features like built-in vector search and
| asynchronous I/O, which are not part of the original
| SQLite. This extension phase is where Microsoft
| historically added proprietary features or extensions to
| standards, making their versions more appealing or
| functional in certain ways.
|
| 3. Extinguish: Although Limbo hasn't reached this phase
| yet, the potential for extinguishing could arise if Limbo
| becomes superior enough to make SQLite less relevant or
| if it leads to a market shift where SQLite users migrate
| to Limbo due to its enhanced capabilities. Microsoft in
| the 90s would often extend standards in such a way that
| their versions became the de facto standard,
| overshadowing or making competitors' versions obsolete.
|
| The "brazen" aspect comes from the transparency and
| public nature of the project. Unlike Microsoft's more
| secretive and strategic approach in the past, Limbo's
| development is open, with Enberg sharing his plans and
| progress publicly. This openness, combined with the
| ambitious goal of completely rewriting a foundational
| piece of software like SQLite, might seem bold or even
| audacious, reminiscent of Microsoft's aggressive market
| strategies but done in a more transparent and community-
| driven manner.
|
| ---
|
| The only thing Grok didn't mention was the overt emphasis
| on the "community" aspect of this project, which is being
| promoted as being strictly, even _obviously_ better than
| sqlite3 's way of doing things. For me personally,
| sqlite3 _not_ being a "community"-focused project is
| actually a huge advantage--modern programming
| "communities", while being good for building hype and
| allegiance for one 's project, generally result in very
| unpleasant nth-order social effects which have nothing to
| do with the quality of software.
|
| [0] https://github.com/tursodatabase/libsql/blob/main/COD
| E_OF_CO...
|
| [1] https://x.com/glcst/status/1887148191344148681
|
| [2] https://x.com/i/grok/share/6PMLqGr2JETaSGF8hhVxs0WJk
| SchwKatze wrote:
| Technically the project name remain being Limbo until the
| milestone[1] be reached. AFAIK the platform continue to use
| libSQL, given the early stage of Limbo.
|
| [1] - https://github.com/tursodatabase/limbo/milestone/1
| no_wizard wrote:
| I wonder if they will subject limbo to (If I recall
| correctly) Jepsen tests.
| dyu- wrote:
| They are open-core. The offline-write feature
| (libsql::Builder::new_synced_database) basically does not
| work with the bare `sqld` server on their `libsql` github
| repository.
|
| In fairness though their
| `libsql::Builder::new_remote_replica` works with the bare
| `sqld`
| avinassh wrote:
| disclosure: I work at Turso.
|
| libSQL is our fork of SQLite. It adds new features on top of
| SQLite, such as vector search, replication, server mode, and
| more. libSQL is open source (MIT license).
|
| The Turso SaaS platform, which provides hosted databases, is
| not open source.
|
| Limbo (which will be renamed to Turso in the future) is our
| Rust rewrite of SQLite. It is also open source (MIT license) -
| https://github.com/tursodatabase/limbo
| anonzzzies wrote:
| Seems not really or only partially anyway; I cannot see, with
| disasters like Fauna, that anyone would trust their core stuff
| with something not open source. But maybe it's just me. I need
| to be able to switch and sure, I can switch to the open source
| fork libSQL (open core I think, so nope) (which they are
| rewriting to Rust for some bizarre reason; sqlite is one of the
| most readable, robust and well tested codebases in the business
| so it looks like burning vc money, but whatever), but once my
| business depends on their offline / replication / etc, then I
| cannot switch to anything, so never going to happen.
|
| Again, opinion. It's _core infra_ , in my opinion at least that
| should _never_ depend on others or, if the others inevitably
| screw you over for a few $, you need to be able to move without
| possibly bankrupting your company.
| DaveMcMartin wrote:
| I would like to test it with a large database to see how it
| handles a 3-5 GB database sync.
|
| In the example, it shows syncing returning a promise. Is there no
| way to track the progress of the sync?
| srameshc wrote:
| I really admire Turso. This Offline Sync is probably like
| ELectric offering but if it will work as anticipated , lot of
| read only data can be closed to the frontend and can improve
| performance and save cost.
| klabb3 wrote:
| What is the realtime and reactivity support in Turso? It's
| starting to look attractive but do updates propagate
| automatically without polling?
| avinassh wrote:
| (I work at Turso)
|
| libSQL does not support realtime/reactivity features. There
| have been attempts (by both our team and the community), but we
| encountered difficulties and ultimately abandoned the idea. I
| can check with the team for the exact reasons.
|
| However, Limbo (our SQLite rewrite in Rust) will include this
| feature, though there is no ETA yet. It's currently being
| discussed here:
| https://github.com/tursodatabase/limbo/discussions/1149. Please
| share your thoughts and use cases.
| klabb3 wrote:
| Thanks. My personal use case is a client with business logic
| written in Go with intermittent connectivity, some data is
| private to the device, some private to the server, and with
| some data synced from server to clients. Many events are time
| critical and need instant updates.
|
| To me it's more general than that though. The expectation of
| having reactive behavior comes up everywhere from chat apps
| to collaborative to even games. Polling and refreshing feels
| very 90s and "web page centric" but of course in some cases
| (say long form blogging or similar) it's not a concern.
| ezekg wrote:
| It'd be nice if the post went into how conflict resolution will
| (?) work because that's the hard part here and the main selling
| point imo.
| bob1029 wrote:
| Conflict resolution can't work in a general sense.
|
| How you reconcile many copies of the same record could depend
| on time of action, server location, authority level of the
| user, causality between certain business events, enabled
| account features, prior phase of the moon, etc.
|
| Whether or not offline sync can even work is very much a domain
| specific concern. You need to talk to the business about the
| pros & cons first. For example, they might not like the
| semantics regarding merchant terminals and offline processing.
| I can already hear the "what if the terminal never comes back
| online?" afternoon meeting arising out of that one.
| ochiba wrote:
| I would say this is why server-authoritative systems that
| allow for custom logic in the backend for conflict resolution
| work well in practice (like Replicache, PowerSync and Zero -
| custom mutators coming in beta for the latter). Predefined
| deterministic distributed conflict resolution such as CRDT
| data structures work well for certain use cases like text
| editing, but many other use cases require deeper
| customizability based on various factors like you said.
| jahewson wrote:
| CRDT falls flat for rich text editing though. So many nasty
| edge cases and nobody has solved them all, despite their
| claims.
| mentalgear wrote:
| have you tried loro
| mmerlin wrote:
| https://loro.dev/
| ezekg wrote:
| Server-authoritative conflict resolution kind of mirrors my
| thinking as well, having resolution work like multiplayer
| net code, where the client and server may or may not
| attempt to resolve _recent_ conflicts, but server has the
| final say on state. Just not sure how this plays out when a
| client starts dropping conflicting data because the server
| says so...
| ochiba wrote:
| Yes, exactly. https://www.gabrielgambetta.com/client-
| side-prediction-serve...
| jahewson wrote:
| This. It's so obvious when you think about it, but everybody
| wants a free lunch.
| larkost wrote:
| CFRDT (Conflict Free Replicated Data Types) can absolutely
| reconcile many-writers situations. There are a number of
| these systems, and they all have their own rules around that
| replication (sometimes very complicated rules that are hard
| to reason about). As long as you can live inside those rules,
| and accept that they are going to have sharp corners that
| don't quite make sense for your use case, then you can get a
| virtually free lunch there.
|
| But living inside of those rules (and sometimes just
| understanding those rules) can be a big ask in some
| situations, so you have to know what you are doing.
| mystifyingpoi wrote:
| I think this point confuses me the most in this regard:
|
| > Local-first architectures allow for fast and responsive
| applications that are resilient to network failures
|
| So are we talking about apps that can work for days and weeks
| offline and then sync a lot of data at once, or are we talking
| about apps that can survive a few seconds glitch in network
| connectivity? I think that what is promised is the former, but
| what will make sense in practice is the latter.
| hnthrow90348765 wrote:
| Local-first is overkill for transient faults. This is
| probably meant for outage and disaster scenarios.
| ochiba wrote:
| There are niche use cases where the former (work for days to
| weeks offline) are useful and even critical - like certain
| field service use cases. Surviving glitches in network
| connectivity is useful for mainstream/consumer applications
| for users in general, especially those on mobile.
|
| In my experience, it can affect the architecture and
| performance in a significant way. If a client can go offline
| for an arbitrary period of time, doing a delta sync when they
| come back online is more tricky, since we need to sync a
| specific range of operation history (and this needs to be
| adjusted for specific scope/permissions that the client has
| access to). If you scale up a system to thousands or millions
| of clients, having them all do arbitrary range queries
| doesn't scale well. For this reason I've seen sync engines
| simply force a client to do a complete re-sync if it "falls
| behind" with deltas for too long (e.g. more than a day or
| so.) Maintaining an operation log that is set up and indexed
| for querying arbitrary ranges of operations (for a specific
| scope of data) works well.
| Matthias247 wrote:
| I'm wondering too.
|
| In general this seems to work only if there's a single offline
| client that accepts the writes.
|
| With limitations to the data scheme (e.g. have distinct tables
| per client), it might work with multiple clients. However those
| would need to be documented and I couldn't see anything in this
| blog post.
| titaphraz wrote:
| A lot of offline sync projects just drop data on conflict and
| pretend it didn't happen. It's the salesman job to divert your
| questions about it.
|
| I found another blogpost from turso where they say they offer 3
| options on conflict: drop it, rebase it (and hope for no
| conflict?) and "handle it yourself".
|
| Writing an offline sync isn't hard. Dealing with conflicts is a
| PITA.
|
| https://turso.tech/blog/introducing-offline-writes-for-turso...
| cheema33 wrote:
| What are some of the differences from Electric SQL?
| thruflo wrote:
| Electric is a read-path sync engine for Postgres that does
| partial replication [1]. It's agnostic to your choice of client
| -- you can sync into anything you like. It doesn't handle or
| mandate a pattern for syncing local writes back to Postgres.
| Instead, it's designed to allow you to handle concerns like
| writes and auth using your existing API [2].
|
| Turso Offline Sync is an active-active replication system for
| distributed SQLite/Turso databases. IIUC, this release adds the
| capability to sync local writes back to the cloud [3].
|
| [1] https://electric-sql.com [2] https://electric-
| sql.com/blog/2024/11/21/local-first-with-yo... [3]
| https://turso.tech/blog/turso-offline-sync-public-beta#what-...
| 0x6c6f6c wrote:
| Since enough comments seem to focus on consolidating writes and
| security issues, please see the post. They explicitly state this
| as something they're working on.
|
| > we're working hard on the following features: > > Automatic and
| manual conflict resolution
|
| As such everything within this thread is conjecture unless
| otherwise informed by their work.
| SahAssar wrote:
| And it is probably not useful for most applications until there
| is some sort of security there.
| kmetan wrote:
| Looks good, but i am fine with midas.dll
| senderista wrote:
| LOL "[conflict] resolution is not yet implemented"
| shanth wrote:
| how about gun.js with sqlite3 storage adapter
| https://github.com/gundb/sqlite
| benwilber0 wrote:
| I just wrote a post about this kind of stuff and why it's (almost
| always) nonsense. You will spend more time and effort trying to
| shoe-horn SQLite into a server database than you will ever get
| any benefits from.
|
| https://benwilber.github.io/programming/2025/03/31/sqlite-is...
| crazygringo wrote:
| Agreed. I've been seeing a lot of these posts lately, about
| embedding SQLite into web servers.
|
| I think a lot of people just don't realize how few resources
| Postgres or MySQL use, and how fast they are. You can run
| Apache and MySQL and a scripting language on a tiny little 512
| MB memory instance, and serve some decent traffic. It works
| great.
|
| Wanting to use SQLite and deal with replication is a nightmare.
| I don't get it. (And I _love_ using SQLite in apps and scripts.
| But not websites!)
| GrumpyCat42 wrote:
| Yup - I just recently learned this lesson the hard way with
| Turso's LibSQL server. While some of the features (like s3
| replication) are cool and interesting, the amount of time
| working around multiple writers and foreign key shenanigans was
| not worth it when Postgres would have just gotten the job done.
| NathanFlurry wrote:
| Can you elaborate on both of these issues?
|
| Vanilla SQLite solved multiple writers a long time ago when
| they introduced WAL in 2010. Does Turso not support this?
|
| Is the issue with foreign keys that they're not enabled by
| default?
| joelkoen wrote:
| > Point-of-Sale Systems -- process transactions regardless of
| internet connectivity
|
| Mentioning this as an example use case is simply untrue, right?
| aabhay wrote:
| I think this might be true in an invoicing based world but I
| agree that PoS is a misnomer. But I can imagine e.g. a field
| sales rep in construction taking orders on site in a rural area
| then syncing up when they return to connectivity.
| brikym wrote:
| Does Turso have any plans to support streaming queries for real
| time applications?
| progx wrote:
| Does it support type safety? Or exists an add / module on top of
| turso that support it?
___________________________________________________________________
(page generated 2025-04-01 23:02 UTC)