[HN Gopher] Sqlite3 WebAssembly
___________________________________________________________________
Sqlite3 WebAssembly
Author : whatever3
Score : 305 points
Date : 2024-10-15 17:45 UTC (5 hours ago)
(HTM) web link (sqlite.org)
(TXT) w3m dump (sqlite.org)
| dang-lover wrote:
| I like it
| szundi wrote:
| We like that you like it
| bsimpson wrote:
| And we loved you in Lethal Weapon.
| akutlay wrote:
| Thankss for sharing
| TiredGuy wrote:
| So after downloading from the official downloads page and
| stripping away all the mjs files and "bundler-friendly" files, a
| minimal sqlite wasm dependency will be about 1.3MB.
|
| For an in-browser app, that seems a bit much but of course wasm
| runs in other places these days where it might make more sense.
| jt2190 wrote:
| The thing to keep in mind is that the WebAssembly sandbox model
| means that in theory the program (SqlLite in this case) can run
| wherever it makes sense to run it. That might mean running it
| locally or it might mean running on a central server or it
| might mean running nearby on the "edge".
| flockonus wrote:
| It's a good consideration, together with the fact browsers
| already have IndexedDB embedded. One use case still for in-
| browser apps like Figma / Photoshop-like / ML apps, where the
| application code and data is very big anyway, 1.3Mb may not add
| that much
|
| Also worth considering parsing of wasm is significantly faster
| than JS (unfortunately couldn't find the source for this claim,
| there is at lease one great article on the topic)
|
| https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_A...
| aidos wrote:
| When we built our frontend sync system we tried a few
| different options. We had a fairly simple case of just trying
| to store entities so we could pull incremental updates since
| you were last online. The one we ran in production for a
| while was IndexedDB but found the overhead wasn't worth it.
|
| I played around with warm sqlite too. That was really nice
| but I decided against it due to the fact that it was totally
| unsupported.
| jsheard wrote:
| It's pretty compressible at least, sqlite3.js+wasm are 1.3MB
| raw but minifying the JS and then compressing both files with
| Brotli gets them down to 410KB.
| rmbyrro wrote:
| A lot of HTML's nowadays have 100 - 300 kb. That's only the
| HTML (!!).
|
| Adding 400 for such a high quality piece of DB actually
| borders reasonability.
|
| And makes me think: what the hell are frontend devs
| thinking!? Multiple MB's in JS for a news website. Hundreds
| of KB's for HTML. It's totally unreasonable.
| coder543 wrote:
| 1.3MB seems perfectly reasonable in a modern web app,
| especially since it will be cached after the first visit to the
| site.
|
| If you're just storing user preferences, obviously don't
| download SQLite for your web app just to do that... but if
| you're doing something that benefits from a full database,
| don't fret so much about 1MB that you go try to reinvent the
| wheel for no reason.
|
| If the other comment is correct, then it won't even be 1.3MB on
| the network anyways.
| telotortium wrote:
| A megabyte here, a megabyte there, pretty soon you're talking
| about a really heavyweight app.
| zdragnar wrote:
| Given how hefty images are, a full database doesn't seem
| too bad for the purpose of an "app" that would benefit from
| it, especially when compression can being the size down
| even lower.
| littlecranky67 wrote:
| We are past the stage where every piece of JS has to be
| loaded upfront and delay the first meaningful paint. Modern
| JS frameworks and module are chunked and can be eager/lazy
| loaded. Unless you make the sqlite DB integral part for
| your first meaningful page load, preloading those 1.3MB in
| the background/upon user request is easy.
| ncruces wrote:
| For server side, you'll likely need a different build of Wasm
| SQLite, that handles concurrency (and file locking)
| differently.
|
| Also, WASI is very far from answer (so far). The SQLite
| amalgamation builds fine for WASI but concurrency is an
| unsolved issue.
|
| I had to build a VFS from scratch to get my Wasm based SQLite
| driver into a usable shape.
|
| https://github.com/ncruces/go-sqlite3/blob/main/vfs/README.m...
| hawski wrote:
| Is there a way to statically compile an application with SQLite
| and the result WASM was smaller. So for example I have an app
| that would use only a specific subset of SQLite. Could the
| SQLite's WASM be built with this in mind cutting down on code
| that is not used? Or is there a way to prune it having the used
| API surface?
|
| In a regular compiler/linker scenario it would just be a static
| compilation. Here we have a JS app and WASM library.
| hoten wrote:
| Since SQL takes arbitrary strings as input, this would
| require explicit compiler flags to disable the knobs you
| don't want. Can't rely on excluding unused symbols really.
| simonw wrote:
| Slight point of confusion: that page says:
|
| > These components were initially released for public beta with
| version 3.40 and will tentatively be made API-stable with the
| 3.41 release, pending community feedback.
|
| But the most recent release of SQLite is 3.46.1 (from 2024-08-13)
|
| Presumably they are now "API-stable" but the page hasn't been
| updated yet.
|
| It would be great if the SQLite team published an official npm
| package bundling the WASM version, could be a neat distribution
| mechanism for them. (UPDATE: They do, see replies to this post.)
|
| My favourite version of SQLite-in-WASM remains the Pyodide
| variant, which has been around since long before the official
| SQLite implementation. If you use Pyodide you get a WASM SQLite
| for free as part of the Python standard library - I use that for
| https://lite.datasette.io/ and you can also try it out on
| https://pyodide.org/en/stable/console.html
| import sqlite3 print(sqlite3.connect(':memory:').execute(
| 'select sqlite_version()' ).fetchall())
|
| That returns 3.39.0 from 2022-06-25 so Pyodide could do with a
| version bump. Looks like it inherits that version from
| emscripten: https://github.com/emscripten-
| core/emscripten/blob/main/tool...
| rblank wrote:
| https://github.com/sqlite/sqlite-wasm
|
| sqlite-wasm loads much faster than Pyodide, so if you don't
| need Python, then the former is a better choice.
| simonw wrote:
| Amazing! npm install @sqlite.org/sqlite-
| wasm
| wg0 wrote:
| This would run on client side I presume? Where the data
| would go?
|
| Okay that's listed here:
| https://sqlite.org/wasm/doc/trunk/persistence.md
|
| EDIT: Self answered.
| simonw wrote:
| I built my own little demo page here:
| https://tools.simonwillison.net/sqlite-wasm
|
| With the help of Claude, though it incorrectly hallucinated
| some of the details despite me pasting in documentation: ht
| tps://gist.github.com/simonw/677c3794051c4dfeac94e514a8e5b.
| ..
| Ciantic wrote:
| > It would be great if the SQLite team published an official
| npm package bundling the WASM version, could be a neat
| distribution mechanism for them.
|
| I think they've been doing that for a while, in JS script you
| can already do this: import sqlite3InitModule
| from "https://cdn.jsdelivr.net/npm/@sqlite.org/sqlite-
| wasm/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs";
| const sqlite3 = await sqlite3InitModule({
| locateFile(file: string) { return
| "https://cdn.jsdelivr.net/npm/@sqlite.org/sqlite-wasm/sqlite-
| wasm/jswasm/sqlite3.wasm"; }, });
| // SQLite's C API const capi = sqlite3.capi;
| console.log("sqlite3 version", capi.sqlite3_libversion(),
| capi.sqlite3_sourceid()); // OO API example below
| oo1 docs https://sqlite.org/wasm/doc/tip/api-oo1.md
| const oo = sqlite3.oo1; const db = new oo.DB();
| const createPersonTableSql = ` CREATE TABLE IF NOT
| EXISTS person ( id INTEGER PRIMARY KEY
| AUTOINCREMENT, name TEXT NOT NULL, age
| INTEGER NOT NULL ); `;
| db.exec([createPersonTableSql]);
|
| It works in regular old script tag with type=module, or Deno. I
| have example HTML here:
|
| https://github.com/Ciantic/experimenting-sqlite-wasm/blob/ma...
| CodeWriter23 wrote:
| > It would be great if the SQLite team published an official
| npm package bundling the WASM version, could be a neat
| distribution mechanism for them.
|
| You may benefit from perusing the FAQ on that page.
| jarpineh wrote:
| You can use DuckDB WASM independently of Pyodide and can extend
| it with SQLite.
|
| Though it seems to be somewhat limited. I couldn't even check
| what version it has, since sqlite_version() was missing.
| Version in the repository [1] is 3.38.1, which is from quite a
| ways ago.
|
| At the moment DuckDB web shell can't load SQLite extension,
| since that hasn't been released for yesterday's 1.1.2. Earlier
| version does work using recently updated WASM edition. That can
| be extended with spatial including GDAL, vector search etc [2].
| Making your own "SQL web shell" wasn't too hard, though docs
| weren't quite complete enough for me.
|
| [1]
| https://github.com/duckdb/sqlite_scanner/blob/main/src/sqlit...
| [2] https://github.com/duckdb/duckdb-wasm/releases/tag/v1.29.0
| sgbeal wrote:
| > Presumably they are now "API-stable" but the page hasn't been
| updated yet.
|
| That's correct. i'll try my best to remember to update that
| reference the next time i'm back on the computer.
|
| > It would be great if the SQLite team published an official
| npm package
|
| Not a chance. We publish only vanilla JS and adamantly refuse
| to go down the rabit hole of supporting out-of-language tools
| (none of which any of our project members use). We support an
| "officially sanctioned" npm build, maintained by Thomas
| Steiner, but do not actively develop for any JS frameworks.
|
| Direct support for any given framework (npm included) would
| give the impression that we endorse that framework, and
| endorsement of third-party projects is something we actively
| avoid.
| rezonant wrote:
| > We publish ONLY vanilla JS and adamantly refuse to go down
| rabit hole of supporting the frameworks du jour
|
| A bit confused at this, NPM is just a package manager /
| distribution mechanism, not a framework. Totally fair if you
| don't want to publish for all the package managers, though
| for Javascript there's only a few that are relevant. NPM has
| been around for a decade.
| sgbeal wrote:
| > A bit confused at this, NPM is just a package manager /
| distribution mechanism, not a framework
|
| It's an out-of-language packaging/distribution framework
| (and it's not the only one). It's not part of the JS
| standards.
|
| My comments above have been edited to reframe our stance on
| npm and frameworks in general.
| rezonant wrote:
| I don't think there will ever be a package manager
| dictated by the Ecmascript standards.
| outlore wrote:
| i've been looking for a Tanstack Query style library that is
| backed by Sqlite (backed by OPFS or some other browser storage)
| and syncs with an API in the background. Does anything like that
| exist? i've seen ElectricSQL and other sync engines but they are
| a bit opinionated. I'm pretty new to local-first but i feel like
| the developer ergonomics are not quite there yet
|
| Meanwhile for "local-only" it would be great to use sqlite in the
| browser + native file system API so that the db could be stored
| on the user's file system and we wouldn't have to worry about
| browser storage eviction. i think that could really open up a
| whole world of privacy preserving offline software delivered
| through the browser
| netghost wrote:
| ElectricSQL and friends seem to be the best option so far, but
| they all come with a lot of caveats. It feels like local-first
| is near, and it's so tantalizing, but I haven't seen anything
| that feels like it's done enough to build on just yet.
| ochiba wrote:
| Not sure if you've looked at PowerSync yet:
| https://www.powersync.com/ (I'm on the team)
|
| For the read path it hooks into Postgres logical replication or
| MongoDB change streams (and MySQL binlog soon). It supports
| partial syncing using declarative rules. For the write path, it
| allows writing to the local SQLite database and also places
| writes into an upload queue, and then uses a developer-defined
| function to upload writes to the backend API.
|
| We did a deep dive on current options for SQLite on the web,
| and are currently using an IndexedDB-based VFS, and looking to
| move to OPFS: https://www.powersync.com/blog/sqlite-
| persistence-on-the-web
|
| We recently released an integration with TanStack Query to
| allow leveraging some of its features in conjunction with
| PowerSync: https://docs.powersync.com/client-sdk-references/js-
| web/java...
|
| > Meanwhile for "local-only" it would be great to use sqlite in
| the browser + native file system API so that the db could be
| stored on the user's file system and we wouldn't have to worry
| about browser storage eviction. i think that could really open
| up a whole world of privacy preserving offline software
| delivered through the browser
|
| Agreed. This is a limitation of IndexedDB and OPFS as
| persistent browser storage currently
| gagik_co wrote:
| I have working to replicate TanStack query experience by
| writing my own queries wrapped around PowerSync, although
| actually on Flutter (using Flutter Hooks! which was cool to use
| coming from React). It's a very internal design meant for my
| app tetr[1] right now (and actually being migrated over from
| Realm). I am hoping to potentially standardize it and publish a
| package once it's mature enough but not too wrap to make your
| own hooks around them for your needs.
|
| [1] https://tetr.app
| gnarbarian wrote:
| How long until we see WebAssembly/WebGPU become a platform
| independent choice for deploying server side code as well?
| ruined wrote:
| as soon as wasi is settled
| gnarbarian wrote:
| https://wasi.dev/
|
| wow I didn't know this was a thing. thanks for filling me in!
| paulddraper wrote:
| Yesterday?
|
| There's a number of WASM platforms/tools: Wasmer, wasmCloud, a
| few others that escape my memory.
| jt2190 wrote:
| https://wasmer.io/
|
| https://wasmcloud.com/
|
| https://wasmtime.dev/
| stackskipton wrote:
| SRE here, it's currently happening in a few parts but overall,
| it's not as attractive on server side. Server Side code running
| is mostly a solved problem and for very few organizations, the
| benefits of WASM don't outweigh any difficulties in getting it
| running.
| 6gvONxR4sf7o wrote:
| > Server Side code running is mostly a solved problem
|
| I know what you mean here, but I think we're very limited in
| what we tend to run. Polyglot programming still isn't really
| a thing, and with things like WASI standardized (someday soon
| I hope), I could imagine it becoming a lot nicer.
| evacchi wrote:
| Shameless plug for Dylibso's latest beta launch :)
| https://www.getxtp.com/blog/meet-xtp
|
| disclaimer: I work there
| Thaxll wrote:
| It's kind of dying on the server, some people thought it would
| replace containers.
| me551ah wrote:
| After years of being able to run SQLite on my mobile phone, my
| tv, my router and gaming consoles, I can finally run it on my
| browser. Which also happens to be running on the most powerful
| machine I own
| ibash wrote:
| surprise! it's been there for decades:
| https://en.wikipedia.org/wiki/Web_SQL_Database
| adregan wrote:
| It _was_ there for a decade: https://caniuse.com/sql-storage
| joemi wrote:
| I wonder why it was unmaintained/dropped. Was there
| something wrong with it, and if so, would that also apply
| to this kind of wasm implementation?
| debugnik wrote:
| Mozilla refused to support it because then every
| implementation would have simply used SQLite, which would
| have promoted any implementation details to a _de facto_
| standard. (Even caniuse erroneously describes the feature
| as "allows SQLite database queries".)
|
| From the latest spec [1]:
|
| > The specification reached an impasse: all interested
| implementors have used the same SQL backend (Sqlite), but
| we need multiple independent implementations to proceed
| along a standardisation path.
|
| [1]: https://www.w3.org/TR/webdatabase/
|
| This won't be a problem for wasm SQLite because it isn't
| a standard being shipped by browsers, just another
| dependency.
| xyc wrote:
| i have a feeling that it set back the web by a decade
|
| https://x.com/chxy/status/1822858746307170640
| akira2501 wrote:
| Did they really assume that they were going to be able to
| _restandardize_ SQL? No wonder IndexDB is hot useless
| garbage.
|
| The standardization issues around SQL already exist, are
| already widely known, and where common workarounds are
| already in practice. It's also an open source project
| that could have _easily_ incorporated compatibility code
| for this specific use case anyways.
|
| They made blind fealty to process more important than the
| outcome to end users.
|
| What a waste.
| koeng wrote:
| For use in Golang, I really like ncruces wasm SQLite package -
| https://github.com/ncruces/go-sqlite3 . Unlike cznic's go package
| (which is great, btw), the wasm version works well on OpenBSD and
| the like.
| ncruces wrote:
| Author here. If you're interested, do ask questions.
| TN1ck wrote:
| Very cool project! Do you know if this would be possible for
| duckdb? Is there something about sqlites APIs and wasm build
| that made it feasible?
|
| Context: Currently using go-duckdb and while it's working for
| us, getting rid of cgo would be a huge help. Would be quite
| interested myself to attempt this.
| ncruces wrote:
| I don't know much about DuckDB's architecture.
|
| Wasm is fine for compute (though concurrency is still a
| somewhat open question).
|
| To have Wasm talk to the outside world, you need "host
| calls" where the guest calls the host.
|
| On a browser that's Wasm calling JavaScript. On my Go
| driver, it's Wasm calling Go.
|
| For server side, there's also a standard set of "host
| calls" modeled around POSIX/Linux syscalls called WASI.
|
| I could've build my project around WASI, but WASI is rather
| limited (and SQLite support for WASI was more limited even,
| it's improved a bit since). DuckDB might work out-of-the-
| box this way.
|
| I, instead, took advantage of SQLite's architecture and
| replaced its VFS layer with one in Go:
| https://sqlite.org/vfs.html
|
| So SQLite in Wasm is just doing compute, and I do all the
| OS level stuff in Go. No need for Wasm concurrency, cause I
| can load multiple instances of my Wasm which act like
| independent OS processes that communicate through the
| filesystem (SQLite excels at this).
|
| As I said, I dunno how well all those decisions would map
| to DuckDB.
| koeng wrote:
| > So SQLite in Wasm is just doing compute, and I do all
| the OS level stuff in Go. No need for Wasm concurrency,
| cause I can load multiple instances of my Wasm which act
| like independent OS processes that communicate through
| the filesystem (SQLite excels at this).
|
| Interesting. So when I am running concurrent readers
| using your package, it is just loading multiple instances
| of the wasm code? (I bottleneck to a single writer in the
| application)
| brandonpollack2 wrote:
| I was trying to get this working in a rust ecosystem some time
| ago but none of the blessed.rs sql (rusqlite, sqlx) wrappers seem
| to take advantage of it yet and wrapping it yourself is a bit
| tricky since when I was trying I couldn't figure out a way to to
| get emscripten wasm code to play nice with wasm32-unknown-unknown
| without some kind of JS wrapper which then requires implementing
| the interface those crates expect and exposing it from JS. Once
| that is done in rust itll be great there too!
| tonygiorgio wrote:
| Yeah I've been waiting awhile for this myself. A few PRs with
| work pending for a year or so. I've seen some proof of concepts
| but nothing anywhere close to usable.
| aabhay wrote:
| I have been working on one. If you're interested in working on
| it or contributing, feel free to chime in here:
|
| https://github.com/rhashimoto/wa-sqlite/discussions/154
|
| This essentially requires that we import the sqlite emscripten
| build via an extern C header in wasm bindgen, and then we need
| to re-implement the VFS in rust while compiling it in multi-
| threaded mode to allow for shared array buffer access. After
| that is all done, we will be able to access SQLite rows as raw
| wasm bytes. That gives us the ability to implement a rust-
| sqlite style wrapper or integration. There would still not be
| some of the niceties such as connection pooling, but in wasm
| you likely want to use the db in exclusive mode.
| insipx wrote:
| Got this SQLite build working with rusts diesel sqlite here:
| https://github.com/xmtp/diesel-wasm-sqlite
|
| I'm gearing up for a 0.2 release this week which should iron
| out a few kinks, but otherwise you can use the diesel ORM as if
| its native
| jjcm wrote:
| As a general question, in what scenarios is it more beneficial to
| send the full DB and let the browser handle the queries? Maybe
| phrased a better way - when would I use this to improve a user
| experience over the traditional server-hosted db model?
| bryanrasmussen wrote:
| >when would I use this to improve a user experience over the
| traditional server-hosted db model?
|
| just my intuition when I read the headline of this post -
| something like the interplay between PouchDB and CouchDB for
| offline first apps
|
| https://medium.com/offline-camp/couchdb-pouchdb-and-hoodie-a...
| harrisi wrote:
| For offline use it can be good when dealing with large amounts
| of data. Anything from like an audio library to 3D modeling
| software. Changes can be made locally and persisted and then
| you can sync things server side regularly or when online again.
| ThatPlayer wrote:
| Personally I'm using it for a statically hosted website, so a
| server-hosted database was never an option. Also with the right
| driver, it's possible to stream the chunks of the database as
| needed rather than sending the full database:
| https://github.com/mmomtchev/sqlite-wasm-http
|
| I can even do Sqlite's full text search without downloading the
| entire FTS database. Just most of it, if the search term is
| short enough.
| simonw wrote:
| Something that would be really fun would be to run SQLite in-
| memory in a browser but use the same tricks as Litestream and
| Cloudflare Durable Objects
| (https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st...)
| to stream a copy of the WAL log to a server (maybe over a
| WebSocket, though intermittent fetch() POST would work too).
|
| Then on subsequent visits use that server-side data to rehydrate
| the client-side database.
|
| From
| https://sqlite.org/forum/info/50a4bfdb294333eec1ba4749661934...
| is looks like WAL mode is excluded from the default SQLite WASM
| build so you would have to go custom with that.
| dustinchilson wrote:
| Are you thinking something like https://electric-sql.com/
| PUSH_AX wrote:
| What's the catch with this thing?
| T-Winsnes wrote:
| The security model is challenging, as it relies on Postgres
| users for iam. Your users essentially log directly into
| your db
| fcanesin wrote:
| Isn't a more advanced/production version of this what the combo
| of couchdb and pouchdb do since several years ago?
| simonw wrote:
| Yeah this kind of thing has certainly been tried before, I
| feel like SQLite WASM plus WAL might be an interesting twist
| on the idea.
| ncruces wrote:
| There are many layers of that's not how it works at play here.
|
| In-memory SQLite databases don't use WAL. Wasm (and browser
| Wasm, in particular) doesn't support anything like the shared
| memory APIs SQLite wants for its WAL mode.
|
| Litestream requires a very precise WAL setup to work (which
| just so happens to work with the default native SQLite setup,
| but is hard to replicate with Wasm).
|
| Cloudflare Durable Objects may have been inspired by Litestream
| but works very differently (as do LiteFS, Turso, etc...)
|
| The general idea of streaming changes from SQLite would work,
| but it's a lot of work, and the concurrency model of in-browser
| Wasm will make it challenging to implement.
|
| (I wrote that forum post some time ago, and have WAL working in
| a server side Wasm build of SQLite, but none of the options to
| make it work would make much sense, or be possible, in browser)
| digdugdirk wrote:
| As someone who uses sqlite fairly regularly, but doesn't
| understand what most of those paragraphs mean, do you have
| any recommendations for learning resources?
|
| I'm gathering that I need to learn about: - WAL - Shared
| Memory APIs - Concurrency models - Durable Objects?
| wyager wrote:
| WAL: Write ahead log, common strategy for DBs (sqlite,
| postgres, etc.) to improve commit performance. Instead of
| fsync()ing every change, you just fsync() a log file that
| contains all the changes and then you can fsync() the
| actual changes at your leisure
|
| Shared memory API: If you want to share (mutable) data
| between multiple processes, you need some kind of procedure
| in place to manage that. How do you get a reference to the
| data to multiple processes, how do you make sure they don't
| trample each other's writes, etc.
|
| Concurrency model: There are many different ways you can
| formalize concurrent processes and the way they interact
| (message passing, locking, memory ordering semantics,
| etc.). Different platforms will expose different
| concurrency primitives that may not work the same way as
| other platforms and may require different reasoning or code
| structure
|
| Durable objects - I think this is some Cloudflare service
| where they host data that can be read or modified by your
| users
|
| This is all from memory, but IME, GPT is pretty good for
| asking about concepts at this level of abstraction
| jauntywundrkind wrote:
| Then don't use in-memory sqlite? Use file backed sqlite but
| have your wasm implementation of those "system calls" just be
| to memory?
|
| I dunno, feels like you're coming down too hard.
| jchanimal wrote:
| If you like, solving these sort of problems, we are tackling
| them at Fireproof.
|
| Our database API is modeled on CouchDB and MongoDB, but our
| storage abstractions are along the lines of what you need to
| build the multi writer WAL you describe.
|
| More details here https://jsr.io/@fireproof/encrypted-
| blockstore
| baq wrote:
| See also https://github.com/electric-sql/pglite (REPL at
| https://pglite.dev/repl/)
|
| (Previously discussed 7 months ago:
| https://news.ycombinator.com/item?id=39477457)
| bhelx wrote:
| I used the wasm build of sqlite and the Chicory runtime to create
| a pure JVM executed sqlite library:
| https://github.com/dylibso/sqlite-zero
|
| It's more of an experiment than an attempt to make something
| production ready, though I could see it being useful to bring
| dependency-less sqlite tooling to the JVM ecosystem.
| ncruces wrote:
| What's the file system access like, WASI?
| bhelx wrote:
| Chicory has some partial wasip1 support.
| https://github.com/dylibso/chicory/tree/main/wasi. We use
| jimfs to keep things simple and secure (and not worry about
| exposing the real filesystem):
| https://github.com/google/jimfs
|
| When I did this experiment a few months ago, what we could
| accomplish was pretty limited. I could load and query
| databases, but not write to them. However the Chicory wasip1
| implementation is advancing.
|
| BTW, we've borrowed a few ideas from wazero so thanks for
| your work there :)
| ncruces wrote:
| If the goal is to improve Chicory WASI support, this is the
| way.
|
| If the goal was pure Java SQLite1, a VFS from scratch would
| be better.
|
| I think since I started my Go/wazero effort, WASI+SQLite
| improved a bunch. I had to start with the demo VFS; the
| Unix VFS now builds. But custom VFS is still the way to go,
| IMO.
|
| And thanks! My contributions to wazero were tiny. Best of
| luck with Chicory!
|
| 1: strong NestedVM vibes here; 11 years ago... gosh, I feel
| old now. https://stackoverflow.com/questions/18186507/pure-
| java-vs-na...
| bhelx wrote:
| > If the goal was pure Java SQLite1, a VFS from scratch
| would be better.
|
| agreed, though this was more an experiment to test
| Chicory once we built initial wasi support. I'd love to
| see it picked up and improved. I think that's the
| direction I'd go if i want some kind of production ready
| library.
| benthecarman wrote:
| Whats needed is a rust-wasm compatible library that can use this.
| aabhay wrote:
| If you're interested in contributing to that, here is a good
| place to start (in early discussion stages):
|
| https://github.com/rhashimoto/wa-sqlite/discussions/154
| catapart wrote:
| I wasn't able to tell from a quick look through the page: could
| someone help me understand the use cases here?
|
| More specifically, would this be able to be a "replacement" for
| indexedDB? Does the data persist, or do I need to keep the sqlite
| file in the filesytemAPI (or indexedDB/localstorage) myself?
| azangru wrote:
| From the about page:
|
| > Specific Goals of this Project
|
| > Insofar as possible, support persistent client-side storage
| using available JS APIs. As of this writing, that includes the
| Origin-Private FileSystem (OPFS) and (very limited) storage via
| the window.localStorage and window.sessionStorage backend.
| catapart wrote:
| Right but, to my eyes, that's vague?
|
| What I'm asking is if I need to manage the sqlite file, as I
| would on an OS's file system, or if accessing the sqlite
| library will automatically persist that data to those web-
| native storages, like the way indexedDB doesn't require me to
| load an "idb" file and then "save" or "commit" that save. I
| just access it and write.
|
| To be clear: I'm not asking academically. I wrote a whole
| library for managing data in indexedDB for local-first apps,
| and while it works well enough for what I need, it's iDB so
| it's subject to data deletion (not common, but allowed in the
| spec if necessary), and it's a pain to work with just because
| of its nature and API. So I've been waiting to move to sqlite
| for a while with the only holdbacks being "is it too heavy?",
| and "how much has to change?". With WASM, I think we're about
| as lightweight as its going to get. So I'm just curious if
| this aims to be a drop-in replacement, or if it still expects
| you to use it like sqlite on a native platform.
| parhamn wrote:
| WebSQL should've just been Sqlite and the whole offline-first
| (and general app storage) ecosystem would've been so much nicer.
|
| Is there any hope of that happening? Instead of abstracting and
| over specifying sqlite, can the spec just specify a version of
| the SQLite API browsers should support and roll the version
| periodically?
| emn13 wrote:
| "Rolling the version periodically" is probably quite
| problematic for browsers. Kind of a key point of the web is
| that stuff if at all possible keeps working. Breaking changes
| like that are hard.
|
| Even if the spec just listed occasional version and the webpage
| could choose which one; that means a potentially tricky
| maintenance burden on browser to support old versions of a
| potentially no longer supported sqlite, and each version is
| another megabyte.
|
| Why not then just choose this solution, and let each website
| pick its own poison?
|
| If the concern is the repeated downloads of common resources,
| well, we've accepted that for other CDN's too, and a solution
| for shared caching of common dependencies would in any case be
| more valuable than merely for sqlite.
|
| The current approach seems better than a browser-provided
| version.
| simonw wrote:
| I for one am glad WebSQL didn't establish itself.
|
| Now we get the most recent version of SQLite when we need it as
| a 410KB compressd WASM blob, as opposed to being stuck on
| browser-mandated versions of SQLite that might even be a decade
| old at this point.
| chrysoprace wrote:
| I've been really interested in the local-first landscape lately
| but embedding SQLite seems really heavy-weight compared to using
| the browser's built-in storage APIs (in particular, IndexedDB)
| and it seems to be what most of the main open source libraries
| do. I'm interested to see a open-source solution (with sync)
| which provides an SQLite-like API but for the browser's native
| storage rather than trying to embed another executable in Web
| Assembly.
___________________________________________________________________
(page generated 2024-10-15 23:00 UTC)