[HN Gopher] SQLite Wasm in the browser backed by the Origin Priv...
___________________________________________________________________
SQLite Wasm in the browser backed by the Origin Private File System
Author : bubblehack3r
Score : 293 points
Date : 2023-01-12 13:11 UTC (9 hours ago)
(HTM) web link (developer.chrome.com)
(TXT) w3m dump (developer.chrome.com)
| throwaway284534 wrote:
| This is honestly very cool. VS Code uses a similar approach to
| their local file system provider, albeit with a wrapper around
| IndexDB instead of SQLite. There's some interesting trade offs
| too, since IndexDB can store the browser's native file handlers
| in a flat map -- so there's no need for a schema.
|
| IMO, the Chrome team is being a bit deceptive with their phrasing
| on synchronous file handles. The problem is that the entire API
| is wrapped up in an asynchronous ceremony.
| `createSyncAccessHandle` is only available in a worker context.
| So you can only communicate with the worker using an asynchronous
| postMesssage event dispatcher. And even when you're in the
| worker, file handles can only be accessed through methods that
| return a promise.
|
| I understand the need for such boundaries when working with a
| single threaded language, but limiting the synchronous APIs to
| just workers seems like one too many layers of indirection. I
| recently attempted to write a POSIX-style BusyBox library and
| this sort of thing was a total show stopper.
| bobkazamakis wrote:
| >limiting the synchronous APIs to just workers seems like one
| too many layers of indirection
|
| An unresponsive script is slowing this window down - kill
| process ot wait?
| throwaway284534 wrote:
| The file handler is already tucked inside an asynchronous
| Promise based API. I think it's reasonable that a mere
| attempt to get a handler synchronously is made possible.
| Whether to support synchronous read/write access is another
| matter.
|
| I would be delighted if handlers were more akin to byte
| arrays that could be written and read synchronously, albeit
| with an asynchronous function to persist changes to the disk.
| tarasglek wrote:
| I wanted to use it in a chrome extension worker, but sync version
| of OPFS isn't available in that context. Hope this gets fixed.
| Existenceblinks wrote:
| The sqlite.wasm is 683kB and the sqlite.js is 300kB. So is it
| supposed to be used in PWA context where these files are cached?
| Why no one is talking about CAP here. Is it real-production-
| ready?
| simonw wrote:
| What do you mean by CAP?
|
| 683KB plus 300KB is less than many web pages these days.
| Existenceblinks wrote:
| The theorem. Where is the source of truth. Does every offline
| app now require CRDT or OT?
|
| Hm.. 1 MB is good (just zero app code) ..okay.
| dboreham wrote:
| It's just a database (not distributed, not replicated),
| that you can use from your browser-hosted application. What
| you do with it is up to you.
| est wrote:
| TIL OPFS.
|
| https://developer.mozilla.org/en-US/docs/Web/API/File_System...
|
| What's next, mmap() for javascript?
| simula67 wrote:
| Sockets, hopefully.
|
| For example, it should be possible to build websites for
| downloading YouTube videos that does not use server side code
| execution or bandwidth. These were popular at some point but
| disappeared after Java applets stopped being supported by most
| browsers.
| jeroenhd wrote:
| And make every website part of a botnet? I sure hope not.
|
| WebSockets are already a problem because many web devs don't
| know that other origins can connect to them and potentially
| fetch data they're not supposed to (even ignoring the fact
| you can construct these without a browser), unleashing raw
| sockets to the web platform would be hell.
|
| WebRTC and WebSockets provide more than enough already. If
| you need even more, your browser just isn't the place for
| this type of code IMO.
| madacol wrote:
| Instead of fighting against that feature, why not instead
| ask permission from the user?, just like the APIs of
| location and notifications do
| tracker1 wrote:
| Or... you can use a websocket gateway if you need this
| type of functionality... a lot of people already are for
| things like VNC or Terminal interfaces.
| heywhatupboys wrote:
| i dont understand how this is not already possible?
| lxgr wrote:
| Or just an XMPP web client.
| pjmlp wrote:
| You had to ask, https://www.npmjs.com/package/mmap-io
| mr_toad wrote:
| https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScr...
| fabiospampinato wrote:
| That's available already, in Bun.
| wvenable wrote:
| What till you hear about WebUSB.
| Kelteseth wrote:
| Fun fact: The headline image is the Stadtbibliothek Stuttgart,
| the nicest library I ever visited:
| https://www.archdaily.com/193568/stuttgart-city-library-yi-a...
| tomayac wrote:
| 0711 love! Viele Grusse!
| mckravchyk wrote:
| I think this will be great for extensions. Currently the only
| solid choice is to use the dead simple storage.local which only
| allows to retrieve things by ID.
|
| There's one problem though, this new API is for the web, so the
| nature of this storage is temporary - obviously the user must be
| able to clear it when clearing site data and this what makes it
| currently an unviable solution for a persistent extensions data
| storage.
| https://bugs.chromium.org/p/chromium/issues/detail?id=138321...
| encryptluks2 wrote:
| Agreed, people want permission based file system access and so
| I don't see why a site can't prompt a user to store a file in a
| user-specified location. If they clear their site settings then
| giving permission to the same file should retain their data. I
| don't want to have to worry about losing something important or
| having my browser act as a file system.
| mr_toad wrote:
| > I don't see why a site can't prompt a user to store a file
| in a user-specified location.
|
| This is already possible with the file system access API.
| Clearing site data has no impact on these files. But you do
| have to re-grant permission every time you reload the page.
|
| https://developer.mozilla.org/en-
| US/docs/Web/API/File_System...
| dfghjkjhg wrote:
| yeah because full access to the filesystem is only solution.
|
| asking for permission to a longer storage.local exemption for
| the user? noooo let's not even think about giving user control
| of things. let's the European union pass post-fact legislation
| after file system access is abused by adnetworks. then they get
| the blame of being annoying, not the browser vendor who sell
| ads.
|
| let's stop being naive. there are a million ways to fix that
| bug with better UX and respect for user control, but this is a
| scape goat to have that feature.
| helf wrote:
| Yay. More ways for people to force webapps upon us.
|
| I miss when the web was essentially a document viewer.
| [deleted]
| swyx wrote:
| > I miss when the web was essentially a document viewer.
|
| said unironically on a web app
| helf wrote:
| No I did think about that ;)
| cudgy wrote:
| ... that looks like a document
| Existenceblinks wrote:
| This comment is literally as a table
| tregoning wrote:
| <meme>Why not both?</meme>
| helf wrote:
| Because they end up migrating more and more functionality to
| their web side? Deprecating desktop or mobile app
| functionality or entirely.
|
| So you end up with piss poor "web apps" that half the time
| require a specific browser to work remotely well and have
| kludged "offline" modes that break every other day.
|
| Do not want. I know I'm probably in the minority but it's
| damned irritating. The UX on most is atrocious and you have
| to deal with a complete hodgepodge of interface styles and
| ideas and designs and also hope your browser keeps running
| smoothly after trying to use multiple more advanced webapps
| simultaneously.
|
| Oh, but all these advancements on wasm and all will continue
| improving performance and fixing problems!!!
|
| ... you mean stuff that was already solved with native
| programs? "But you can work on your stuff from anywhere with
| just an account!"
|
| Oh, like a native app's data synchronization?
|
| But but...
|
| It would be nice if we had both and both were equals. But
| that isn't the case. It won't be the case. And most companies
| see this as a wonderful way to vendor lock in even more than
| before and implement more and more IAP and subscription crap.
|
| No thanks.
| endisneigh wrote:
| How is that any different than desktop apps that require
| you to be online and only work with particular versions of
| particular operating systems?
| sirjaz wrote:
| When it is a desktop or even a mobile app I as the end
| user have a level of control over my environment that I
| don't with a webapp. In addition, my interface doesn't
| change at the whim of the company or developer.
| endisneigh wrote:
| A completely offline web app can have the same
| properties.
| helf wrote:
| Please show me a completely offline web app that's worth
| a damn. I'm unaware of any.
|
| Also, "completely offline webapp" is such a damn oxymoron
| lol.
| samwillis wrote:
| The web is an open platform accessible to all and anyone.
| Mobile platforms in comparison are, depending on which one,
| either a walled garden or nearly so. Web apps are better for
| humanity than the closed ecosystem of other platforms.
| rektide wrote:
| I have yet to see whether implementations allow users direct
| access via the filesystem to these contents. The idea of each
| origin having some files sounds fine. But if I genuinely want to
| help & empower my users, give them the most user agency, I'd want
| them to be able to access the sqlite sb themselves directly, with
| whatever hazards they thereby assume.
| samwillis wrote:
| From my reading previously, no the currently OPFS
| implementations do not expose the files as files in directories
| the user would access.
|
| The File System Access API does give the webpage the ability
| access files on the users file system, with permission. However
| not at the block level, that is only enabled for the OPFS. In
| order to provide ACID compliance SQLite needs to be able to
| hold a lock to a file and write at the block level, which is
| only possible within the OPFS sandbox.
| pfoof wrote:
| Cybersec is like "here we go again"
| [deleted]
| typingmonkey wrote:
| Using SQLite in the browser is great until you have users that
| open your app in more then one browser tab.
| Existenceblinks wrote:
| Or many devices while data is consistent. Where is source of
| truth. Per user or per resource or per service. A lot of
| questions.
| tantaman wrote:
| What's wrong with many tabs? The DB handles getting hit by many
| tabs correctly -- it's just as if many processes were hitting
| the db in a non-browser environment.
|
| Example of syncing your app across tabs with SQLite here:
| https://vlcn.io/docs/guide-solving-tabs
| omniglottal wrote:
| If your app cannot handle concurrency while your back-end i/o
| is via a machine-local singleton, that sounds like a a problem
| which will come up for more than just multiple tabs. If
| architecting around this known-fact is a challenge, an engineer
| may have bigger problems.
| [deleted]
| samwillis wrote:
| See here: https://sqlite.org/wasm/doc/trunk/persistence.md#vfs-
| locking
|
| It's mostly works with OPFS but with a few edge cases. They are
| working with the browser vendors on improving this in OPFS.
|
| Alternatively run it in a SharedWorker
| (https://developer.mozilla.org/en-
| US/docs/Web/API/SharedWorke...), which seems to have made a
| comeback after being dropped for security concerns. Or do some
| sort of leader election with browser tabs and use a
| BroadcastChannel.
| tantaman wrote:
| SQLite WASM uses SharedArrayBuffers and Atomics to turn async
| OPFS apis into blocking APIs (really just _one_ api --
| getSyncAccessHandle)
|
| Given the use of SharedArrayBuffers and that
| SharedArrayBuffers can't be used in a shared worker (funny),
| SQLite WASM doesn't work in shared workers.
|
| If getSyncAccessHandle was made synchronous then maybe
| everything would just work. It'd also improve SQLite WASM
| perf by 30% according to their measurements [1]
|
| [1] https://sqlite.org/forum/forumpost/af64f73911e5410cf9a640
| c9e...
| littlecranky67 wrote:
| You can't write to SQLite from multiple processes either, so
| you mitigate that exactly the same way you do for a desktop app
| (i.e. show "Can't access DB for writing because other instance
| is running")
| tristanho wrote:
| Does anyone have a good idea of when OPFS will be broadly
| available in all major browsers? And what's best to use as a
| pseudo-polyfill in the mean time?
|
| Something relevant is absurd-sql, but unfortunately that's not
| even close to production ready :/
|
| https://github.com/jlongster/absurd-sql
| DylanSp wrote:
| Does anyone have hands-on experience with how well different
| browsers support this? This blog post and caniuse both say Chrome
| support is currently somewhat limited (though being improved);
| caniuse also mentions that Safari (both desktop and iOS) only
| support the Origin Private File System part of the File System
| API, and that Firefox doesn't support it all. I'm curious how all
| that actually shakes out in practice.
| Existenceblinks wrote:
| Yeah, so tired of excitement without thinking the full
| production process.
| tomayac wrote:
| The Origin Private File System is coming to all browsers,
| Chrome, Safari, and Firefox. It's a universally agreed-on Web
| standard: https://fs.spec.whatwg.org/.
| DylanSp wrote:
| Just because the standard exists doesn't mean every browser
| has already implemented 100% of it, hence why I was asking if
| anyone's had any experience actually trying to use it.
|
| EDIT: I saw from your other comments on this post that
| Firefox has implemented at least the origin private file
| system part in nightly builds; _that 's_ useful info that I
| was looking for.
| tomayac wrote:
| Yeah, sorry. There's a lot of side threads to read and
| potentially respond to.
| dboreham wrote:
| What happens is that one browser (usually Chrome) implements a
| useful new feature. Then applications are released that use
| that feature. Then the other browser vendors start hearing from
| their users that they suck because they don't support the nice
| new feature and hence can't run the nice new applications. And
| eventually they add support.
| dmitriid wrote:
| > What happens is that one browser (usually Chrome)
| implements a useful new feature.
|
| What happens is Chrome releases a Chrome-only non-standard
| (at a rate of 400 APIs per year). It cobbles together a
| barely legible spec [1], and then uses its multiple
| propaganda channels like web.dev to present them as actual
| completed standards that other browsers are just slow to
| implement.
|
| Even if other browsers only learn about this a few days
| before Chrome ships, or have multiple objections.
|
| [1] Example, File System API https://wicg.github.io/file-
| system-access/ "not a W3C Standard nor is it on the W3C
| Standards Track."
| tomayac wrote:
| Please see
| https://developer.chrome.com/blog/capabilities/#how-will-
| we-... for the actual process. Nothing happens behind
| closed doors; other vendors can chime in at any time, and
| are actually very much encouraged to do so.
| dmitriid wrote:
| That's a nice sentiment at the link. The _reality_ is
| that Chrome ships hundreds of APIs with little to no
| input and only barely notifies other browser vendors that
| "yeah, we're shipping it".
|
| We're literally in the discussion about File System API
| that is:
|
| - not on any standards track
|
| - considered harmful by other browser vendors
|
| - shipped by default in Chrome
|
| Edit: I can give several examples from Web Transport to
| HID to Constructible Stylesheets that were either shipped
| with no input from other vendors, or shipped against the
| _multiple_ objections from other browser vendors. And
| that 's just the ones I know about.
| encryptluks2 wrote:
| Simply not true but believe what you need to.
| dmitriid wrote:
| I literally provided just some of the examples. Those are
| easily verifiable.
|
| Web Transport is shipped by default. What was the input
| from other browsers?
|
| Here's the timeline for HID:
| https://github.com/mozilla/standards-
| positions/issues/459#is...
|
| Constructible Stylesheets: the spec contained a trivially
| reproducible race condition, the API was badly specified.
| Google shipped against any objections and refused to
| bring it back under the flag. Full discussion here:
| https://github.com/WICG/construct-stylesheets/issues/45.
| Shipped in Chrome https://github.com/WICG/construct-
| stylesheets/issues/45#issu... (may be hidden on mobile)
| despite _multiple_ unresolved issues. Two years later
| Chrome did add a better API that people originally
| requested, other issues potentially remain.
|
| You may believe in benevolent Google working hand in hand
| with others for the betterment of the web. The _reality_
| is radically different.
|
| Edit: more about hardware APIs:
| https://news.ycombinator.com/item?id=34356651
|
| Edit2: As to the number of APIs, Chrome actually tracks
| how much they ship: https://web-
| confluence.appspot.com/#!/confluence
| simonw wrote:
| What do you think of the Chrome origin trial process?
| dmitriid wrote:
| It's a good idea, or at least originally it was a good
| idea. In reality it devolved to, well, see the links I
| provided.
|
| Edit:
|
| Note: those three are the ones I remember because I was
| reading about them at the time, or participating in some
| form in the discussion. How many are there that just slip
| through? Well, you can kinda see for yourself (second
| hand guessing, by proxy etc.):
|
| 1. MDN lists various web APIs, including experimental
| ones: https://developer.mozilla.org/en-US/docs/Web/API
| Almost every single one that's marked "experimental" is
| shipped by default in Chrome
|
| 2. Mozilla lists its position on various standards.
| Almost all of those marked as "considered
| harmful/negative" are shipped by default in Chrome
| https://mozilla.github.io/standards-positions/
| (exceptions are a few deprecated and removed ones like
| HTML Imports)
|
| 3. Chrome currently ships 1100 more APIs more than
| Firefox: https://web-confluence.appspot.com/#!/confluence
| (note: a single standard like "File System Access API"
| may expose dozens of these APIs) Is it because Firefox is
| lagging, or is it because Chrome just powers through, all
| consensus be damned?
| skybrian wrote:
| The underlying API seems to be available for multiple browsers
| but not Firefox yet:
|
| https://developer.mozilla.org/en-US/docs/Web/API/FileSystemF...
| yyyk wrote:
| The Web SQL standard was deprecated (by Mozilla) on the grounds
| of a single implementation.
|
| The result today: A single SQL implementation with no standard
| (beyond SQLite) and therefore no easy way for anyone to create a
| compatible alternate implementation.
| [deleted]
| ngrilly wrote:
| Nothing is preventing you to "create a compatible alternate
| implementation".
| CJefferson wrote:
| However, now every webbrowser doesn't have to keep it correct,
| secure and up to date for the next 40 years.
| lxgr wrote:
| Yes, now every single site using SQLite has to do that.
| simonw wrote:
| Thanks to Mozilla rejecting Web SQL we now get to run the exact
| version of SQLite we need, compiled to WASM and downloaded to
| the browser rather than being baked in and unable to upgrade.
|
| SQLite not right for a project? DuckDB has a WASM build too.
|
| Heck, so does PostgreSQL now!
| https://www.crunchydata.com/blog/crazy-idea-to-postgres-in-t...
| yyyk wrote:
| (Replying to this and other comments in unison)
|
| A SQL standard never precluded a separate filesystem access
| standard and implementing databases on top of it.
|
| But rejecting the Web SQL standard means that every SQL-using
| webapp will be tied to its own database - replicating the SQL
| incompatibility mess when we had a chance to finally enforce
| a little standardization based on current SQL standards - and
| require depending on WASM when WASM might have not been
| necessary.
| skybrian wrote:
| A SQL-level standard wouldn't do much because different
| apps will be using different database schemas anyway. (It's
| similar to how two applications that both use JSON format
| usually aren't going to be compatible, because they use it
| to store different things.) Only generic, schema-
| independent database tools and libraries would be helped.
|
| For a portable application file format, you'd want to pick
| a SQL implementation (probably SQLite), decide on a schema,
| and probably wrap it in a library, but once you do that it
| could be used in multiple apps.
| acdha wrote:
| > But rejecting the Web SQL standard means that every SQL-
| using webapp will be tied to its own database - replicating
| the SQL incompatibility mess when we had a chance to
| finally enforce a little standardization based on current
| SQL standards
|
| Web SQL was dropped for precisely this reason:
|
| > The specification reached an impasse: all interested
| implementors have used the same SQL backend (Sqlite), but
| we need multiple independent implementations to proceed
| along a standardisation path.
|
| http://www.w3.org/TR/webdatabase/
|
| Without a magic want to force multiple implementations, all
| of those web apps would have been tied to a single
| implementation anyway _and_ it would have been harder to
| update it so even after something was fixed you'd need to
| wait until the oldest browsers were updated before you
| could rely on it. That is possible, of course, but it's a
| recipe for problems with something like a SQL database.
| xenadu02 wrote:
| SQLite is the one project that deserves an exception.
| There is a near zero chance anyone else can do better
| with similar quality (and test coverage!)
|
| Sometimes you need to be pragmatic IMHO.
| samwillis wrote:
| SQLite fits one generic model, and is a row level data
| store. DuckDB is a column store, and other DBs are
| designed to operate for specific use cases. This is open
| to all options.
|
| The point is that SQLite is a brilliant genetic option,
| but should not be the only option. WASM and OPFS is the
| right route forward to enable application developers to
| use the best data store architecture for their use case.
|
| Even within SQLite there are extension that would never
| have been an available to WebSQL that with this design
| are.
| treis wrote:
| But we have 0 options now and 1 is better than 0.
| acdha wrote:
| I certainly would agree that SQLite is exemplary but
| consider how many different things are handled under the
| aegis of the W3C: if you allow one exception, then you
| will have other people saying their favorite thing should
| also have one.
| josefx wrote:
| So how many implementations of widevine are around? Web
| standards are well past the point where anyone involved
| could claim the procedural, moral or intellectual high
| ground.
| acdha wrote:
| The apparent inconsistency here is because you're
| misunderstanding what Wide Vine is:
|
| The standard is Encrypted Media Extensions:
| https://www.w3.org/TR/encrypted-media/
|
| That provides a standard way for web developers to say
| "Our audio/video files are encrypted. Use this decoder to
| decrypt the bytes for playback.". Wide Vine is a popular
| Content Decryption Module used by EME but it's not the
| only one (Apple devices use FairPlay, Microsoft ships
| PlayReady) and the standards process required
| interoperability to be demonstrated.
|
| Web SQL's standardization process halted precisely
| because nobody was interested in building a second
| implementation. If Web SQL had a second implementation,
| it could have been standardized just like EME was.
| [deleted]
| yyyk wrote:
| This ignores the fact that SQL _already_ has standards
| with multiple implementations and is very well
| established. You can have a basic portable subset - the
| problem was already very well specified at the offline
| level.
|
| A simple SQL standard would therefore not have incurred
| an unnecessary dependency on SQLite (had WASM/filesystem
| existed at the time, I suspect we could have also
| supported Web SQL with it).
|
| --
|
| Bugs of course, exist with IndexedDb as well and will
| exist with these new WASM implementation.
| acdha wrote:
| > This ignores the fact that SQL already has standards
| with multiple implementations and is very well
| established. You can have a basic portable subset - the
| problem was already very well specified at the offline
| level.
|
| This is only partially true, as you can see from the many
| apps which only support one SQL database even if they're
| using something like an ORM which is ostensibly portable.
| Sometimes that's caused by use of things like using
| extensions but in other cases it can be caused by
| differences in things like data validation and conversion
| or how transactions are processed. That doesn't mean that
| portability is impossible but everyone doing it seriously
| tests continuously against multiple implementations to
| make sure they aren't inadvertently introducing a
| dependency on one flavor. Both SQLite and MySQL are
| especially notorious for this because you'd have cases
| where code which was invalid would either be
| automagically converted (hopefully to what you wanted) or
| the error silently suppressed so you'd have code which
| only works, for example, on a database where DATE can be
| "0000-00-00".
|
| As simple example, at the time Web SQL was proposed as a
| standard, SQLite did not support foreign key checks at
| all. It still does not implement full Unicode support so
| you'd have to require all implementations enable that
| extension with the same configuration or you'd have data-
| dependent application bugs.
|
| I would not personally have minded terribly if SQLite was
| effectively standardized like this but I understand why
| people wanted to see n>1 implementations. The browser
| developers are going to be supporting things for many
| years and as soon as a new feature launches any and all
| behaviors it exposes will become an API which has to be
| managed carefully to avoid breaking someone's production
| application.
| derefr wrote:
| _Standards_ are for platform implementors. _Developers_
| inevitably ignore standards, and instead code
| "empirically" to whatever the implementation they're
| developing against is willing to expose--both in terms of
| features (guaranteed by the standard or not), and in
| terms of bugs.
|
| (Yes, those devs are doing a bad thing. They _should_
| only obey the letter of the standard, and not "look
| through" to what the implementation is doing. But we
| can't just intentionally break support for the stuff they
| build through this empirical-discovery-of-features-and-
| bugs process. That would just hurt the innocent users who
| just want to access their websites -- which might very
| well be abandonware by the point that we discover and fix
| whatever standard-nonconformance bug such sites relied
| upon.)
|
| What we can do to control developer behavior, is to
| impose incentives.
|
| If there are multiple implementations of the standard in
| popular use in different browsers, each with _different_
| bugs, then developers will feel the need to support those
| different browsers, and so will test in them, find where
| the deviations from the standard cause breakage, and so
| end up coding portably.
|
| But if all the implementations in use in different
| browsers have the _same_ bugs (perhaps because they 're
| all actually the same implementation), then even
| supporting multiple browsers won't do anything to break
| people's assumption on those bugs; and so people _will_
| write brittle code that depends on exactly those bugs
| being present to work, and will break even if the
| platform implementor just upgrades the implementation to
| a newer version without the bug.
|
| This is the "bug-for-bug compatibility" that Microsoft
| worries so much about across Windows versions. It's a
| royal PITA to support, and W3C/WHATWG/etc have thusfar
| assiduously avoided _allowing_ anything into the
| standards which has only one implementation, and
| therefore which devs could ever "lock onto" the bugs of
| with brittle implementations.
|
| ---
|
| Also, this ignores that most of the "pragmatics" of
| managing a SQL DBMS _are not specified by the SQL
| standard_. SQL is not as standard as people think it is!
| ANSI SQL is mostly concerned with specifying how DML
| queries work; the DDL stuff is almost entirely a mess of
| implementation-defined syntax that happens to share some
| _conventions_ , but with no standards _guarantees_.
|
| Check this out:
| https://www.postgresql.org/docs/current/sql-
| createindex.html...
|
| > CREATE INDEX is a PostgreSQL language extension. _There
| are no provisions for indexes in the SQL standard._
|
| Do you really want to expose a SQL API where you can't
| create indices, because there's no portable syntax for
| creating indices? Would such an API even be _useful_?
|
| If we really wanted a "web SQL API", the designers of
| said API couldn't just lean on the existing SQL standard;
| they would actually have to standardize what "web SQL"
| syntax is _themselves_ -- probably as a superset of ANSI
| SQL, but still laying out a bunch of additional
| requirements. It 'd be similar in scope to WebGL. That's
| a _big_ ask.
|
| And even if they did that, that'd likely mean that they'd
| end up with _no_ existing implementations of the
| standard! (Because it 's not like you'd want to lock in
| all of SQLite's nonstandard DDL syntax choices as "the"
| WebSQL syntax, right? You'd want to think these out "from
| first principles", considering what's most
| "implementation-neutral.")
| yyyk wrote:
| I don't think there was so much risk of relying on db
| behaviour much less bugs. There's a DML standard,
| breaking it would have had massive consequences far
| beyond the web platform - we could have relied on SQLite
| being broadly compatible within the simple
| INSERT/UPDATE/DELETE/SELECT subset (I think more could
| have been done, but this is enough for a start).
|
| As for DDL, not much was initially required beyond CREATE
| TABLE, CREATE VIEW and CREATE (UNIQUE) INDEX (which in
| practice have a basic common syntax). IMHO the ask was
| not anywhere as complex as WebGL.
| abtinf wrote:
| Isn't the logical conclusion of this view the end of web
| browsers, replacing them with executable software
| distribution?
| cudgy wrote:
| Blasphemy! More fun to deal with compatibilities of running
| in different browsers. Seems like we have returned full
| circle (25 yrs later) to a new form of ActiveX plugins.
| acdha wrote:
| That's the difference between a platform and an application
| using that platform. If SQLite is baked into your browser, you
| can only use the subset of functionality supported by the
| oldest / weirdest browser which your users use.
|
| Think about the DOM APIs: jQuery was super popular in the
| 2000s. Say all of the browsers had decided to bundle it like
| WebSQL bundled SQLite. Would you be happy using a copy of
| jQuery 1.7 until all of Apple, Microsoft, Mozilla, and Google
| shipped 1.8 and all of your users upgrade or would you prefer
| to update your site to reference the version with the features
| you want to use? (or switch to a different framework in 5
| years)
|
| Browser developers always have to balance the tension of adding
| something new with the knowledge that they're going to need to
| support it for long periods of time. WASM is a good compromise
| position because it reduces performance as the reason why
| something should live in the browser engine rather than being
| shipped as an app resource.
| yyyk wrote:
| >jQuery was super popular in the 2000s. Say all of the
| browsers had decided to bundle it like WebSQL bundled SQLite.
|
| This ignores the fact that SQL _already_ has standards with
| multiple implementations and is very well established. It 's
| not a recent-ish JS library. It's not new tech. It's an
| almost 50 year old tech predating the internet.
|
| Standardizing on a basic standard subset using the current
| SQL standards would have been enough for nearly all apps, and
| would have allowed alternate implementations without any real
| SQLite path dependency since SQL already has standards and
| well trotted paths for implementing them (at least a basic
| subset).
|
| [EDIT: I don't wish to spam the comments. If you wish to
| reply to this, please use the other thread you also replied
| to? I should have done it myself, sorry about that.]
| treis wrote:
| >Would you be happy using a copy of jQuery 1.7 until all of
| Apple, Microsoft, Mozilla, and Google shipped 1.8 and all of
| your users upgrade
|
| I'd polyfill 1.8 until enough browsers support it like is
| done for every other browser api
| roblabla wrote:
| Or an alternative read of the result today: A simple
| implementation of filesystem access, on top of which any number
| of database (SQL or not) can be implemented, thus allowing for
| more innovation in the database space.
| systems wrote:
| what is exactly the use case for this?
| 0x_rs wrote:
| Plenty. Suppose you're working on a cross-platform application,
| building for web will get even easier, as up until a few months
| ago there weren't even official WASM builds from SQLite (but
| some amazing community run ones such as sql.js!). As for why
| somebody would want that, an example: your browser and
| everyone's is probably always open when you're at a PC,
| attrition is next to none compared to having to download an
| application, it's great for even just trying something out.
| SQLite really makes user data storage trivial in most
| circumstances, key:value doesn't.
| dboreham wrote:
| You want to run some code in the browser that wants to use a
| persistent SQL database.
| jchw wrote:
| Based on this note:
|
| > In our blog post Deprecating and removing Web SQL, we
| promised a replacement for Web SQL based on SQLite. The SQLite
| Wasm library with the Origin Private File System persistence
| backend is our fulfillment of this promise.
|
| I'd venture to guess the best answer is "whatever the hell
| people used Web SQL for." Doesn't really answer the question,
| but alas.
| [deleted]
| samwillis wrote:
| It's an alternative to using IndexedDB which is a browser
| provided API.
|
| On use cases, you can give your web app offline support by
| locally caching data in an SQL database and have it be fully
| queryable.
|
| Say you are building an app like Notion, they already have
| "offline mode" for the mobile and desktop apps, this would
| enable you to build that for the web app.
|
| This is very much one of the final jigsaw pieces needed to make
| PWAs (progressive web apps) competitive for the majority of use
| cases. We just need Apple to catch up and fill in a few other
| blanks too.
|
| A design pattern that is beginning to emerge is "offline/local
| first". You design your app to fundamentally work offline,
| using things such as this, and the server component only works
| to synchronise clients. It's a bit like the design move to
| "mobile first" that happed 10 years ago, but going to another
| level.
| wargames wrote:
| I haven't used IndexedDB in a while.. but my recollection is
| that it is basically SQLite. Not sure I see the advantage of
| this above and beyond what IndexedDB is already providing.
| What are the pros and cons of each approach?
| actionfromafar wrote:
| https://nolanlawson.com/2021/08/22/speeding-up-indexeddb-
| rea...
|
| IndexedDB seems like a whole bag of gotchas. SQLite +
| simpler browser provided file backing seems interesting to
| me.
| azangru wrote:
| > but my recollection is that it is basically SQLite.
|
| SQLite, which has a SQL database engine, is, by definition,
| a relational database. IndexedDB is a non-relational, or
| noSQL, database. One can't be basically the same as the
| other :-)
| AndrewSChapman wrote:
| IndexDb is not a relational data store. It's much closer to
| Mongo than it it to Sqlite.
|
| If you want relational tables, joins, aggregations etc, you
| want something like this (or the original Web SQL that was
| deprecated).
|
| There is definitely a use case for it.
| simonw wrote:
| One is a key value store, the other is a full relational
| database.
|
| Think Redis vs PostgreSQL.
| Existenceblinks wrote:
| "Offline/local first" sounds easy but it's the distributed
| problem. How is data be consistent?
| julianeon wrote:
| It feels like you're assuming that Apple is a student here
| that forgot to do his homework, as opposed to a miscreant who
| is trying to block progress on purpose.
|
| Apple wants to handicap web apps to encourage people to build
| for iOS.
| samwillis wrote:
| That may or may not be to some extent true, however I find
| negativity isn't generally the best way to encourage people
| to do something you would like.
|
| I think Safari has also made progress over the last couple
| of years, and there is indication of them being more
| receptive to PWAs. I'm going to cheerlead for that.
|
| It's also worth noting that the EU and UK competition
| authorities are looking at forcing Apple to make changes to
| some of their rules around web browsers on iOS. They have a
| big stick, I don't, hence my nicer words.
| phaedrus wrote:
| So thick client applications get the last laugh in the end?
|
| It reminds me of an old concept, that has a catchy name I
| can't recall, positing an endless cycle in systems evolution
| where local peripherals grow processing capability until
| someone notices and (re)centralizes it, only for the local
| processing capability to quietly begin growing again, and so
| on.
| maizek wrote:
| Fantastic, would love to know the name if you end up
| remembering. I see this pattern so often.
| cloudking wrote:
| Perhaps this could be used to create more privacy focused apps
| where your data stays local and works offline. I'm not sure
| about the persistence though, like if you clear the cache does
| it wipe out the local database?
| ehutch79 wrote:
| Or if you switch devices?
|
| Or need to collaborate on the data (i.e. a company with
| empoyees)
|
| Or have regulations/controls on where data lives 'at rest'
| azangru wrote:
| SQL but in the web browser. Mobile apps have long been using
| sqlite for their needs. Now web apps can too.
| [deleted]
| simonw wrote:
| It lets you build static web applications that can manage
| relational data for your users that is stored on their own
| machines.
|
| This is a really powerful capability. You could build the
| equivalent of full desktop applications - things like Word,
| Excel, Access, Evernote, task trackers, Blender... all without
| any server side storage mechanism at all.
| cudgy wrote:
| So basically a desktop app that runs in a desktop app on a
| desktop?
| simonw wrote:
| Yup - but crucially one that people can start using by
| clicking a link, without having to download and install
| anything first.
|
| And it's an app that runs in a very robust sandbox, so much
| less risky than installing software.
| cudgy wrote:
| True, but you have it backwards. Web app needs user to
| download and install a web browser first; desktop app
| does not need anything downloaded first. Both can be
| started by clicking a link.
| simonw wrote:
| I guess you mean downloading a browser version that
| supports the new origin filesystem feature? Every user
| has a web browser installed.
| cudgy wrote:
| Quite simple. I do not need to download a web browser at
| all to run the desktop app. Deployment of the desktop app
| is simpler and provides the same or even more
| functionality without adding a complex desktop browser
| and sandbox container to run the app. Why is this so hard
| to get across?
| mg wrote:
| Let's hope Firefox starts to support the File System Access API
| soon.
|
| For me, it is the last missing piece to make web apps as
| comfortable as native apps.
|
| I use a lot of web apps I fine-tuned exactly to my liking. I do
| all my writing in this web app for example:
| https://www.gibney.org/writer - I hit F11 and boom! I am in
| distraction free writing heaven :)
|
| If Firefox would support the File System Access API, it would be
| much more comfortable to load and save my writings.
| laboratorymice wrote:
| Genuinely curious, why do you use that web app over whichever
| bare-bones editor is available on your system?
| mg wrote:
| Because the web app is exactly the experience I want to have
| while writing:
|
| Nothing on my screen except two areas: The writing area and
| the margin around it.
|
| I want to be able to easily resize the writing area the way I
| can resize the textarea.
|
| I want to be able to easily change the font size via ctrl+
| and ctrl-.
| airstrike wrote:
| https://www.thebookdesigner.com/7-distraction-free-
| writing-e...
| 0x457 wrote:
| I get that with vim and suckless terminal... Why do you
| need a browser for that?
| mg wrote:
| I have never seen a terminal solution that allows to have
| the terminal in fullscreen mode and still have a way to
| have a user defined writing area which can be resized on
| the fly.
|
| I have experimented with various distraction free vim
| plugins but they were all fragile and cumbersome.
|
| If you have a solution, a screenshot would be very
| interesting.
| flamwenco wrote:
| I haven't maintained a Vim config in a few years now
| (more of an Emacs man now), but I do remember using Goyo
| in college. Looking back at it, I think it might scratch
| your itch as far as Vim plugins go, it even allows you to
| resize the area on the fly.
|
| https://github.com/junegunn/goyo.vim
| mg wrote:
| Goyo is one of the plugins I experienced as brittle and
| cumbersome. IIRC it creates the centered text area by
| creating 9 windows. The outter 8 to center the one in the
| middle, which is used to edit the text.
|
| In my attempts to use it, that created a bunch of issues.
| rapind wrote:
| tmux would be worth checking out. If you're happy with
| the browser though then that works too!
| shepherdjerred wrote:
| Not everybody wants to use a terminal
| [deleted]
| kennywinker wrote:
| does it have spellcheck? text-to-speech so you can have
| it read back to you? A browser window would have those
| things (on a mac, and I assume there's equivalents in
| windows)
| flamwenco wrote:
| Funny, Firefox on Mac sure doesn't integrate with the
| native text to speech, or offer it's own without plugins
| as far as I can tell. Gee it's almost like all browsers
| aren't exactly the same
|
| And yes, of course Vim has spellcheck.
| peoplefromibiza wrote:
| > Nothing on my screen except two areas: The writing area
| and the margin around it.
|
| what's wrong with the so called "zen mode" that has been
| available on every moderately popular editor for decades?
|
| See also JRR Martin still using WordStar 4.0 for DOS
|
| https://cdn.eteknix.com/wp-
| content/uploads/2014/05/37719_2_g...
| dmitriid wrote:
| > Let's hope FireFox starts to support the File System Access
| API soon.
|
| File System Access considered harmful, will not be implemented:
| https://mozilla.github.io/standards-positions/
|
| Note: in true Google fashion Chrome team implemented and
| released at least three different APIs all having something to
| do with files.
|
| Status of File System Access? "not a W3C Standard nor is it on
| the W3C Standards Track." https://wicg.github.io/file-system-
| access/ Shipped in Chrome, of course.
| lopkeny12ko wrote:
| What Mozilla/Firefox needs to understand is that end users do
| not care about standards purity/principles. If core
| functionality doesn't work in your browser, people simply
| aren't going to use it. This is why the web is increasingly
| broken on Firefox--nowadays, about 10% of the sites I visit
| are simply utterly broken or slow on Firefox but work fine on
| Chrome.
|
| Hell, even the $LARGE_CORPORATION I work at has all its
| developer tooling (including documentation, custom
| extensions, etc.) on Chrome. I've reported countless Firefox-
| specific issues, often due to missing API support, because no
| one internally tests on anything other than Chrome. You have
| to meet your users where they are.
| mftb wrote:
| These are the same sort of arguments that surrounded
| ActiveX and IE, back in the day. When $LARGE_CORPORATION is
| stuck on Chrome years after they would like to be (and
| other fallout), it will have been entirely predictable.
| Good on you for continuing to report issues you see to FF,
| but to your final point, if everyone is going to a bad
| place, no, you do not have to go.
| dmitriid wrote:
| > If core functionality doesn't work
|
| Somehow "everything that Chrome ships" is core
| functionality now
|
| > broken or slow on Firefox but work fine on Chrome.
|
| Somehow you blame Firefox for this. Even though "only works
| in IE" isn't even ancient history
| ilyt wrote:
| > What Mozilla/Firefox needs to understand is that end
| users do not care about standards purity/principles. If
| core functionality doesn't work in your browser, people
| simply aren't going to use it. This is why the web is
| increasingly broken on Firefox--nowadays, about 10% of the
| sites I visit are simply utterly broken or slow on Firefox
| but work fine on Chrome.
|
| It's not broken because they are fussy on standards.
|
| it's broken because their marketshare dropped so low
| companies that make money on developing web stuff are often
| not contractually obliged to even support it. Few of our
| clients have contracts where we're obliged to support any
| browser above 5% market share and FF is already below even
| that!
| fimdomeio wrote:
| But at the same time if it's not for purity/principles why
| would anyone choose Firefox over Chrome?
| KRAKRISMOTT wrote:
| Single issue user. Adblocking. Not because of some grand
| belief in the idea that Mozilla would be a better steward
| for the web like what many others here are advocating.
| vbezhenar wrote:
| I agree. I'm developing a system which uses webapp to
| interact with device over USB. It works in Chrome. It's sad
| for me to not support Firefox but the only way is to ask
| user to run some binary which will open websocket on
| localhost to interact with USB device. This is strictly
| worse from security PoV. This is strictly worse from
| usability PoV (user have to download and install manually
| additional component). This introduces update hurdles (I
| need to develop reliable updater). So much friction and
| issues just because of purity stand which hurts user in the
| end.
| dmitriid wrote:
| > I'm developing a system which uses webapp to interact
| with device over USB. It works in Chrome. It's sad for me
| to not support Firefox
|
| As a cautionary tale: the moment Firefox shipped support
| for a hardware API they immediately ran into sites
| abusing them. Of course, Chrome doesn't even hide them
| behind a user prompt.
| https://twitter.com/denschub/status/1582730985778556931
| (and comment)
|
| There are _reasons_ Firefox opposes these standards.
| Chrome is neither your friend nor does it care for the
| web. Why are you so willing to give a carte blanche to a
| power struggle (Chrome /Web vs. Android) inside a web
| advertising company is beyond me.
| magicalist wrote:
| You're looking for https://fs.spec.whatwg.org/
|
| https://mozilla.github.io/standards-positions/ is positive on
| it
|
| and implementation is well underway:
| https://bugzilla.mozilla.org/show_bug.cgi?id=1748667
| whimsicalism wrote:
| You're wrong for the API discussed in GP.
| dmitriid wrote:
| That's why I wrote this: "in true Google fashion Chrome
| team implemented and released at least three different APIs
| all having something to do with files." :)
|
| I didn't want to dig through all of them on mobile when I
| was responding.
|
| Both Safari and Mozilla are open (and implement/have
| implemented) to Origin Private File System. Because it
| gives web sites an access to a file system without a chance
| of escaping and damaging user data. Can't remember which
| spec introduced it, and it doesn't really matter at this
| point.
|
| For the more general part of File System Access Mozilla's
| and Safari's positions are: nope.
| https://github.com/mozilla/standards-positions/issues/154
|
| There was also Storage Foundation API but it was thankfully
| never shipped (it was like a third way of handling files or
| something),and also received negative responses:
| https://chromestatus.com/feature/5670244905385984
| sgbeal wrote:
| > https://mozilla.github.io/standards-positions/ is
| positive on it
|
| To hopefully add a bit of clarity: indeed, that page has
| OPFS flagged as positive under the label "File System."
| Further down the page is a "File System Access" entry which
| is flagged negative.
| tomayac wrote:
| Mozilla opposes the picker methods of the File System Access
| API. They are fully behind the Origin Private File System
| part, and actually have the SQLite demo from the article
| running in Firefox Nightly.
| geysersam wrote:
| > but Mozilla could be supportive of parts, provided this
| were segmented better.
|
| They seem quite positive to parts of the spec. Don't know if
| that's the parts relevant here or not.
| redwall_hp wrote:
| Yep, Chrome is the new IE. Only instead of ActiveX and small
| proprietary HTML additions and EEE, it's a full coup de tat
| on web standards and Embrace, Extend, Control.
| madeofpalk wrote:
| Isn't Safari the new IE?
| mrighele wrote:
| They are both the new IE but at different point in time
|
| Chrome is the young IE, the one disregarding the
| competition and imposing their own standards.
|
| Safari is the old IE, the one that is behind other
| browsers in terms of standard as but that you cannot
| ignore because it ha too big of a market share
| encryptluks2 wrote:
| So if people abandoned Chrome and Safari for Firefox,
| wouldn't Firefox be the new IE? I don't remember IE being
| open source and community-developed like Chromium which
| Chrome is based off of. I use Chromium on Linux and while
| there are a few things to complain about, it has been a
| pretty positive experience overall and worked better than
| Firefox.
| dmitriid wrote:
| Chromium is open source but is not community-developed.
| dmitriid wrote:
| A great article on this, Breaking the Web Forward https:/
| /www.quirksmode.org/blog/archives/2021/08/breaking_th...
| meitham wrote:
| They are one and the same thing now, WebKit!
| c0wb0yc0d3r wrote:
| I can't tell if you're trying to say both Chrome and
| Safari use the same rendering engine, but Chrome hasn't
| used WebKit for nearly a decade.
| meitham wrote:
| You're absolutely right! I wasn't aware of Blink!
| tete wrote:
| I think they are on phones mostly.
| toyg wrote:
| _> coup de tat_
|
| Coup d'etat: French, "stroke (coup) over/of the State
| (etat)".
|
| https://en.wikipedia.org/wiki/Coup_d%27%C3%A9tat
|
| /pedantry
| [deleted]
| smallerfish wrote:
| Very nice. I hope Firefox end up supporting OPFS.
|
| My use case is an entirely offline webapp that I'm building as a
| side project, partly just to see what is possible. I've tried
| indexeddb, pouchdb, absurd-sql, and various others, and have
| ended up back at localstorage, which is plain but low on edge
| cases.
|
| The other big pain point I am hacking around is cross device
| sync; I'm leaning towards simple chrome and firefox extensions
| that'll slurp data into your profile in order to sync, which'll
| work for cross computer sync for my needs. However, chrome on
| android doesn't support extensions, and there's no other good way
| that I've found to sync between a user's chrome profile and
| android, so that is still a hole in the story. Some kind of
| webrtc based thing is imaginable, but it's clunky.
| byhemechi wrote:
| i believe OPFS is supported on nightly? don't quote me on this
| yeah
| galleywest200 wrote:
| From what I can see they are very much writing code for this.
| This request for move() in OPFS was closed four months ago.
|
| https://bugzilla.mozilla.org/show_bug.cgi?id=1789116
| gfodor wrote:
| Ironically I was just about to drop in absurd-sql [1] to a
| project, which uses indexeddb to back SQLite. This seems better.
|
| [1] https://github.com/jlongster/absurd-sql
| Dork1234 wrote:
| Will either SQLite WASM or absurd-sql work on Firefox and
| Safari?
| tantaman wrote:
| If it helps, I detail some of the pros/cons of using the
| official SQLite build vs unofficial builds (wa-sqlite, which is
| similar to absured-sql) here:
|
| https://vlcn.io/docs/guide-persistence#persistence-options
| DylanSp wrote:
| This is definitely better, yeah. The blog post[1] on absurd-sql
| notes that it's a hacky solution that would be improved with
| file system access; it references the old Storage Foundation
| proposal [2], which has since evolved into the current File
| System Access API proposal(s).
|
| [1] https://jlongster.com/future-sql-web
|
| [2] https://developer.chrome.com/docs/web-platform/storage-
| found...
| samwillis wrote:
| This is a nice overview!
|
| Discussion from back when the SQLite project announced this last
| year: https://news.ycombinator.com/item?id=33374402
|
| There isn't an "official" NPM package or ES6 model yet, but I
| believe they are working on it. It's also very much designed for
| the browser currently, but again I believe they are intending to
| support server WASM environments too eventually.
| sgbeal wrote:
| > There isn't an "official" NPM package or ES6 model yet
|
| (The sqlite team's JS/WASM Guy here...)
|
| An ES6 module build was added shortly after the 3.40 release.
|
| NPM/node.js are nowhere on our radar. Our build is structured
| such that people who want to plug it in the resulting JS file
| to whatever their favorite toolchain is are welcomed to do so,
| but we have neither the ambition nor the bandwidth to support
| every build/bundling platform out there, especially ones none
| of us otherwise use.
|
| > It's also very much designed for the browser currently, but
| again I believe they are intending to support server WASM
| environments too eventually.
|
| The JS code is targeted solely at browsers and there are no
| plans on changing that in the foreseeable future.
|
| The WASM build itself, we are working to provide server-side
| support for. We have a branch which builds under wasi-sdk, but
| we cannot create a JS binding for that build until/unless we
| find some substitute for Emscripten's transparent translation
| of POSIX I/O APIs.
| [deleted]
| justsomeuser wrote:
| Being able to `npm install` this for a frontend app would be
| very useful.
| simonw wrote:
| I do think it would be worth SQLite having an "official"
| npm package - even though setting it up in the first place
| would require quite a bit of effort.
|
| The JavaScript world is culturally so dependent on npm now
| that adoption of SQLite in that world would massively
| increase given the availability of a good "official"
| package.
|
| And without an official one I imagine there will quickly
| emerge dozens of unofficial ones, many of which will end up
| poorly maintained in the future.
| tracker1 wrote:
| Especially with the number of "unofficial" packages doing
| this in a less than standard way.
| [deleted]
| lovasoa wrote:
| Hey, I am the maintainer of sql.js. I was in contact with them,
| and I do not think they are working on a npm module, but I
| offered them to work on the existing "official" sql.js module.
| samwillis wrote:
| Looking back at the thread, you are right. It's the ES Module
| that they do intend to eventually have.
| coliveira wrote:
| He knows he is right. It is you who need to thank him for
| pointing this out.
___________________________________________________________________
(page generated 2023-01-12 23:00 UTC)