https://gist.github.com/pesterhazy/4de96193af89a6dd5ce682ce2adff49a

Skip to content
 
[                    ]
All gists Back to GitHub Sign in Sign up
 
Sign in Sign up
{{ message }}

Instantly share code, notes, and snippets.

@pesterhazy

pesterhazy/indexeddb-problems.md

Last active Aug 12, 2021

  * Star 38
  * Fork 0

Star
Code Revisions 48 Stars 38
Embed
What would you like to do?
">Embed Embed this gist in your website.
Share Copy sharable link for this gist.
Clone via HTTPS Clone with Git or checkout with SVN using the
repository's web address.
 
Learn more about clone URLs
[<script src="https:/]
 
Download ZIP
The pain and aguish of using IndexedDB: problems, bugs and oddities
Raw
indexeddb-problems.md

This gist lists challenges you run into when building offline-first
applications based on IndexedDB, including open-source libraries like
Firebase, pouchdb and AWS amplify.

Note that some of the following issues affect only Safari. Out of the
major browsers, Chrome's IndexedDB implementation is the best.

 Backing file on disk (WAL file) keeps growing (Safari)

When this bug occurs, every time you use the indexeddb, the WAL file
grows. Garbage collection doesn't seem to be working, so after a
while, you end up with gigabytes of data.

 Random exceptions when working with a large number of indexeddb
databases (Safari)

Sometimes Safari gets into a funky state, where exceptions like
"Unable to open database file on disk" are thrown. Other exceptions
related to indexeddb methods are also possible.

When that happens, incredibly, hostname resolution starts failing.
This seems unrelated but occurs at the same time, for unknown
reasons.

It's unclear what is causing this issue but I've been observing this
problem mainly when there are many indexeddb databases for a given
origin. The databases don't need to be opened for this bug to occur.
You can fix the issue by deleting your local state.

You can work around this problem by only creating a small number of
indexeddb databases.

 State deleted after 7 days of inactivity (Safari)

After 7 days of inactivity, Safari deletes all browser storage,
including cookies, localstorage, websql and indexeddb. Other browsers
don't do this.

The feature, called Intelligent Tracking Prevention (ITP), is meant
to prevent advertisers from abusing indexeddb as a way to track
users. But it also has the consequence that purely in-browser apps
like excalidraw cannot treat browser storage as durable storage for
the user's data; they can only use it as a disposable cache.

The IsLoggedIn API might allow browsers to tell tracking from
legitimate use of browser storage in the future but is still in the
planning phase.

 IndexedDB operations hang without progress or error (Safari)

In our offline-first product, we've had reports from Safari users
that sometimes data isn't persisted to disk even though no exceptions
are printed to the console. The problem seems to be that in rare
circumstances, the promise chain stalls, with no progress or error -
the worst kind of failure mode, because changes are silently dropped.

At first we thought that the problem was in our code, but we
discovered that similar problems in Safari have been reported in the
Firebase SDK:

    A few times, I've seen Firestore in a state where changes stop
    getting written to the cloud, or even IndexDB. There are no
    errors in the console, and only a refresh seems to fix it. Upon
    refreshing the page, the user's changes are simply gone.

    Stepping through the code in this state, I can see that the write
    is enqueued in AsyncQueue, but never gets executed.

As far as we can tell, the issue seems to occur when the Safari tab
is sent to or comes back from the background. We know that Safari
throttles tab activity aggressively to safe energy. But of course,
this shouldn't lead to indexeddb operations being delayed
indefinitely.

 Is IndexedDB ACID compliant?

IndexedDB has the concepts of transactions. Does this mean that
IndexedDB is ACID compliant? Let's look at atomicity (A) and
isolation (I) in particular:

  * IndexedDB provides atomic transactions. When multiple writes are
    wrapped in a transaction, IndexedDB guarantees that either all
    writes will go through or none of them will. This avoids
    inconsistencies due to writes interrupted mid-way.

  * IndexedDB does not provide transaction isolation. As far as I
    know, concurrent transactions (in multiple tabs or even in a
    single tab) are never rolled back because they touch the same
    object stores. The only exception, as far as I can tell, is
    exceptions thrown when a primary key constraint is violated. This
    can be used to achieve some level of transaction isolation.
    However, the guarantees are nowhere near as extensive as those
    provided by SQL databases.

 Confusing transaction auto-commit semantics

There are semantic differences between how Safari interprets the
IndexedDB spec compared to Chrome and Firefox. In Safari, IndexedDB
transactions auto-close more aggressively when nothing is being done
to a transaction in a stackframe. This problem is apparent when using
indexeddb with promises, because the use of Promise.resolve().then(
() => ... ) can lead to the transaction being closed prematurely,
causing an exception - especially in Safari and Firefox.

The idb promise wrapper contains an warning but transaction
auto-commit semantics in combination with promises are still
difficult to get right. For example, Firebase went so far as to
implement their own version of Promise for the purpose of working
around these issues.

Safari is most finicky, though operating within the spec. In any case
the behavior differences compared to other implementations are
confusing. In fact it's fair to say that it's hard to understand when
transactions are supposed to auto-close, either as a application
developer or browser implementer (see this bug report)

In general, asynchronous operations or promises mixed with IndexedDB
transactions are fraught with peril. Consider the case of deleting a
number of keys from an index. This will require two steps:

 1. Call .getAllKeys() on the index
 2. Iterate over all the keys and call .delete() on the ObjectStore
    for each key

It's tempting to do all of this in a single transaction - after all,
they two parts semantically belong together. However, the frist step
is necessarily asynchronous. As a result, it's possible that once we
get to step 2, the transaction has already auto-committed.

Unfortunately, this will result in a latent bug that only manifests
itself in rare circumstances, depending on timing. For example,
Firefox will sometimes throw an exception with the message
Transaction is already committing or done, but not in a reproducible
manner.

The solution is to perform all write operations (.put, .delete, etc)
synchronously in a single stackframe, the same stackframe that
created the transaction. In other words, it's not safe to perform
steps 1 and 2 in a single transaction. Instead, each step should be a
done in its own transaction (the first one being readonly, the second
readwrite).

 No support for IDBTransaction.commit (Safari)

Safari is the only major browser that doesn't support
IDBTransaction.commit()

 No locking primitives (Safari, Firefox)

The Web Platform lacks a standardized way - or any proper way, really
- to lock a resource across multiple tabs of the same browser.
Multiple tabs of an app using IndexedDB will invariably write to the
same IndexedDb database. Without cross-tab locking, database
corruption is hard to avoid.

Attempts have been made to builds locks on top of existing APIs like
localStorage and postMessage, but clearly locking requires a new
primitive and cannot be polyfilled. Proposed by Google, the Web Locks
API adds the capabilities desparately needed to implement safe
concurrent use of IndexedDB. However, while Chrome shipped Web Locks
in 2018, Safari and Firefox haven't implemented the API yet. Similar
things can be said about the new Atomics API.

 Private Browsing Mode (Firefox)

Firefox is the only major browser without support for IndexedDB in
Private Browsing mode. In a Private Browsing window (or if Private
Browsing is enabled as a global setting), Firefox throws with a
cryptic error when you call indexeddb.open:

A mutation operation was attempted on a database that did not allow mutations

Because there is no API to tell if Private Browsing is active, the
only workaround is to try to open a test database and to regard the
API as unavailable if this fails.

 Quotas are confusing

Browser vendors enforce qutoas for all browser storage, including
IndexedDB. When the quota is exceeded, writes will fail wih
QuotaExceededError or similar exceptions.

However, how quotas work is different from browser to browser. One
surprising finding is that in Firefox, quotas seem to apply, not per
subdomain, but to all subdomains belonging to a certain site name
(more precisely, an eTLD+1) taken together. So for the purposes of
quota, a.example.com and b.examples.com will be counted together. As
a result, clearing storage data for the current subdomain may not
prevent quota exceptions because the quota may be used up by other
subdomains.

 Web SQL - the database that could have been

Before IndexedDB there was Web SQL, a thin wrapper around the
legendary SQLite embedded database. Web SQL is more powerful (a
proper superset of IndexedDB) and arguably better designed than
IndexedDB. So why did browser vendors end up favoring the inferior
IndexedDB standard?

As it turns out, Mozilla objected to the Web SQL interface as having
no alternative implementation available. So IndexedDB, an API with
less surface, was chosen instead. Ironically, IndexedDB is internally
implemented based on SQLite in Firefox (Chrome uses the simpler
LevelDB instead).

In a similar vein, wiki service Notion discusses problems with their
IndexedDb backend:

    Before SQLite, we relied on IndexedDB for client-side storage.
    But we encountered storage quotas, a number of bugs, and
    performance concerns on Windows machines in particular, which
    meant IndexedDB wouldn't scale with Notion's growing user base.

Their solution? To move to an SQLite db bundled with their Electron
app running outside the browser context.

 Bugs, bugs, bugs (Safari)

The Webkit team keeps shipping critical storage-related bugs into
production, again and again. In May 2021, Safari 14.1.1 was released
with a bug where IndexedDB databases would fail to open() when the
browser is loaded for the first time (discussion). This came a month
after the Safari 14.1 release, which fundamentally broke localStorage
(discussion).

@Pacerier

This comment has been minimized.

Sign in to view
Copy link Quote reply

@Pacerier Pacerier commented May 4, 2021

    IndexedDB is internally implemented based on SQLite...

Missing the point as that's an implementation detail.

    In a similar vein...

Missing the point again. Electron is not part of the Web.

@dalmaer

This comment has been minimized.

Sign in to view
Copy link Quote reply

@dalmaer dalmaer commented Jun 17, 2021

Good stuff.

There is also the quota issue where if you run the web app from the
Home Screen on iOS you can never send the quota prompt to the user.
So, your most loyal users who add to Home Screen can never get more
quota. Ugh.

@dalmaer

This comment has been minimized.

Sign in to view
Copy link Quote reply

@dalmaer dalmaer commented Jun 17, 2021

And there's:

https://twitter.com/feross/status/1404568122158313474?s=21

Sign up for free to join this conversation on GitHub. Already have an
account? Sign in to comment

  * (c) 2021 GitHub, Inc.
  * Terms
  * Privacy
  * Security
  * Status
  * Docs

 

  * Contact GitHub
  * Pricing
  * API
  * Training
  * Blog
  * About

You can't perform that action at this time.
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session.