[HN Gopher] Use DuckDB-WASM to query TB of data in browser
___________________________________________________________________
Use DuckDB-WASM to query TB of data in browser
Author : mlissner
Score : 97 points
Date : 2025-10-31 17:37 UTC (5 hours ago)
(HTM) web link (lil.law.harvard.edu)
(TXT) w3m dump (lil.law.harvard.edu)
| mlissner wrote:
| OK, this is really neat: - S3 is really cheap static storage for
| files. - DuckDB is a database that uses S3 for its storage. -
| WASM lets you run binary (non-JS) code in your browser. - DuckDB-
| Wasm allows you to run a database in your browser.
|
| Put all of that together, and you get a website that queries S3
| with no backend at all. Amazing.
| timeflex wrote:
| S3 might be relatively cheap for storing files, but with
| bandwidth you could easily be paying $230/mo. If you make it
| public facing & want to try to use their cloud reporting,
| metrics, etc. to prevent people for running up your bandwidth,
| your "really cheap" static hosting could easily cost you more
| than $500/mo.
| theultdev wrote:
| R2 is S3 compatible with no egress fees.
|
| Cloudflare actually has built in iceberg support for R2
| buckets. It's quite nice.
|
| Combine that with their pipelines it's a simple http request
| to ingest, then just point duckdb to the iceberg enabled R2
| bucket to analyze.
| greatNespresso wrote:
| Was about to jump in to say the same thing. R2 is a much
| cheaper alternative to S3 that just works and I have used
| it with DuckDB, works smoothly
| 7952 wrote:
| I think this approach makes sense for services with a small
| number of users relative to the data they are searching. That
| just isn't a good fit for a lot of hosted services. Think how
| much that TB's of data would cost on Algolia or similar
| services.
|
| You have to store the data somehow anyway, and you have to
| retrieve some of it to service a query. If egress costs too
| much you could always change later to put the browser code on
| a server. Also it would presumably be possible to quantify
| the trade-off between processing the data client side and on
| the server.
| rubenvanwyk wrote:
| Or use R2 instead. It's even easier.
| thadt wrote:
| S3 is doing quite a lot of sophisticated lifting to qualify as
| _no backend at all._
|
| But yeah - this is pretty neat. Easily seems like the future of
| static datasets should wind up in something like this. Just
| data, with some well chosen indices.
| theultdev wrote:
| Still qualifies imo. Everything is static and on a CDN.
|
| Lack of server/dynamic code qualifies as no backend.
| amazingamazing wrote:
| Neat. Can you use duckdb backed on another store like rocksdb or
| something? Also, I wonder how one stops ddos. Put the whole thing
| behind Cloudflare?
| wewewedxfgdf wrote:
| I tried DuckDB - liked it a lot - was ready to go further.
|
| But found it to be a real hassle to help it understand the right
| number of threads and the amount of memory to use.
|
| This led to lots of crashes. If you look at the projects github
| issues you will see many OOM out of memory errors.
|
| And then there was some indexed bug that crashed seemingly
| unrelated to memory.
|
| Life is too short for crashy database software so I reluctantly
| dropped it. I was disappointed because it was exactly what I was
| looking for.
| lalitmaganti wrote:
| +1 this was my experience trying it out as well. I find that
| for getting started and for simple usecases it works amazing.
| But I have quite a lot of concerns about how it scales to more
| complex and esoteric workloads.
|
| Non-deterministic OOMs especially are some of the worst things
| in the sort of tools I'd want to use DuckDB in and as you say,
| I found it to be more common than I would like.
| tuhgdetzhh wrote:
| I can recommend earlyoom (https://github.com/rfjakob/earlyoom).
| Instead of freezing or crashing your system this tool kills the
| memory eating process just in time (in this case duckdb). This
| allows you repeat with smaller chunks of the dataset, until it
| fits into your mem.
| wewewedxfgdf wrote:
| Yeah memory and thread management is the job of the
| application, not me.
| QuantumNomad_ wrote:
| When I there is a specific program I want to run with a limit
| on how much memory it is allowed to allocate, I have found
| systemd-run to work well.
|
| It uses cgroups to enforce resource limits.
|
| For example, there's a program I wrote myself which I run on
| one of my Raspberry Pi. I had a problem where my program
| would on rare occasions use up too much memory and I wouldn't
| even be able to ssh into the Raspberry Pi.
|
| I run it like this: systemd-run --scope -p
| MemoryMax=5G --user env FOOBAR=baz ./target/release/myprog
|
| The only difficulty I had was that I struggled to find the
| right name to use in the MemoryMax=... part because they've
| changed the name of it around between versions so different
| Linux systems may or may not use the same name for the limit.
|
| In order to figure out if I had the right name for it, I
| tested different names for it with a super small limit that I
| knew was less than the program needs even in normal
| conditions. And when I found the right name, the program
| would as expected be killed right off the bat and so then I
| could set the limit to 5G (five gigabytes) and be confident
| that if it exceeds that then it will be killed instead of
| making my Raspberry Pi impossible to ssh into again.
| thenaturalist wrote:
| This looks amazing!
|
| Have you used this in conjunction with DuckDB?
| mritchie712 wrote:
| what did you use instead? if you hit OOM with the dataset in
| duckdb, I'd think you'd hit the OOM with most other things on
| the same machine.
| wewewedxfgdf wrote:
| The software should manage its own memory not require the
| developer to set specific memory thresholds. Sure, a good
| thing to be able to say "use no more than X RAM".
| thenaturalist wrote:
| How long ago was this, or can you share more context about data
| and mem size you experienced this with?
|
| DuckDB has introduced spilling to disk and some other tweaks
| since a good year now: https://duckdb.org/2024/07/09/memory-
| management
| wewewedxfgdf wrote:
| 3 days ago.
|
| The final straw was an index which generated fine on MacOS
| and failed on Linux - exact same code.
|
| Machine had plenty of RAM.
|
| The thing is, it is really the responsibility of the
| application to regulate its behavior based on available
| memory. Crashing out just should not be an option but that's
| the way DuckDB is built.
| jdnier wrote:
| Yesterday there was a somewhat similar DuckDB post, "Frozen
| DuckLakes for Multi-User, Serverless Data Access".
| https://news.ycombinator.com/item?id=45702831
| 85392_school wrote:
| This also reminded me of an approach using SQLite:
| https://news.ycombinator.com/item?id=45748186
| SteveMoody73 wrote:
| My initial thought is why query 1TB of data in a browser, maybe
| I'm the wrong target audience for this but it seems that it's
| pushing that everything has to be in a browser rather than using
| appropriate tools
| cyanydeez wrote:
| Browsers are now the write-once works everywhere target. Where
| java failed, many hope browsers succeed. WASM is definitely a
| key to that, particularly because it can be output by tools
| like rust, so they can also be the appropriate tools.
| majormajor wrote:
| Why pay for RAM for servers when you can let your users deal
| with it? ;)
|
| (Does not seem like a realistic scenario to me for many uses,
| for RAM among other resource reasons.)
| some_guy_nobel wrote:
| The one word answer is cost.
|
| But, if you'd like to instead read the article, you'll see that
| they qualify the reasoning in the first section of the article,
| titled, "Rethinking the Old Trade-Off: Cost, Complexity, and
| Access".
___________________________________________________________________
(page generated 2025-10-31 23:00 UTC)