[HN Gopher] DuckDB is probably the most important geospatial sof...
___________________________________________________________________
DuckDB is probably the most important geospatial software of the
last decade
Author : dbreunig
Score : 92 points
Date : 2025-05-03 19:30 UTC (3 hours ago)
(HTM) web link (www.dbreunig.com)
(TXT) w3m dump (www.dbreunig.com)
| fithisux wrote:
| I agree
| jeffbee wrote:
| Ehh I tried to do some spatial stuff but there just wasn't enough
| there, or I could not figure out how to use it. Loading spatial
| information into ipython and fiddling with it is well-traveled
| and it doesn't seem to me that SQL is an inherently lower hurdle
| for the user.
| wodenokoto wrote:
| I'm not sure I agree that "install geospatial" is a game changer
| in simplicity compared to "pip install geopandas".
|
| They are both one line.
| maxxen wrote:
| I think a big part is that duckdbs spatial extension doesnt
| have any transitive dependencies (except libc). It statically
| packages the standard suite of foss gis tools (including a
| whole database of coordinate systems) for multiple platforms
| (including WASM) and provides a unified SQL interface to it
| all.
|
| (Disclaimer, I work on duckdb-spatial @duckdblabs)
| WD-42 wrote:
| Is it that much simpler than 'load extension postgis'? I know
| geos and gdal have always kinda been a pain, but I feel like
| docker has abstracted it all away anyway. 'docker pull postgis'
| is pretty easy, granted I'm not familiar with what else duckdb
| offers.
| dbreunig wrote:
| Yes. The difference between provisioning a server and running
| 'install spatial' in a CLI is night and day.
|
| Docker _has_ been a big improvement (when I was first learning
| PostGIS, the amount of time I had to hunt for proj directories
| or compile software just to install the plugin was a major
| hurdle), but it 's many steps away from:
|
| ``` $ duckdb D install spatial; ```
| frainfreeze wrote:
| I mean I like duckdb but this feels like you're pushing for
| it. On my system postgis comes from apt install, and it's one
| command to activate the "plugin". Is the night and day part
| not having to run random sh script from the internet to
| install software on my system?
| tomnipotent wrote:
| DuckDB doesn't require a running server. I run duckdb in a
| terminal, query 10,000 CSV or parquet files and run SQL on
| them while joining to data hosted in sqlite, a separate
| duckdb file using its native format, or even Postgres.
| frainfreeze wrote:
| It is not simpler. I use it with testcontainers in the
| notebooks usually https://testcontainers-
| python.readthedocs.io/en/latest/
| larsiusprime wrote:
| "import geopandas" also exists and has for some time. Snark
| aside, WHAT is special about duckDB? I wish the author had
| actually shown some practical examples so I could understand
| their claims better.
| joshvm wrote:
| I haven't used duckDB but the real comparison is presumably
| postgis? Which is also absent from the discussion, but I think
| what the author alludes to.
|
| I have no major qualm with pandas and geopandas. However I use
| it when it's the only practical solution, not because I enjoy
| using it as a library. It sounds like pandas (or similar) vs a
| database?
| dbreunig wrote:
| Author here: what's special is that you can go from 0 to
| spatial data incredibly quickly, in the data generalist tool
| you're already using. It makes the audience of people working
| with geospatial data much bigger.
|
| (Geopandas is great, too.)
| tmpz22 wrote:
| I've been researching DuckDB - while it has many technical
| merits I think the main argument will be ease of use. It has a
| lot of the operational advantages of sqlite paired with strong
| extensibility and good succinct documentation.
|
| Folks who have been doing DevOps work are exasperated with
| crummy SaaS vendors or antiquated OSS options that have a high
| setup cost. DuckDB is just a mature project that offers an
| alternative, hence an easy fan favorite among hobbyists (I
| imagine at scale the opportunity costs change and it becomes
| less attractive).
| maxxen wrote:
| I replied to another comment, but I think a big part is that
| duckdbs spatial extension provides a SQL interface to a whole
| suite of standard foss gis packages by statically bundling
| everything (including inlining the default PROJ database of
| coordinate projection systems into the binary) and providing it
| for multiple platforms (including WASM). I.E there are no
| transitive dependencies except libc.
|
| Yes, DuckDB does a whole lot more, vectorized larger-than-
| memory execution, columnar compressed storage and a ecosystem
| of other extensions that make it more than the sum of its
| parts. But while Ive been working hard on making the spatial
| extension more performant and more broadly useful (I designdd a
| new geometry engine this year, and spatial join optimization
| just got merged on the dev-branch), the fact that you can e.g.
| convert too and from a myriad of different geospatial formats
| by utilizing GDAL, transforming through SQL, or pulling down
| the latest overture dump without having the whole workflow
| break just cause you updated QGIS has probably been the main
| killer feature for a lot of the early adopters.
|
| (Discmaimer, I work on duckdb-spatial @ duckdblabs)
| timschmidt wrote:
| I'm not the OP, but thank you for such a detailed answer. The
| integration and reduced barriers to entry you mention mirror
| my own experiences with tooling in another area, and your
| explanation made parallels clear.
| larsiusprime wrote:
| This is an excellent reply and what I wish the article had
| been, thanks!
| getnormality wrote:
| Everything is special about DuckDB. Pandas is way, way behind
| the state of the art in tabular data analysis.
| tsss wrote:
| For one it doesn't have the god awful pandas API.
| jjtheblunt wrote:
| duckdb has parquet support and can operate, in SQL syntax, on
| enormous 'tables' spread across huge collections of parquet
| files as if one virtual file. i believe the underlying
| implication is opportunities to leverage vector instructions on
| parquet. it's very "handy".
| twelvechairs wrote:
| DuckDB is a great thing for geospatial but most important of the
| past decade? There's so many tools in different categories it
| wouldnt come near top for me. Some might be QGIS, postGIS (still
| the standard), ArcGIS online (still the standard), JS mapping
| tools like mapbox (i prefer deckgl), new data types like COG,
| geopackage and geoparquet, photogrammetry tools, 3d tiles, core
| libraries like gdal and now pdal, shapely, etc.
| walleeee wrote:
| also xarray and friends
| dbreunig wrote:
| Most of those tools came out circa ~2000.
|
| Yeah, I feel old.
| wenc wrote:
| I'm a big fan of DuckDB and I do geospatial analysis, mostly
| around partitioning geographies (into Uber H3 hexagons),
| calculating Haversine distances, calculating areas of geometries,
| figuring out which geometry a point falls in, etc. Many of these
| features have existed in some form or other in geopandas or
| postgis, so DuckDB's spatial extensions bring nothing new.
|
| But what DuckDB as an engine does is it lets me work directly on
| parquet/geoparquet files at scale (vectorized and parallelized)
| on my local desktop. It beats geopandas in that respect. It's a
| quality of life improvement to say the least.
|
| DuckDB also has an extension architecture that admits more exotic
| geospatial features like Hilbert curves, Uber H3 support.
|
| https://duckdb.org/docs/stable/extensions/spatial/functions....
|
| https://duckdb.org/community_extensions/extensions/h3.html
| fidotron wrote:
| Honestly, I think it's actually https://www.uber.com/en-
| CA/blog/h3/
| bingaweek wrote:
| We need a "come on" clause for these absurd headlines. Come on.
| badmonster wrote:
| How might embedding spatial capabilities directly into general-
| purpose data tools like DuckDB reshape who participates in
| geospatial analysis--and what kinds of problems they choose to
| solve?
| jparishy wrote:
| I work on geospatial apps and the software I think I am most
| excited about is https://felt.com/. I want to see them expand
| their tooling such that maps and data source
| authentication/authorization was controllable by the developer,
| to enable tenant isolation with proprietary data access. They
| could really disrupt how geospatial tech gets integrated into
| consumer apps.
|
| This article doesn't acknowledge how niche this stuff is and it's
| a lot of training to get people to up to speed on coordinate
| systems, projections, transformations, etc. I would replace a lot
| of my custom built mapping tools with Felt if it were possible,
| so I could focus on our core geospatial processes and not the
| code to display and play with it in the browser, which is almost
| as big if not bigger in terms of LOC to maintain.
|
| As mentioned by another commenter, this DuckDB DX as described is
| basically the same as PostGIS too.
| dbreunig wrote:
| Author here: the beauty of DuckDB spatial is that the
| projections and CRS options are hidden until you need them. For
| 90% of geospatial data usage people don't and shouldn't need to
| know about projections or CRS.
|
| Yes, there are so many great tools to handle the complexity for
| the capital-G Geospatial work.
|
| I love Felt too! Sam and team have built a great platform. But
| lots of times a map isn't needed; an analyst just needs it as a
| column.
|
| PostGIS is also excellent! But having to start up a database
| server to work with data doesn't lend itself to casual usage.
|
| The beauty of DuckDB is that it's there in a moment and in
| reach for data generalists.
| jparishy wrote:
| I think we're mostly making the same point about complexity,
| ya.
|
| To me, I think it's mostly a frontend problem stopping the
| spread of mapping in consumer apps. Backend geo is easy tbh.
| There is so much good, free tooling. Mapping frontend is hell
| and there is no good off the shelf solution I've seen. Some
| too low level, some too high level. I think we need a GIS-
| lite that is embeddable to hide the complexity and let app
| developers focus on their value add, and not paying the tax
| of having frontend developers fix endless issues with maps
| they don't understand.
|
| edit: to clarify, I think there's a relationship between
| getting mapping valued by leadership such that the geo work
| can be even be done by analysts, and having more mapping
| tools exist in frontend apps such that those leaders see them
| and understand why geo matters. it needs to be more than just
| markers on the map, with broad exposure. hence my focus on
| frontend web. sorry if that felt disjointed
| patja wrote:
| SQL Server has geospatial capabilities without any extensions or
| add-ons. I've been happily using geospatial datatypes on the free
| Express version for years, probably well over a decade.
| cyanydeez wrote:
| No. QGIS is.
|
| Good god.
| dbreunig wrote:
| Author here.
|
| QGIS is amazing. It's really great. It also came out in 2002,
| so I think the headline is safe.
___________________________________________________________________
(page generated 2025-05-03 23:00 UTC)