[HN Gopher] Show HN: Flatmap - a new tool to make vector tiles f...
___________________________________________________________________
Show HN: Flatmap - a new tool to make vector tiles from
OpenStreetMap data fast
Author : zylepe
Score : 102 points
Date : 2021-10-25 11:52 UTC (11 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| lalaland1125 wrote:
| Nice tool OP, but I think we as a community should try to avoid
| creating tools that use already common names.
|
| Googling "flatmap" gives a bunch of bad results. There is a value
| in having unique names.
| stevage wrote:
| Looks great! Potentially a game changer for projects that need a
| basemap for a smallish area (like one country), get a fair bit of
| traffic, but don't have budget for ongoing tile subscriptions to
| Mapbox or MapTiler.
| ryantgtg wrote:
| We self-host OpenMapTiles, but are super lazy about replacing
| them with updated tiles (and we don't love the watermark). I'm
| really glad that there are options for generating them ourselves!
| Starred, and will give it a try soon.
| tony_cannistra wrote:
| This is really sweet, awesome work. I really love onthegomap too.
|
| I maintain an openmaptiles pipeline, and we've got our planet
| runtimes down to about ~4 days on AWS.
|
| Would be excited to see if we could use this tool, but we really
| heavily depend on PostGIS data manipulation to ensure that the
| data we want ends up in our output tiles. How difficult do you
| think migrating this SQL to your tool would be? It seems like you
| can support modifying the data fairly heavily, both before and
| after MBtile creation.
| zylepe wrote:
| Flatmap calls into user-defined profiles in 2 places: first
| when processing each input element to map it to a vector tile
| feature, then a second time right before emitting all vector
| features in a layer.
|
| That second call lets you manipulate vector features on each
| tile using JTS geometry utilities (i.e. merge nearby polygons
| or lines with the same tags). PostGIS uses GEOS which is ported
| from JTS so you have access to pretty much the same geometry
| utilities - often with the same name.
|
| It's definitely not as flexible as a PostGIS based solution -
| especially if you join faraway features that don't appear on
| the same tile, but in practice it was enough to port the entire
| OpenMapTiles schema.
|
| See the basemap layers package:
| https://github.com/onthegomap/flatmap/tree/main/flatmap-base...
| for example see the landcover layer
| https://github.com/onthegomap/flatmap/blob/main/flatmap-base...
| which was ported from the SQL contained in
| https://github.com/openmaptiles/openmaptiles/tree/master/lay...
|
| Also, this OpenMapTiles PR might help improve your existing
| pipeline after it gets merged:
| https://github.com/openmaptiles/openmaptiles-tools/pull/383
|
| Edit: formatting
| tantalor wrote:
| Clever name... but not exactly unique.
| reichardt wrote:
| Fantastic work! Are the vector tiles compatible with OpenMapTiles
| styles?
| zylepe wrote:
| Thanks! The default "basemap" style is ported from OpenMapTiles
| v3.12.2.
| reichardt wrote:
| That's great to hear. I'd love to create some vector tiles
| with building and building height data on zoom levels down to
| 11. Is this the correct place to perform this change? https:/
| /github.com/onthegomap/flatmap/blob/e402a3333bc5127e5...
|
| Or could I even achieve this via the config file?
| https://github.com/onthegomap/flatmap/blob/main/config-
| examp...
| RicoElectrico wrote:
| Nice :)
|
| There's an ongoing effort to research vector tile solutions for
| openstreetmap.org. I wonder how your software stack compares to
| tilemaker, which has seen some optimization lately.
|
| https://github.com/openstreetmap/operations/issues/565
| zylepe wrote:
| Thanks! Tilemaker is a great tool and probably the closest
| alternative (no intermediate db). The most recent benchmarks I
| saw though were around a day for the planet, so roughly 10x
| longer than flatmap. If
|
| Edit: benchmark link
| https://github.com/systemed/tilemaker/issues/315#issue-99432...
| zylepe wrote:
| Creator here. I maintain https://onthegomap.com as a side
| project. I originally used Google Maps API but after their price
| hike switched to OpenStreetMap data. I self-host GraphHopper for
| routing and tried to self-host OpenMapTiles but gave up and used
| Stadia Maps since it would have taken over 100 days [1] to
| generate a map of the world myself. Stadia Maps has been great
| and I highly recommend them, but the problem continued to nag me,
| so earlier this year I started prototyping a new way to generate
| vector tiles faster. The idea worked and ended up being able to
| generate a ~100GB planet.mbtiles file in as little as 59 minutes
| on a single c5ad.16xlarge instance with 64 cpus and 128GB RAM
| (although smaller machines appear to be more economical).
|
| I'm open-sourcing Flatmap today. Let me know what you think, or
| if you have any suggestions!
|
| [1]
| https://github.com/openmaptiles/openmaptiles/issues/654#issu...
| bmitch2112 wrote:
| I use onthegomap almost daily. Such a great tool. Thank you for
| it.
| zylepe wrote:
| Thanks! Glad you find it useful!
| epaulson wrote:
| A bit off-topic, but do people who host their own map tiles worry
| that other apps will use/abuse their tileserver? I would worry
| that I'd come in one Monday morning and discover that I've got a
| $20K bandwidth bill because someone else got popular.
| tschesnok wrote:
| This is cool. I will dig in deeper and take a look. I'm working
| on a spherical display.. so maps with fast zoom will be a must.
| jmnicolas wrote:
| > at least 1.5x as much free RAM as the input .osm.pbf file size
|
| Ouch! Is there any plan to reduce the amount of ram or it's
| impossible?
|
| I'm currently building a tile server with Europe's map, the PBF
| is 28GB, I have 32GB ram.
| Symbiote wrote:
| The benchmark says the whole world can be done on a "DO 16cpu
| 128GB" machine in 3 hours, so about $3 from Digital Ocean. Call
| it $5 to allow some time for transferring data.
|
| Just rent the VM.
| willcodeforfoo wrote:
| Thanks so much for this tool! Looks like one could generate a
| planet with less than $5 on Digital Ocean.
| zylepe wrote:
| I generated the planet a few times in testing:
|
| with z13 building merge (extra 14 CPU hours of processing):
| - $4.68 on Digital Ocean 128GB 16 CPU instance (4 hours @
| $1.17/hr)
|
| without z13 building merge: - $2.75 on AWS EC2
| c5ad.16xlarge instance (1 hour @ $2.75/hr) - $2.88 on
| Linode 128GB 50 dedicated CPU instance (2 hours @ $1.44/hr)
|
| Unfortunately it costs $10 in egress bandwidth fees to get the
| result out of AWS ;-)
|
| Edit: formatting
___________________________________________________________________
(page generated 2021-10-25 23:01 UTC)