[HN Gopher] Show HN: Simple Mbtiles Server - Self-host the entir...
___________________________________________________________________
Show HN: Simple Mbtiles Server - Self-host the entire planet of
OpenStreetMaps
Author : markuman123
Score : 193 points
Date : 2024-08-07 13:07 UTC (5 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| archerx wrote:
| I wish the readme explained what this project actually was
| instead of the incredibly vague and unhelpful "self host the
| entire planet".
| basemi wrote:
| Found little more info in homepage
| https://github.com/markuman/sms
| fodkodrasz wrote:
| Still barely more detailed.
|
| The problem with the whole OSM ecosystem is that it expects a
| lot of domain knowledge of technologies, formats, tools,
| their relations and features, limitations, that is rarely
| explained well, or explanaitions are often terribly outdated.
| Feels much like the JavaScript Frontend frameworks for an
| outsider, it is a lot of effort to get to an overview and
| understanding, despite the concepts are not new.
| davidy123 wrote:
| Usually this is a sign of a project that isn't ready for a
| lot of attention. It's very brave and a little bit stupid
| to offer Open Source to the world (especially without a
| sustainability plan), if you put a lot of effort into
| 'marketing' at an early stage you've probably just
| overpromised to people not focused on that domain, won't be
| able to deliver, and will burn out, and it will become yet
| another abandoned project.
|
| Perhaps in a few years, as the project picks up uses and
| contributors, it will get its own website and support
| network and be easy to understand and use by "anyone."
| bspammer wrote:
| I can't tell if your comment was meant to be ironic but
| OSM has been around for 20 years, and its data is used by
| billions of people.
| Guvante wrote:
| I don't think this project is looking for a lot of
| attention.
|
| It is more like a sample of cool stuff you can do.
| PontifexMinimus wrote:
| > incredibly vague and unhelpful
|
| As is very common with projects on Github. They should have a
| paragraph at the start explaining simply what they do.
| MrGilbert wrote:
| And a screenshot for every project with a UI. I'd love to see
| that more often.
| totaldude87 wrote:
| Copilot could easily scan and auto generate the read me.
| toolz wrote:
| They 'could' have a paragraph at the beginning to help
| potential users understand the purpose of the project.
|
| It might seem inconsequential, but maintainers often deal
| with a lot of angry entitled users and it could be helpful to
| stay away from language that implies that the people giving
| away their code for free (and for fun) have any obligation to
| provide anyone anything.
| squarefoot wrote:
| True on principle, but if an explanation or a screenshot
| that costs 5 hours to make would prevent replying or even
| just reading support requests for 6 hours, I would call it
| a victory.
| Aachen wrote:
| Anyone looking for this will already be in map/OpenStreetMap
| context. That people originating from StumbleUpon or a niche
| little place called ycombinator news are confused is not too
| surprising tbh: that very likely wasn't the audience they had
| in mind
| input_sh wrote:
| Considering the title starts with "Show HN" and that the HN
| username is very closely tied to the GitHub username, I'm
| gonna go out on a limb here by saying HN is indeed the target
| audience.
| EGreg wrote:
| That's true, I was super confused what "self-host" entailed.
| They should go into detail explaining that. Leaflet?
|
| Because I understood about "the whole planet" in the context of
| OSM.
| 808essa wrote:
| explanation of what this is would be nice
| orblivion wrote:
| OpenStreetMap is a map ecosystem based on an open data set.
| There are numerous redundant ways of achieving the same thing
| with this data, whether it's hosting the "tiles" (the thing you
| see on the map), search, UI, location bookmarking, navigation,
| etc. This is great for flexibility, but it's a difficult thing
| to dive into and understand.
|
| With that said, the "tile hosting" part is notoriously
| difficult to actually self-host. It's a lot of data, it
| traditionally took a lot of time to transform the raw data into
| the png files representing the tiles. Many if not most web apps
| actually call out to openstreetmap.org or similar to do the
| tiles for them. That's not fully self-hosting. But more
| recently, there have been new ways to do tiles, namely with
| vector formats instead of pngs. They take up less space,
| they're easier to share around.
|
| So this submission appears to be a simple few lines to get your
| fully self-hosted tile server of the whole planet up and
| running. You should then be able to use this in whatever web
| app you're making to make it fully self hosted (though if you
| wanted self-hosted search, that's another hard one...).
| elashri wrote:
| Entire planet maps/topology I guess. But my mind had much much
| more ambitious first thought.
| fareesh wrote:
| Looks like every single "tile" image from OpenStreetMaps (OSM) a
| service that hosts them so that you can pass the same OSM
| parameters to the URL and get maps
| hakube wrote:
| It would be great if it's an actual planet with its own
| ecosystem. This is just a tile map
| dsr_ wrote:
| OSM is OpenStreetMaps. This appears to be a server to locally
| mirror the ~90GB global tile set and serve them with your choice
| of style to OSM client applications.
| andrewshadura wrote:
| It's not OpenStreetMaps, it's OpenStreetMap.
| Aachen wrote:
| I've always wondered why people keep adding the s. Is it
| marketing because Google Map sounds like a spelling mistake
| (missing 's after Google) and we're noticing the effect of
| that other brand, or is there something about OpenStreetMap
| that makes it seem like a plural project? People don't say
| Linuxes, Signals, or Androids
| axelthegerman wrote:
| Maybe because "Google Map" is actually called "Google Maps"
| (with the s)?
| karussell wrote:
| But where does the plural come from?
| e_y_ wrote:
| Maybe the old days when you needed multiple maps folded
| up on paper instead of one zoomable super map containing
| the entire world.
|
| Also "Google Map" sounds like a map of Google.
| NoboruWataya wrote:
| I think it's most likely that most of the other commonly
| used alternatives (most significantly Google Maps but also
| Apple Maps, Bing Maps, Maps.me) use the plural.
| Doctor_Fegg wrote:
| Whereas when OSM started, they didn't (StreetMap and
| Multimap in the UK, MapQuest in the US).
| sbarre wrote:
| Same feedback as everyone else: You fell into the trap of
| assuming that everyone else understands this topic/project/thing
| that you're deeply familiar with because you've been living it
| for so long.
|
| Not everyone knows what "OSM" is.
|
| Even a 2-3 sentence paragraph in your TL;DR that explains (a)
| what this is and (b) why people should care or want this, would
| be immensely helpful.
| Aachen wrote:
| > Not everyone knows what "OSM" is.
|
| Obligatory: https://xkcd.com/1053/
| dagw wrote:
| On the other hand if you don't know what "OSM" is, then you
| probably don't need this, and probably don't need to care.
| jakepage91 wrote:
| Image, screenshot, Gifs, anything really so I can see what it
| looks like before allocating 90GBS would be nice.
| mobilemidget wrote:
| 90 is okay, when I did years ago I was on 300+ but it doesn't
| really state which zoom level is available.
| btzs wrote:
| Is there any explanation why it has decreased in size?
| dagw wrote:
| There are two ways to store map tile data like OSM. One is
| as pre-rendered raster tiles (256x256 png images) and the
| other is as tiles of vector data that get rendered either
| on the fly by dedicate rendering middleware or rendered
| client side by the browser. Vector tiles are anywhere
| between 20-50% smaller that corresponding raster tiles.
| Raster tiles was the standard way of doing web maps up
| until a few years ago, when client side rendering in the
| browser became feasible and now most people use vector
| tiles.
| dagw wrote:
| It seems to be 0-14 according to the GitHub repo.
| moffkalast wrote:
| Yeah going to full 20 requires about 2TB iirc, OSM doesn't
| host beyond level 19 anymore either to reduce traffic. 14
| is around 10m per pixel, best you'll be able to make out on
| that is like a street. Not useful enough to bother with
| imho, even 19 leaves a lot to be desired up close.
|
| https://wiki.openstreetmap.org/wiki/Zoom_levels
| dagw wrote:
| Worth noting that these are vector tiles and not raster
| tiles, so zoom levels work slightly differently.
| Generally you don't need as high zoom level with vector
| tiles since it is much easier to 'zoom in' on vector data
| than on raster data. A vector zoom level of 14 stores the
| data with roughly 50cm precision which is good enough for
| most casual mapping uses.
| moffkalast wrote:
| Ah quite right, I see they're mvt. In that case it might
| actually be usable.
| trojan13 wrote:
| Where is my .exe file to run the entire planet in ~30 minutes?
|
| To add some value to my joke comment: I would assume this is
| running "Open Street Map" locally on a webserver.
| robertlagrant wrote:
| Because various comments are upset that this isn't actually self-
| hosting a planet simulation[0] or that the readme isn't good
| enough - this is really cool. Premade tile server for
| OpenStreetMaps is lovely.
|
| [0] https://news.ycombinator.com/item?id=41223091
| IgorPartola wrote:
| I remember when I worked on a project in 2010-2012 that relied on
| OSM it took days to set up a part of the system, the
| documentation was mostly in German or if you were lucky in an
| arcane bash script, and which part did what required a deep
| understanding of the entire thing. I don't need this anymore but
| this is really nice.
| mosselman wrote:
| So I assume if I run this I can aim something like leaflet at it
| for the map tiles?
|
| Any information you have on how production-ready this is would be
| nice. Things such as performance, stability, max expected clients
| with a small vps or something.
| tuxone wrote:
| Some things you may want to consider when hosting map tiles in
| production:
|
| - load balancing and caching. Some tiles are much more popular
| than others.
|
| - how will you update the tiles (sync from OSM).
|
| - if required, how will you manage customizations (and merges
| with OSM).
| leighleighleigh wrote:
| A lotta negativity in here... My immediate reaction was "fuck
| yeah I always wanted to host OSM but I couldn't figure out how
| last time". thanks dude :)
| dschep wrote:
| without a means to update the map, this isn't a breakthrough. I'd
| rather use protomaps[0](which has regular builds, from a better-
| known source) with caddy.
|
| [0] https://protomaps.com/
| kh_hk wrote:
| While useful, this still uses a server to process tile requests.
| Why not https://github.com/protomaps/PMTiles ?
|
| For the record, I am currently using maptiler over at
| https://citybik.es to serve tiles, and I am evaluating moving to
| serving static pmtiles on range requests to cut the middle man
| wiredfool wrote:
| PMTiles is great, but there's a risk hosting multi-gb files on
| metered hosting -- a client might not respect range requests
| and you open yourself up to a denial of wallet attack.
|
| I can't really see doing it on S3, unless it's behind a
| lightweight tile server. I'd do it on hetzner or linode or one
| of the lower cost bandwidth providers.
|
| (edit -- citybik.es is pretty cool, are you backing up to GBFS
| or a different set of data feeds? e.g.
| https://github.com/MobilityData/gbfs/blob/master/systems.csv is
| a list of ~ 800 gbfs feeds, some dockless, some docked I've
| done a bit of work in this space, feel free to ping me)
| kh_hk wrote:
| Will keep that in mind if I go that route.
|
| GBFS is just one format of the many supported. This is where
| we keep all instance data
| https://github.com/eskerda/pybikes/tree/master/pybikes/data .
| This is the list of GBFS systems we have https://github.com/e
| skerda/pybikes/blob/master/pybikes/data/...
| bdon wrote:
| To mitigate the "denial of wallet" attack on S3 there are a
| few options:
|
| AWS Lambda: https://docs.protomaps.com/deploy/aws lightweight
| tile server: https://docs.protomaps.com/pmtiles/cli#serve
| sorenjan wrote:
| Hosting maps of the entire planet in a single file doesn't seem
| practical. How do you keep it updated? PMTiles is good for
| small local maps, but I don't think they're meant for this
| scale.
| kh_hk wrote:
| I assumed you would keep it updated the same way you do with
| mbtiles, but I see that mbtiles are an sqlite so I guess you
| can do atomic updates with mbtiles.
|
| Note though that there are many cases on which served tile
| information rarely changes. A lot of us are serving a once-
| downloaded mbtile file and are using this just to _not_ use a
| provider (mapbox, ...)
| sorenjan wrote:
| > Note though that there are many cases on which served
| tile information rarely changes.
|
| It depends on what you're doing, but outdated maps can be
| worse than no maps. On low zoom values that probably works
| fine, but new buildings and streets are built everyday all
| over the world.
| dagw wrote:
| _everyday all over the world_
|
| If you need updates more often than once a week, you're
| going to have to take a different approach, since OSM
| only publish their database exports once a week. In that
| case you're going to have to use the OSM API to query
| their database for changes directly and it is only
| reasonable to track and manage continuous updates on a
| relatively small surface area.
|
| All of this is of course ignoring the fact that there can
| be a time lag of anything from hours to decades between a
| house being built and that house being added to OSM.
| dagw wrote:
| For updates you're probably better off just regenerating and
| replacing the mbtiles file from scratch every time you want a
| new version, rather than trying to incrementally update it.
| bdon wrote:
| Here's an example of the entire planet as a PMTiles:
|
| https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fpro.
| ..
|
| This is zoom 0-15, or 1,431,655,765 addressed tiles. So it is
| designed for this scale - for a production internet site you
| can add a lambda, server or CDN as an additional layer for
| lower latency: https://docs.protomaps.com/deploy/
| hk__2 wrote:
| Why copy the README as a gist instead of posting a link to the
| repo? https://github.com/markuman/sms?tab=readme-ov-file
| andrewshadura wrote:
| It's OpenStreetMap, without the "s" at the end, please fix the
| title.
| lopkeny12ko wrote:
| OSM has been around for ages and is not novel.
| donalhunt wrote:
| 20 years +- a few days...
|
| https://birthday20.openstreetmap.org/
| xrd wrote:
| Does anyone know if there is a way to host just a few of the
| tiles in a static way? For example, if I wanted to build a web
| page which just shows a map at zoom level 6 for a lat/lng point,
| and then go to zoom level 13. That would require a tiny subset of
| tiles; is there a simple way to add the tiles plus the JS code in
| a static way so no external downloads were necessary?
| Raphaellll wrote:
| You can simply pre-render them and store the png tiles in a
| folder. The folder structure and naming has to be such that
| e.g. leaflet understands it. Did this for a research demo here:
| https://map2seq.schumann.pub/nllni/demo/
| orblivion wrote:
| Is there actually a simple way to pre-render? AFAIK it's a
| lot of work and learning at least if you're coming from
| scratch.
| wiredfool wrote:
| QGIS can make a tile pyramid of the current map/project at
| a specific set of bounds.
| jonah wrote:
| I've used TileMill to create tiles for a specific layer of
| data for a specific area. (Fire evacuation zones for a
| county.)
|
| https://github.com/tilemill-project
| brennopost wrote:
| I had success with Protomaps (https://protomaps.com/).
| orblivion wrote:
| Look at https://protomaps.com/ - You can have a single file in
| a that represents the vector data for a region at multiple zoom
| levels. There used to be a really nice UI where you could just
| select a region. They've changed things around a bit since then
| so I'm not sure how it works now.
| bdon wrote:
| The replacement for the UI is to use the "pmtiles extract"
| command line tool:
|
| https://docs.protomaps.com/guide/getting-
| started#_3-extract-...
| l72 wrote:
| I've used maptiler to do this. You can give it a boundary and
| zoom levels, along with a theme, and generate prerendered tiles
| that you can then server just using http. This has worked well
| with leaflet.
| simonw wrote:
| Yes, that's pretty easy. I did it for Half Moon Bay, California
| here: https://simonw.github.io/hmb-map/
|
| Code here: https://github.com/simonw/hmb-map
|
| TIL here: https://til.simonwillison.net/gis/pmtiles
| gregoryw3 wrote:
| Tangentially related, does anyone know of an open source 3D
| terrain system?
|
| I've been thinking of making a native map and globe program
| similar to Google earth to embed inside a dashboard (visually
| similar to kerbal space program).
|
| So far I was thinking about just using ArcGIS (or something else)
| and taking in free Lidar data and imposing the depth to satellite
| or OSM tiles then rendering them as OBJs in OpenGL (or maybe a
| full 3d engine like unity). This unfortunately wouldn't be an
| automatic process as lidar data is taxing to process in real
| time, especially for a SBC.
|
| I do see that there are some 3D tiles sets out there but I don't
| see where I can pull them from nor how it compares to the lidar
| data.
|
| However, the main goal is a 3D terrain viewer in an offline
| native program.
| cyberax wrote:
| > Tangentially related, does anyone know of an open source 3D
| terrain system?
|
| You can get the heightmap from NASA. It's 10x10 meter
| resolution, which is enough for static geography.
| simlevesque wrote:
| I'd like to find a cheap way to self-host a stripped version of
| Nominatim planet. There's lots of data that I don't really need
| to deal with and I don't want to have to use 32/64gb of ram.
| Borg3 wrote:
| Hmm, Im not really sure why MBTiles are popular? They seems to be
| cache unfriendly. Good old tiles via /{x}/{y}/{z}.png seems to be
| better. And you can slap web cache in front.
|
| Hosting those is super simple. The problem I have is how to
| generate them efficently w/o big resources.
___________________________________________________________________
(page generated 2024-08-12 23:01 UTC)