https://github.com/onthegomap/flatmap Skip to content Sign up * Why GitHub? Features - + Mobile - + Actions - + Codespaces - + Packages - + Security - + Code review - + Issues - + Integrations - + GitHub Sponsors - + Customer stories- * Team * Enterprise * Explore + Explore GitHub - Learn and contribute + Topics - + Collections - + Trending - + Learning Lab - + Open source guides - Connect with others + The ReadME Project - + Events - + Community forum - + GitHub Education - + GitHub Stars program - * Marketplace * Pricing Plans - + Compare plans - + Contact Sales - + Education - [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} onthegomap / flatmap Public * Notifications * Star 135 * Fork 2 * Fast and flexible tool to build vector tiles from OpenStreetMap data Apache-2.0 License 135 stars 2 forks Star Notifications * Code * Issues 0 * Pull requests 0 * Discussions * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Security * Insights main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 1 branch 4 tags Code Latest commit @msbarry msbarry Switch from github maven packages to maven central (#7) ... e402a33 Oct 25, 2021 Switch from github maven packages to maven central (#7) e402a33 Git stats * 215 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github Switch from github maven packages to maven central (#7) Oct 25, 2021 .idea CI and docs Oct 20, 2021 .mvn/wrapper CI and docs Oct 20, 2021 diagrams CI and docs Oct 20, 2021 flatmap-basemap basemap README tweaks Oct 24, 2021 flatmap-benchmarks CI and docs Oct 20, 2021 flatmap-core Linode test (#1) Oct 21, 2021 flatmap-dist CI and docs Oct 20, 2021 flatmap-examples Switch from github maven packages to maven central (#7) Oct 25, 2021 planet-logs Linode test (#1) Oct 21, 2021 scripts Switch from github maven packages to maven central (#7) Oct 25, 2021 .editorconfig initial commit Apr 6, 2021 .gitignore CI and docs Oct 20, 2021 ARCHITECTURE.md tweak wording in ARCHITECTURE.md Oct 24, 2021 CONTRIBUTING.md CI and docs Oct 20, 2021 LICENSE apache 2.0 license Sep 21, 2021 NOTICE.md CI and docs Oct 20, 2021 PLANET.md clarify cost of z13 building merge Oct 24, 2021 README.md typo Oct 25, 2021 config-example.properties Change transportation name defaults (#2) Oct 23, 2021 grafana.json comments and cleanup Sep 10, 2021 mvnw CI and docs Oct 20, 2021 mvnw.cmd CI and docs Oct 20, 2021 pom.xml Switch from github maven packages to maven central (#7) Oct 25, 2021 quickstart-from-source.sh CI and docs Oct 20, 2021 View code [ ] Flatmap Demo Usage To build the map: To view tiles locally: Generating a Map of the World Creating a Custom Map Benchmarks Alternatives Features Limitations Roadmap Contributing Support Acknowledgement Author License and Attribution README.md Flatmap Flatmap is a tool that generates Mapbox Vector Tiles from geographic data sources like OpenStreetMap. Flatmap aims to be fast and memory-efficient so that you can build a map of the world in a few hours on a single machine without any external tools or database. Vector tiles contain raw point, line, and polygon geometries that clients like MapLibre can use to render custom maps in the browser, native apps, or on a server. Flatmap packages tiles into an MBTiles (sqlite) file that can be served using tools like TileServer GL or even queried directly from the browser. See awesome-vector-tiles for more projects that work with data in this format. Flatmap is named after the "flatmap" operation that it performs: map input elements to rendered tile features, flatten them into a big list, and sort by tile ID to group into tiles. The output is also a "flat map" where zoom level 0 contains the entire planet in a 256x256 px tile, and each zoom level splits parent tiles into 4 quadrants, revealing more detail. See ARCHITECTURE.md for more details on how Flatmap works or this blog post for more of the backstory. Demo See the live demo of vector tiles created by Flatmap. Flatmap Demo Screenshot Style (c) OpenMapTiles * Data (c) OpenStreetMap contributors Usage To generate a map of an area using the basemap profile, you will need: * Java 16+ or Docker * at least 1GB of free disk space plus 5-10x the size of the .osm.pbf file * at least 1.5x as much free RAM as the input .osm.pbf file size To build the map: Using Java, download flatmap.jar from the latest release and run it: wget https://github.com/onthegomap/flatmap/releases/latest/download/flatmap.jar java -Xmx1g -jar flatmap.jar --download --area=monaco Or using Docker: docker run -e JAVA_TOOL_OPTIONS="-Xmx1g" -v "$(pwd)/data":/data ghcr.io/onthegomap/flatmap:latest --download --area=monaco [?][?] This starts off by downloading about 1GB of data sources required by the basemap profile including ~750MB for ocean polygons and ~240MB for Natural Earth Data. To download smaller extracts just for Monaco: Java: java -Xmx1g -jar flatmap.jar --download --area=monaco \ --water-polygons-url=https://github.com/onthegomap/flatmap/raw/main/flatmap-core/src/test/resources/water-polygons-split-3857.zip \ --natural-earth-url=https://github.com/onthegomap/flatmap/raw/main/flatmap-core/src/test/resources/natural_earth_vector.sqlite.zip Docker: docker run -e JAVA_TOOL_OPTIONS="-Xmx1g" -v "$(pwd)/data":/data ghcr.io/onthegomap/flatmap:latest --download --area=monaco \ --water-polygons-url=https://github.com/onthegomap/flatmap/raw/main/flatmap-core/src/test/resources/water-polygons-split-3857.zip \ --natural-earth-url=https://github.com/onthegomap/flatmap/raw/main/flatmap-core/src/test/resources/natural_earth_vector.sqlite.zip You will need the full data sources to run anywhere besides Monaco. To view tiles locally: Using Node.js: npm install -g tileserver-gl-light tileserver-gl-light --mbtiles data/output.mbtiles Or using Docker: docker run --rm -it -v "$(pwd)/data":/data -p 8080:8080 maptiler/tileserver-gl -p 8080 Then open http://localhost:8080 to view tiles. Some common arguments: * --download downloads input sources automatically and --only-download exits after downloading * --area=monaco downloads a .osm.pbf extract from Geofabrik * --osm-path=path/to/file.osm.pbf points Flatmap at an existing OSM extract on disk * -Xmx1g controls how much RAM to give the JVM (recommended: 0.5x the input .osm.pbf file size to leave room for memory-mapped files) * --force overwrites the output file * --help shows all of the options and exits Generating a Map of the World See PLANET.md. Creating a Custom Map See the flatmap-examples project. Benchmarks Some example runtimes (excluding downloading resources): Input Profile Machine Time mbtiles Logs size s3://osm-pds/2021/ DO 16cpu 3h9m logs, planet-211011.osm.pbf Basemap 128GB cpu:42h1m 99GB VisualVM (65GB) avg:13.3 Profile Daylight Distribution 3h13m v1.6 with ML Basemap DO 16cpu cpu:43h40m 101GB logs buildings and admin 128GB avg:13.5 boundaries (67GB) Basemap s3://osm-pds/2021/ (without Linode 50cpu 1h9m logs, planet-211011.osm.pbf z13 128GB cpu:24h36m 97GB VisualVM (65GB) building avg:21.2 Profile merge) Basemap s3://osm-pds/2021/ (without c5ad.16xlarge 59m planet-211011.osm.pbf z13 (64cpu/128GB) cpu:27h6m 97GB logs (65GB) building avg:27.4 merge) Alternatives Some other tools that generate vector tiles from OpenStreetMap data: * OpenMapTiles is the reference implementation of the OpenMapTiles schema that the basemap profile is based on. It uses an intermediate postgres database and operates in two modes: 1. Import data into database (~1 day) then serve vector tiles directly from the database. Tile serving is slower and requires bigger machines, but lets you easily incorporate realtime updates 2. Import data into database (~1 day) then prerender every tile for the planet into an mbtiles file which takes over 100 days or a cluster of machines, but then tiles can be served faster on smaller machines * Tilemaker uses a similar approach to Flatmap (no intermediate database), is more mature, and has a convenient lua API for building custom profiles without recompiling the tool, but takes about a day to generate a map of the world Some companies that generate and host tiles for you: * Mapbox - data from the pioneer of vector tile technologies * Maptiler - data from the creator of OpenMapTiles schema * Stadia Maps - what onthegomap.com uses in production If you want to host tiles yourself but have someone else generate them for you, those companies also offer plans to download regularly-updated tilesets. Features * Supports Natural Earth, OpenStreetMap .osm.pbf, and Esri Shapefiles data sources * Java-based Profile API to customize how source elements map to vector tile features, and post-process generated tiles using JTS geometry utilities * Merge nearby lines or polygons with the same tags before emitting vector tiles * Automatically fixes self-intersecting polygons * Built-in basemap profile based on OpenMapTiles v3.12.2 * Optionally download additional name translations for elements from Wikidata * Export real-time stats to a prometheus push gateway using --pushgateway=http://user:password@ip argument (and a grafana dashboard for viewing) * Automatically downloads region extracts from Geofabrik using geofabrik:australia shortcut as a source URL * Unit-test profiles to verify mapping logic, or integration-test to verify the actual contents of a generated mbtiles file ( example) Limitations * It is harder to join and group data than when using database. Flatmap automatically groups features into tiles, so you can easily post-process nearby features in the same tile before emitting, but if you want to group or join across features in different tiles, then you must explicitly store data when processing a feature to use with later features or store features and defer processing until an input source is finished (boundary layer example) * Flatmap only does full imports from .osm.pbf snapshots, there is no way to incorporate real-time updates. Roadmap * [*] Enough flatmap-core functionality to support basemap profile based on OpenMapTiles * [ ] Basemap profile based on OpenMapTiles v3.12.2 + [*] Port all layers + [*] Download name translations from wikidata + [*] Merge buildings at z13 + [*] adm0_l/adm0_r boundary labels + [ ] Abbreviate road names to improve visibility + [ ] Poi layer agg_stop tag * [ ] Get flatmap-core into Maven Central * [ ] Remove geotools dependency for reading shapefiles (not in Maven Central) * [ ] Remove graphhopper dependency for reading OSM files, and use LocationsOnWays to skip node location storage when present and reduce memory requirement by 70%. * [ ] "Sparse mode" to only store node and relation data for elements used by a profile * [ ] Support zoom levels higher than 14 * [ ] Handle nodes and relations in relations (only ways handled now) * [ ] Lake centerline support in flatmap-core * [ ] Improve line merging to combine nearby parallel roads * [ ] Basemap schema improvements for onthegomap.com * [ ] Accept other kinds of data sources * [ ] Extract reusable utilities for complex schemas from flatmap-basemap to flatmap-core * [ ] Other schemas Contributing Pull requests are welcome! See CONTRIBUTING.md for details. Support Have a question or want to share something you've built? Start a GitHub discussion. Found a bug or have a feature request? Open a GitHub issue to report. This is a side project, so support is limited. If you have the time and ability, feel free to open a pull request to fix issues or implement new features. Acknowledgement Flatmap is made possible by these awesome open source projects: * OpenMapTiles for the schema and reference implementation that the basemap profile is based on * Graphhopper for utilities to process OpenStreetMap data in Java * JTS Topology Suite for working with vector geometries * Geotools for shapefile processing * SQLite JDBC Driver for reading Natural Earth data and writing MBTiles files * MessagePack for compact binary encoding of intermediate map features * geojson-vt for the basis of the stripe clipping algorithm that flatmap uses to slice geometries into tiles * java-vector-tile for the basis of the vector tile encoder * imposm3 for the basis of OSM multipolygon processing and tag parsing utilities See NOTICE.md for a full list and license details. Author Flatmap was created by Michael Barry for future use generating custom basemaps or overlays for On The Go Map. License and Attribution Flatmap source code is licensed under the Apache 2.0 License, so it can be used and modified in commercial or other open source projects according to the license guidelines. Maps built using flatmap do not require any special attribution, but the data or schema used might. Any maps generated from OpenStreetMap data must visibly credit OpenStreetMap contributors. Any map generated with the profile based on OpenMapTiles or a derivative must visibly credit OpenMapTiles as well. About Fast and flexible tool to build vector tiles from OpenStreetMap data Resources Readme License Apache-2.0 License Releases 3 v0.1.0 Latest Oct 25, 2021 + 2 releases Packages 3 Languages * Java 99.7% * Shell 0.3% * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.