Generating MBTiles from OSM GeoJSON

Run the whole chain — extract to layers to ranked GeoJSON to an archive — with enough checks at the end that you know the archive is correct without opening a map.

Prerequisites

Conceptual minimum

An MBTiles archive is a SQLite database with two things that matter: a tiles table keyed on zoom, column and row, and a metadata table of key-value pairs describing the tile set. The metadata is not decoration — clients read minzoom, maxzoom, bounds and the json field describing layers and their attributes, and a client given an archive with wrong metadata will request tiles that do not exist or fail to find the layers a style names.

The row numbering is the detail that catches people out. MBTiles stores rows in TMS order, where row 0 is at the bottom, while the tile addressing used by most web clients and by the tile pyramid model puts row 0 at the top. The conversion is \(y_{\text{tms}} = 2^{z} - 1 - y\), and getting it wrong produces a map that is vertically mirrored at the archive level — every tile individually correct, in the wrong place.

The four stages of a build and the artefact each one produces Four consecutive stages. The filter stage runs osmium over the verified extract once per layer, producing a small PBF holding only that layer's features. The export stage converts each layer PBF into line-delimited GeoJSON with stable identifiers. The rank stage adds per-feature minimum and maximum zoom properties derived from the tags. The build stage runs the generator over every ranked layer at once, producing a single MBTiles archive with metadata describing all of them. Four stages, four artefacts, one archive filter per-layer PBF osmium tags-filter one pass per layer smallest intermediate export GeoJSON lines osmium export stable type-prefixed ids largest intermediate rank ranked GeoJSON minzoom and maxzoom from the OSM tags the cartographic decision build MBTiles all layers in one run metadata written once the shipped artefact Running the generator once over all layers rather than once per layer is what produces a single coherent metadata record.
The export stage produces by far the largest intermediate, which is why filtering happens before it rather than after.

Runnable solution

bash 
#!/usr/bin/env bash
# Build an MBTiles archive from a verified OSM extract.
set -euo pipefail

EXTRACT="${1:?usage: build.sh <extract.osm.pbf>}"
WORK="$(mktemp -d)"
OUT="osm.mbtiles"
MAXZOOM=14
trap 'rm -rf "$WORK"' EXIT

# One filter pass per layer. Each reads the extract once and writes a small file.
filter_layer () {   # name, then osmium tags-filter expressions
  local name="$1"; shift
  osmium tags-filter --output "$WORK/$name.osm.pbf" "$EXTRACT" "$@"
  osmium export --output-format=geojsonseq --add-unique-id=type_id \
    --output "$WORK/$name.geojsonseq" "$WORK/$name.osm.pbf"
  python3 assign_zoom.py < "$WORK/$name.geojsonseq" > "$WORK/$name.ranked.geojsonseq"
  wc -l < "$WORK/$name.ranked.geojsonseq" | xargs echo "$name features:"
}

filter_layer transportation w/highway r/route=road
filter_layer water          nwr/natural=water nwr/waterway w/landuse=reservoir
filter_layer landuse        nwr/landuse nwr/leisure=park nwr/natural=wood
filter_layer place          n/place

tippecanoe \
  --output="$OUT" --force \
  --minimum-zoom=0 --maximum-zoom="$MAXZOOM" \
  --named-layer=transportation:"$WORK/transportation.ranked.geojsonseq" \
  --named-layer=water:"$WORK/water.ranked.geojsonseq" \
  --named-layer=landuse:"$WORK/landuse.ranked.geojsonseq" \
  --named-layer=place:"$WORK/place.ranked.geojsonseq" \
  --coalesce --reorder --maximum-tile-bytes=500000 \
  --name="OSM base map" --attribution="© OpenStreetMap contributors" \
  2>&1 | tee build.log
python 
from __future__ import annotations

import json
import logging
import sqlite3
import zlib
from pathlib import Path

import mapbox_vector_tile

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.tiles.verify")

EXPECTED_LAYERS = {"transportation", "water", "landuse", "place"}


def metadata(conn: sqlite3.Connection) -> dict[str, str]:
    return {k: v for k, v in conn.execute("SELECT name, value FROM metadata")}


def sample_tile(conn: sqlite3.Connection, z: int, x: int, y: int) -> bytes | None:
    # MBTiles rows are TMS-ordered: row 0 is at the BOTTOM of the pyramid.
    tms_y = (1 << z) - 1 - y
    row = conn.execute(
        "SELECT tile_data FROM tiles WHERE zoom_level=? AND tile_column=? "
        "AND tile_row=?", (z, x, tms_y)).fetchone()
    return row[0] if row else None


def verify(path: Path, z: int, x: int, y: int) -> bool:
    conn = sqlite3.connect(path)
    ok = True
    meta = metadata(conn)

    for key in ("name", "format", "minzoom", "maxzoom", "bounds", "attribution"):
        if key not in meta:
            logger.error("metadata missing %r", key)
            ok = False
    logger.info("zoom range %s..%s, bounds %s",
                meta.get("minzoom"), meta.get("maxzoom"), meta.get("bounds"))

    # The json field advertises the layers a style will look for.
    declared = {layer["id"] for layer in
                json.loads(meta.get("json", '{"vector_layers":[]}'))["vector_layers"]}
    if declared != EXPECTED_LAYERS:
        logger.error("layers declared %s, expected %s", declared, EXPECTED_LAYERS)
        ok = False

    total, = conn.execute("SELECT COUNT(*) FROM tiles").fetchone()
    logger.info("%d tile(s) in the archive", total)
    if total == 0:
        return False

    blob = sample_tile(conn, z, x, y)
    if blob is None:
        logger.error("no tile at %d/%d/%d — check the TMS row conversion", z, x, y)
        return False
    # Tippecanoe gzips tile payloads; decompress before decoding.
    raw = zlib.decompress(blob, 16 + zlib.MAX_WBITS) if blob[:2] == b"\x1f\x8b" else blob
    decoded = mapbox_vector_tile.decode(raw)
    logger.info("sample tile carries layers %s", sorted(decoded))
    if not set(decoded) & EXPECTED_LAYERS:
        logger.error("sample tile has none of the expected layers")
        ok = False

    conn.close()
    return ok


if __name__ == "__main__":
    logger.info("verification %s",
                "PASSED" if verify(Path("osm.mbtiles"), 14, 9111, 5455) else "FAILED")

Step-by-step walkthrough

  1. Filter before exporting. Each tags-filter pass reads the extract and writes a small per-layer file, so the expensive GeoJSON conversion only ever sees features that will actually be rendered.
  2. Use line-delimited GeoJSON. A single feature collection has to be parsed as one document; the line-delimited form streams and can be processed with ordinary text tools.
  3. Emit stable identifiers. Type-prefixed identifiers survive into the verification step and into any later reconciliation against the source.
  4. Rank each layer separately. The ranking rules differ per layer, and running the script per file keeps each histogram readable.
  5. Build all layers in one run. A single invocation writes one coherent metadata record; building per layer and merging archives afterwards produces metadata that is wrong in ways clients do not report.
  6. Set name and attribution at build time. Attribution is a licence obligation, and the archive’s metadata is where a client will look for it.
  7. Convert the row when reading. The verification’s sample lookup converts from top-origin to TMS ordering explicitly, which is both correct and a reminder of the convention.
  8. Decompress before decoding. Tile payloads are gzipped; feeding a compressed blob to a decoder produces a confusing parse error rather than a clear one.
What the verification checks and what each failure would have looked like in production A grid of four checks against the production symptom each one prevents. Checking the metadata keys prevents a client that cannot determine the zoom range and requests tiles outside it. Checking the declared layer list prevents a style that silently matches nothing and renders a blank map. Checking the tile count prevents shipping an archive whose build failed partway. Checking a decoded sample tile prevents an archive whose tiles exist but are empty or wrongly addressed. Four checks, four production failures avoided Would have looked like Metadata keys present client requests absent zooms Layer list matches blank map, no errors Tile count non-zero archive ships empty Sample tile decodes tiles exist, render nothing Every one of these failures is silent on the client: a blank map produces no console error and no failed request.
That silence is why the checks belong in the build rather than in somebody opening the map afterwards.
Relative size of each intermediate artefact in the build, for one country Five artefacts compared on disk size for a country-sized build. The source extract is the baseline. The per-layer filtered files together are a fraction of it, because only rendered features survive. The exported GeoJSON is many times the extract, because the text format is far more verbose than the binary one. The ranked GeoJSON is slightly larger again, having gained two properties per feature. The final archive is smaller than the extract, because tiles hold generalized geometry rather than full precision. The text intermediate dwarfs everything else Source extract 1x baseline Filtered per-layer files about 0.4x Exported GeoJSON about 5x Ranked GeoJSON about 5.7x Final MBTiles about 0.6x Filtering before exporting keeps the third bar at five times the extract rather than fifteen, which is the reason for the ordering.
The archive being smaller than its own source is the point: tiles carry what a reader can see, not what the data knows.

Verification

  • Metadata carries the required keys. Name, format, zoom range, bounds and attribution must all be present.
  • The declared layer list matches the build. A style written against a layer name absent from the metadata renders nothing and reports nothing.
  • The tile count is plausible. Compare against the rough expectation for the area and zoom range; an order-of-magnitude shortfall means the build stopped early.
  • A sample tile decodes and carries layers. Pick a tile in a dense area, decode it, and confirm the expected layers are present.
  • The row conversion is right. Request a tile whose contents you can recognise; a mirrored archive returns a tile from the wrong latitude rather than nothing.

Common errors and fixes

Symptom Root cause One-line fix
Map is vertically mirrored TMS row ordering not converted Convert with the zoom-dependent row formula
Decoder raises a parse error Tile payload still gzipped Decompress before decoding
Style renders nothing Layer names differ from the metadata Name layers explicitly and verify the declared list
Archive far smaller than expected Build stopped early on an input error Check the build log; each layer should report a count
Attribution missing from the client Not set at build time Pass name and attribution to the build
Intermediate GeoJSON fills the disk Export ran before filtering Filter to a per-layer PBF first
Layers have inconsistent zoom ranges Built per layer and merged Build every layer in a single invocation

Specification reference

An MBTiles archive is a SQLite database containing a tiles table keyed on zoom_level, tile_column and tile_row, and a metadata table of name-value pairs including name, format, bounds, minzoom, maxzoom and, for vector tile sets, a json field describing the vector layers and their fields. Tile rows are numbered in TMS order with row zero at the bottom. See the MBTiles specification for the required metadata keys and the row ordering.

Frequently Asked Questions

Why is my map vertically mirrored?

Because MBTiles numbers tile rows from the bottom while most client addressing numbers them from the top. Each individual tile is correct; they are simply being placed in the wrong rows. The conversion is a single expression involving the zoom level, and applying it in exactly one place — the archive reader — keeps the rest of the pipeline in one convention.

Should I build each layer separately and merge the archives?

No. A single build writes one metadata record describing every layer and one consistent zoom range; merging separately built archives leaves metadata that describes only one of them, which clients read and then fail to find the other layers. If the layers genuinely must be built separately for scheduling reasons, regenerate the metadata deliberately rather than accepting whichever archive’s record survives the merge.

Why filter to a per-layer PBF instead of exporting once?

Because GeoJSON is an order of magnitude more verbose than PBF, so every feature you export and then discard costs disk and parsing time. Filtering first means the expensive conversion only touches features that will be rendered. On a country extract the difference is routinely tens of gigabytes of intermediate output and a large fraction of the build time.

Is the attribution field really necessary?

Yes, and it is the one field with a licence obligation attached rather than a technical one. OpenStreetMap data carries an attribution requirement, and the archive’s metadata is where a client looks for the text to display. Setting it at build time means every consumer of the archive inherits it automatically, rather than depending on whoever wires up the map remembering to add it.

Up one level: Building OSM Tiles with Tippecanoe.