Running Planetiler on a Regional Extract

Get a complete tile archive out of a country-sized extract in one run, on hardware you actually have, by sizing the two storage decisions before you start rather than discovering them at hour four.

Prerequisites

Conceptual minimum

A Planetiler run has two storage decisions and everything else follows from them.

Node location storage. Building way geometry needs every referenced node’s coordinate. Planetiler can hold that map on the Java heap, in off-heap memory-mapped files, or in a direct memory-mapped array indexed by node identifier. The array is the fastest and is sized by the highest node identifier in the planet, not by your region — which is why a small country extract still wants tens of gigabytes of address space. Memory mapping makes that workable: the operating system pages in what is touched.

The temporary feature store. Every feature the profile emits is written to a temporary store, sorted spatially, then read back per tile. This is sequential-write then sequential-read, so it wants throughput rather than low latency, and it wants room — typically a few times the extract size.

Neither decision is about the tile output. Both are about whether the build completes.

The four phases of a Planetiler run and what each one is limited by Four stacked phases. The read phase streams the extract and runs the profile per element, limited by profile cost and by node store access. The sort phase orders the temporary feature store spatially, limited by disk throughput and available memory for the merge. The render phase reads the sorted store and encodes tiles, limited by CPU and by the maximum zoom. The write phase appends tiles to the archive, limited by disk and by the archive format's write pattern. Four phases, four different bottlenecks Read Stream elements, run the profile profile and node store Sort Order the feature store spatially disk throughput Render Encode tiles from the sorted store CPU and maximum zoom Write Append tiles to the archive disk write pattern Because each phase has a different limit, a build that is slow tells you which resource to add only once you know which phase is slow.
Watching which phase dominates is faster than guessing, and the progress output names each one as it runs.

Runnable solution

bash 
#!/usr/bin/env bash
# Build a regional tile archive with Planetiler.
set -euo pipefail

EXTRACT="${1:?usage: build.sh <extract.osm.pbf>}"
AREA="$(basename "$EXTRACT" .osm.pbf)"
TMP="/fast-local/planetiler-tmp"       # MUST be local disk, not network
OUT="${AREA}.pmtiles"
MAXZOOM=14

mkdir -p "$TMP"
FREE_KB="$(df -Pk "$TMP" | awk 'NR==2 {print $4}')"
EXTRACT_KB="$(du -k "$EXTRACT" | cut -f1)"
# The temporary store plus the node store want several times the extract.
if [ "$FREE_KB" -lt $(( EXTRACT_KB * 6 )) ]; then
  echo "need ~$(( EXTRACT_KB * 6 / 1024 / 1024 )) GiB free on $TMP, have $(( FREE_KB / 1024 / 1024 ))" >&2
  exit 1
fi

java -Xmx8g \
  -jar planetiler.jar \
  --osm-path="$EXTRACT" \
  --output="$OUT" \
  --maxzoom="$MAXZOOM" \
  --tmpdir="$TMP" \
  --nodemap-type=array \
  --nodemap-storage=mmap \
  --force \
  2>&1 | tee "build-${AREA}.log"

# The log records per-phase timings; keep it beside the archive.
grep -E "^\s*(read|sort|render|write)" "build-${AREA}.log" || true
python 
from __future__ import annotations

import logging
import re
import sys
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.planetiler.log")

# Planetiler prints a line per phase with an elapsed time, e.g. "  read: 12m34s".
PHASE_RE = re.compile(r"^\s*(\w+)\s*:\s*(?:(\d+)m)?(\d+(?:\.\d+)?)s", re.M)


def parse_phases(log: str) -> dict[str, float]:
    phases: dict[str, float] = {}
    for name, minutes, seconds in PHASE_RE.findall(log):
        phases[name] = float(minutes or 0) * 60 + float(seconds)
    return phases


def diagnose(log_path: Path) -> str:
    phases = parse_phases(log_path.read_text(encoding="utf-8"))
    if not phases:
        return "no phase timings found — did the build finish?"
    total = sum(phases.values())
    for name, seconds in sorted(phases.items(), key=lambda kv: -kv[1]):
        logger.info("%-8s %6.0fs  %4.1f%%", name, seconds, 100 * seconds / total)

    worst = max(phases, key=phases.get)
    share = phases[worst] / total
    if worst == "read" and share > 0.55:
        return "READ dominates: the profile is doing too much work per element"
    if worst == "sort" and share > 0.45:
        return "SORT dominates: give the temporary store faster disk or more memory"
    if worst == "render" and share > 0.45:
        return "RENDER dominates: lower the maximum zoom or add cores"
    return f"balanced; {worst} is largest at {share:.0%}"


if __name__ == "__main__":
    logger.info("verdict: %s", diagnose(Path(sys.argv[1])))

Step-by-step walkthrough

  1. Check the disk before launching. The script refuses to start when free space is under roughly six times the extract, because failing at the start costs seconds and failing at hour four costs the whole run.
  2. Keep the temporary directory local. The feature store is written and read at high throughput; a network volume turns a two-hour build into an overnight one.
  3. Choose the array node map with memory mapping. The array is indexed directly by node identifier, which is the fastest lookup available, and memory mapping means the address space it reserves is not resident memory.
  4. Keep the heap modest. Planetiler does most of its heavy storage off-heap, so an enormous heap does not help and takes memory the page cache would use better.
  5. Cap the maximum zoom deliberately. The render phase scales with tile count, which quadruples per level; this is the single largest lever on that phase.
  6. Tee the log. Per-phase timings are the only cheap diagnostic, and they are gone if the output is not kept.
  7. Diagnose from the phase shares. The parser turns the log into a verdict naming the resource to change, rather than a wall of numbers.
Reading a Planetiler build's phase timings to decide what to change A decision node taking the dominant phase as input, with three outcomes. When the read phase dominates, the profile is doing too much work per element and needs its lookups precomputed. When the sort phase dominates, the temporary feature store is limited by disk throughput or by memory available for the merge. When the render phase dominates, the maximum zoom is too high for the available cores, and lowering it or adding cores is the fix. Which phase dominated the build? Read the phase shares first The log names every phase One usually stands out Read dominates Profile cost per element; precompute its lookups Sort dominates Temporary store disk throughput or merge memory Render dominates Maximum zoom too high for the cores available A balanced build with no phase above half is already close to the hardware's limit, and the next gain is a bigger machine.
Each branch points at a different resource, which is why adding RAM to a render-bound build changes nothing.
The three node map storage options and what each one trades Three panels. A sorted table keyed on node identifier uses the least space because it stores only nodes that exist, but every lookup is a binary search. A direct array indexed by identifier gives constant-time lookup but reserves space for every possible identifier up to the planet maximum. Memory mapping either structure moves the reservation from resident memory to address space, letting the operating system page in what is touched, at the cost of depending on fast local disk. Three storage options, one of them is really two Sorted table Stores only existing nodes Smallest footprint Binary search per lookup Good on a small machine Slower read phase Direct array Indexed by identifier Constant-time lookup Sized by the planet maximum Fastest read phase Needs the space reserved Memory mapping Applies to either above Address space, not RAM OS pages in what is used Needs fast local disk The usual production choice Memory mapping an array is what makes a country build fit on an ordinary machine, provided the temporary directory is local.
The third panel is not a third structure; it is how the first two become affordable.

Verification

  • The archive opens and declares the expected zoom range. Read its metadata before anybody points a client at it.
  • A dense tile is non-trivial in size. Fetch a city-centre tile at the maximum zoom; a few kilobytes means the profile matched almost nothing.
  • Every expected layer appears. Decode a sample tile and compare the layer set against the profile.
  • Phase timings look sane. No phase should be over about three-quarters of the total on a well-provisioned machine.
  • The temporary directory is empty afterwards. Left-over files mean the run did not finish cleanly, whatever the exit status said.

Common errors and fixes

Symptom Root cause One-line fix
Build fails hours in, out of disk Temporary store under-provisioned Check for several times the extract size before starting
Build far slower than expected Temporary directory on a network volume Point it at local disk
Read phase dominates Profile doing per-element lookups Precompute into an immutable structure before the run
Render phase dominates Maximum zoom set too deep Lower it and rely on client overzoom
Out of memory with a large heap Heap taking memory the page cache needs Reduce the heap; storage is mostly off-heap
Tiles nearly empty Profile matches almost nothing Log per-layer match counts during a short run
Archive missing layers Profile branch never reached Assert each layer produced features before shipping

Specification reference

Planetiler builds a vector tile archive from an OSM extract in a single process, storing node locations in a configurable map — on the Java heap, in off-heap memory, or in a memory-mapped array indexed by node identifier — and writing intermediate features to a temporary directory that is sorted and read back during rendering. See the Planetiler documentation for the storage options, their memory implications, and the per-phase progress output.

Frequently Asked Questions

Why does a small country extract need so much address space?

Because the array node map is indexed by node identifier, and identifiers are assigned globally across the whole planet rather than per region. A recently created node in a small country can carry an identifier in the billions, so the array must be large enough to index it. Memory mapping is what makes this practical: the reservation is address space rather than resident memory, and the operating system pages in only the parts actually touched.

Should I give the process a very large heap?

No. Planetiler deliberately keeps its heavy storage off-heap and memory-mapped, so a large heap mostly takes memory away from the page cache that the node map and feature store rely on. A modest heap with plenty of free system memory for caching usually outperforms a configuration that hands most of the machine to the runtime.

How do I tell whether my profile is the bottleneck?

Read the phase timings. Profile cost lands entirely in the read phase, so a read phase taking well over half the total run is a strong signal that the per-element path is doing more than it should. The usual culprits are compiling patterns, allocating collections, or consulting a large lookup structure inside the callback — all of which are multiplied by the element count.

Can I run it against the full planet on one machine?

Yes, and that is what it is designed for, though it wants a substantial machine: a lot of memory, many cores and fast local storage for the temporary feature store. The build is vertical rather than distributed, which in practice is simpler and cheaper than a cluster for something that runs in hours. Size the temporary directory generously; it is the constraint that fails builds latest and most expensively.

Up one level: Planetiler & Tilemaker Workflows.