Splitting a Planet File into Regional Extracts

Produce a dozen regional .osm.pbf files from one parent extract in a single pass, instead of reading the parent once per region and turning a twenty-minute job into a four-hour one.

Prerequisites

Conceptual minimum

osmium extract accepts a --config file listing many outputs. It reads the parent once and writes every output as it goes, which is the entire point: the extraction arithmetic is cheap and reading tens of gigabytes off disk is not.

Bytes read and wall-clock for three ways of cutting twelve regions A bar chart. Twelve separate osmium extract runs read 341 gigabytes and take four hours twelve minutes. Three batched runs of four regions each read 85 gigabytes and take one hour four minutes. One configured run reads 28.4 gigabytes and takes twenty-two minutes. Reads scale with runs, not with outputs cutting 12 regions from a 28.4 GB parent 12 separate runs 341 GB read · 4 h 12 m 3 batched runs of 4 85 GB read · 1 h 04 m 1 configured run 28.4 GB read · 22 m The extraction work is identical in all three. The only thing that changes is how many times the parent is read off disk.
One pass, many outputs. The configured run does the same extraction work and reads the parent once instead of twelve times.

The saving is linear in the number of regions, and it is the difference between a job that fits in a nightly window and one that does not. It also removes a subtler cost — twelve runs against a network-mounted parent transfer the file twelve times.

The config format

json 
{
  "directory": "/data/extracts",
  "extracts": [
    {
      "output": "ireland.osm.pbf",
      "description": "Republic of Ireland",
      "polygon": { "file_name": "/data/boundaries/ireland.poly", "file_type": "poly" }
    },
    {
      "output": "scotland.osm.pbf",
      "polygon": { "file_name": "/data/boundaries/scotland.geojson", "file_type": "geojson" }
    },
    {
      "output": "greater-london.osm.pbf",
      "bbox": [-0.51, 51.28, 0.33, 51.69]
    }
  ]
}

directory is the shared output directory; each output is a filename within it. Region geometry comes from one of three keys, and the choice is the same one discussed in the parent topic, Extract Clipping & Boundary Polygons.

Three region descriptions available in an osmium extract config Three panels. A bbox entry takes west, south, east and north, is the cheapest per-node test, suits tiles and test cuts, is wrong for border-shaped regions and needs no separate file. A poly entry names a file with type poly, the OSM-native boundary format, suited to countries and regions, and should be version-controlled. A geojson entry names a file with type geojson, comes straight out of GIS tools, suits generated boundaries, and must be a single Feature rather than a collection. Three ways to describe a region in the config, and when each fits bbox "bbox": [w, s, e, n] Cheapest test per node Right for tiles and test cuts Wrong for anything border-shaped No file to keep in sync poly "polygon": {"file_name": …, "file_type": "poly"} The OSM-native boundary format Right for countries and regions Keep the .poly under version control geojson "polygon": {"file_name": …, "file_type": "geojson"} Comes straight out of GIS tools Right when boundaries are generated One Feature, not a collection A FeatureCollection with several features is read as several regions, not as one multi-part boundary — a distinction that silently changes what you get.
The FeatureCollection caveat is the one that bites: several features are read as several regions, which quietly changes the shape of the output.

Two config details matter more than they look. The strategy is set once on the command line and applies to every extract in the run, so a batch that mixes regions needing smart with regions where complete_ways would do must either use smart throughout or be split into two runs. And description is not decoration: it is written into the output file’s header, which is the only place the reason for a cut survives.

Runnable solution

python 
#!/usr/bin/env python3
"""Cut many regions from one parent in a single osmium pass, then verify each output."""
from __future__ import annotations

import json
import logging
import shutil
import subprocess
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
logger = logging.getLogger(__name__)

# Rough output size as a fraction of the parent, used only for the disk pre-flight.
SIZE_FRACTION = 0.05


def preflight(config: dict, parent: Path) -> None:
    """Refuse to start a run that cannot finish for want of disk."""
    out_dir = Path(config["directory"])
    out_dir.mkdir(parents=True, exist_ok=True)
    n = len(config["extracts"])
    need = int(parent.stat().st_size * SIZE_FRACTION * n * 1.1)
    free = shutil.disk_usage(out_dir).free
    logger.info("pre-flight: %d region(s), ~%.1f GB needed, %.1f GB free",
                n, need / 1e9, free / 1e9)
    if free < need:
        raise RuntimeError(f"insufficient disk: need ~{need/1e9:.1f} GB, have {free/1e9:.1f} GB")
    for e in config["extracts"]:
        poly = e.get("polygon", {}).get("file_name")
        if poly and not Path(poly).exists():
            raise FileNotFoundError(f"boundary missing for {e['output']}: {poly}")


def split(config_path: Path, parent: Path, strategy: str = "smart") -> None:
    subprocess.run(
        ["osmium", "extract", "--config", str(config_path),
         "--strategy", strategy, "--overwrite", str(parent)],
        check=True,
    )


def verify_all(config: dict) -> None:
    """Every output, not just the last one — a batch run hides a single bad boundary."""
    out_dir = Path(config["directory"])
    failures: list[str] = []
    for entry in config["extracts"]:
        path = out_dir / entry["output"]
        if not path.exists():
            failures.append(f"{entry['output']}: not written")
            continue
        info = json.loads(subprocess.run(
            ["osmium", "fileinfo", "--extended", "--json", str(path)],
            capture_output=True, text=True, check=True,
        ).stdout)
        nodes = info["data"]["count"]["nodes"]
        if nodes == 0:
            failures.append(f"{entry['output']}: zero nodes — check the boundary")
        else:
            logger.info("%-28s %12d nodes  %8.1f MB",
                        entry["output"], nodes, path.stat().st_size / 1e6)
    if failures:
        raise RuntimeError("verification failed:\n  " + "\n  ".join(failures))


if __name__ == "__main__":
    cfg_path = Path("extracts.json")
    cfg = json.loads(cfg_path.read_text())
    parent_file = Path("planet-latest.osm.pbf")
    preflight(cfg, parent_file)
    split(cfg_path, parent_file)
    verify_all(cfg)

Step-by-step walkthrough

preflight does two things that turn a class of overnight failures into an immediate one. It estimates total output size and compares it against free disk, and it checks that every boundary file named in the config exists. Both failures otherwise appear hours into the run, after most of the work is done, and osmium will have written partial outputs by then.

split is one subprocess call. --strategy on the command line applies to every entry; --overwrite is needed for a repeatable job, since osmium will not clobber an existing output.

verify_all is the part most implementations omit. A batch run exits zero as a whole, so a region whose boundary was written in the wrong axis order produces an empty file that nothing complains about. Iterating every declared output and asserting a non-zero node count catches it before the next stage consumes it.

Four failure modes of a batched extract run A grid of four failures. Disk filling mid-run leaves partial outputs with no error on the remainder, fixed by pre-flighting the total size and writing to a scratch volume. One wrong boundary produces a single empty file among good ones, fixed by verifying every output rather than the last. Duplicate output names cause files to overwrite each other, fixed by ensuring unique names since the directory is shared. An out-of-memory kill after hours is fixed by fewer regions per run, because the smart strategy holds an identifier set per region. Where a multi-extract run actually fails symptom fix disk fills mid-run partial outputs, no error on the rest pre-flight the total, write to a scratch volume one boundary is wrong one empty file among eleven good ones verify every output, not the last one outputs overwrite each other fewer files than regions unique output names; directory is shared run killed by the OOM killer no output at all after hours fewer regions per run; smart holds an id set per region The second row is the one to design for: a batch run reports success as a whole, so a single bad boundary hides among good outputs.
A batch run succeeds or fails as a whole, which is precisely why per-output verification has to be explicit.

Verification

Expect one log line per region with a plausible node count and file size. Two patterns in that output indicate a problem: a region whose node count is orders of magnitude away from its neighbours of similar area, and a file size close to zero. Both mean a bad boundary rather than a sparse region.

bash 
ls -la /data/extracts/*.osm.pbf | awk '{print $5, $9}' | sort -n | head -3

If the smallest outputs are a few hundred bytes, those boundaries are wrong. A genuinely small region still produces a file in the megabytes.

Common errors and fixes

Symptom Root cause Fix
Some outputs missing, no error Disk filled part-way through Pre-flight the total; write to a dedicated volume
One empty file among many good ones That boundary has swapped axes Verify every output; fix the boundary
Config file error at startup Trailing comma or a file_type typo Validate the JSON before invoking osmium
Fewer files than regions Two entries share an output name Make output names unique
Killed after hours with no output smart holds an id set per region Split into several runs of fewer regions
Every output is tiny Parent does not cover the boundaries Check the parent’s own bbox first

Specification reference

osmium extract --config FILE reads a JSON document with an optional directory and a list of extracts. Each entry needs an output and exactly one of bbox, polygon or multipolygon. The input is read once and all extracts are written concurrently; --strategy applies to the whole run.

Budgeting a split

Two resources bound a multi-region run, and they bind in opposite directions, which is why a run that fits on one machine fails on another with more disk.

Disk is the easy one. Regional extracts add up to well under the parent — a planet file split into every country produces roughly 55 to 65 percent of the planet’s own size, because the strategies duplicate only the objects near boundaries. Estimating five percent of the parent per country-sized region, as the pre-flight above does, is deliberately generous and errs toward refusing a run that would in fact have fitted. That is the right direction to err: a refusal costs a minute, and a disk that fills two hours in costs the run.

Memory is the harder one because it does not scale with output size. Under the smart strategy each region under construction holds a set of the object identifiers it has decided to keep, and that set is sized by the number of objects near that region’s boundary. A compact region with a short boundary is cheap; a long coastal country, or an administrative area with many enclaves, is not. Twenty regions cut concurrently from a continent will sit comfortably in tens of gigabytes; the same twenty cut from the planet will not, because every identifier set is drawn from a hundred times as many candidate objects.

The practical consequence is a two-stage split for anything planet-scale. Cut the planet into continents once, in a run of six or seven regions, then cut countries from their continent in separate runs. The parent is read eight times rather than once, but each read is of a much smaller file and no run holds more than a handful of identifier sets. On the measurements above that shape runs in about ninety minutes for a full country-level split of the planet, against a single-stage run that does not complete at all on a 64 GB machine.

One further economy is worth knowing. If the same regions are cut every week, keeping the continent-level intermediates means the weekly job never touches the planet file: only the continents that actually changed need re-cutting from a fresh planet, and everything downstream of an unchanged continent can be skipped entirely. That turns a full split into an incremental one without any additional tooling.

Frequently Asked Questions

How many regions can go in one run?

Disk and memory decide it, not the tool. Every region being cut concurrently holds its own identifier set under smart, so budget roughly a gigabyte per region on a continent-sized parent and rather more on a planet. Twelve to twenty regions in a run is comfortable on a machine with 64 GB; beyond that, splitting into several runs still reads the parent far fewer times than cutting each region separately.

Can I cut overlapping regions in the same run?

Yes. The regions are independent — an object inside two boundaries is written to both outputs — so overlapping extracts such as a country and one of its cities cost no more than disjoint ones. This is often the cheapest way to produce a nested set of extracts, because the alternative is cutting the city out of the country afterwards, which reads the country file again.

Should I cut the regions I need from the planet, or from continents first?

From continents, if you need more than a handful of regions per continent. A two-stage split — planet to continents once, then continents to countries — reads the planet once and each continent once, whereas cutting fifty countries directly from the planet in one configured run also reads the planet once but holds fifty identifier sets in memory at the same time. The two-stage version trades a little extra disk for a much lower memory ceiling.

Does a batched run write outputs incrementally or at the end?

Incrementally, as the parent is read. That is why a run interrupted by a full disk leaves several complete-looking files behind: they are not truncated, they are simply missing everything that came after the failure point. Nothing in the file marks it as partial, which is exactly why the verification step compares node counts rather than trusting the exit code.

Up one level: Extract Clipping & Boundary Polygons.