Chaining osmium-tool Commands in a Shell Pipeline

Three osmium commands in sequence is the most common OSM pipeline there is, and the naive form writes two intermediate gigabyte files that nothing ever reads twice. Most of the time it does not need to.

Prerequisites

Conceptual minimum

Osmium commands read a file and write a file, and most of them can read from standard input and write to standard output when told the format explicitly. That makes piping possible, and piping avoids the intermediate write entirely.

Three rules govern the composition.

Cheapest reduction first. Each pass costs roughly its input size, so a pass that removes ninety percent of the data should run before one that removes ten. Spatial cuts are usually the cheapest large reduction, tag filters next, and anything reference-completing last.

Not every command streams. Commands that need random access — sorting, reference completion over a whole file, some export modes — must materialise. Piping into one of those gains nothing and can cost, because it may buffer the entire input anyway.

The format must be declared on a pipe. A file name carries its format; standard input does not. Omitting the format flag produces an immediate and clear failure, which is the one mistake here that announces itself.

The trade against piping is restartability. A pipeline of four piped commands that fails at the third restarts from the beginning; one with a materialised intermediate after the expensive first pass restarts from there. On a pipeline that takes an hour, that matters more than the disk it costs.

When to pipe and when to materialise between osmium passes A grid of four situations against whether to pipe or to write an intermediate file. A cheap pass feeding another cheap pass should be piped, since the intermediate would cost more than it saves. An expensive pass whose output several later passes consume should be materialised, so the expensive work happens once. A pass that needs random access cannot be piped into and must read a file. A long pipeline where a late failure is likely benefits from one materialised checkpoint after the most expensive stage. Four situations, two answers Do this Because Cheap then cheap pipe the file costs more Expensive, reused materialise do it once Needs random access materialise cannot stream Long, failure-prone one checkpoint restart from there The last row is the one experience teaches: a pipeline that must restart from zero after fifty minutes is worse than one intermediate file.
Piping is the default and materialising is the deliberate exception, rather than the other way round.

Runnable solution

bash 
#!/usr/bin/env bash
# Compose osmium passes without writing every intermediate to disk.
set -euo pipefail

EXTRACT="${1:?usage: pipeline.sh <extract.osm.pbf>}"
POLY="${2:?usage: pipeline.sh <extract.osm.pbf> <boundary.poly>}"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT

# --- The naive form, for comparison: two intermediate files nobody reads twice.
naive () {
  osmium extract --polygon "$POLY" -o "$WORK/a.osm.pbf" "$EXTRACT"
  osmium tags-filter -o "$WORK/b.osm.pbf" "$WORK/a.osm.pbf" \
    w/highway r/route=road
  osmium export -o roads.geojsonseq --output-format=geojsonseq \
    "$WORK/b.osm.pbf"
}

# --- Piped: nothing between the passes touches disk.
#     -F declares the INPUT format; -f declares the OUTPUT format. On a pipe
#     neither can be inferred from a file name, so both must be explicit.
piped () {
  osmium extract --polygon "$POLY" -f pbf -o - "$EXTRACT" \
  | osmium tags-filter -F pbf -f pbf -o - - w/highway r/route=road \
  | osmium export -F pbf --output-format=geojsonseq -o roads.geojsonseq -
}

# --- Checkpointed: one intermediate after the expensive spatial cut, because a
#     failure later should not repeat an hour of work.
checkpointed () {
  local cut="$WORK/cut.osm.pbf"
  if [ ! -s "$cut" ]; then
    osmium extract --polygon "$POLY" -o "$cut" "$EXTRACT"
  fi
  osmium tags-filter -f pbf -o - "$cut" w/highway r/route=road \
  | osmium export -F pbf --output-format=geojsonseq -o roads.geojsonseq -
}

# Choose by input size: below a few hundred megabytes the checkpoint is not
# worth the disk, above it a restart from zero is not worth the time.
SIZE_MB=$(( $(stat -c%s "$EXTRACT") / 1024 / 1024 ))
if [ "$SIZE_MB" -gt 300 ]; then
  echo "input is ${SIZE_MB} MB: checkpointing after the spatial cut" >&2
  checkpointed
else
  echo "input is ${SIZE_MB} MB: fully piped" >&2
  piped
fi

wc -l < roads.geojsonseq
bash 
#!/usr/bin/env bash
# Measure whether the ordering you chose is the right one.
set -euo pipefail
EXTRACT="$1"; POLY="$2"

time_pass () { local label="$1"; shift; local start=$SECONDS
  "$@" >/dev/null 2>&1; echo "${label}: $(( SECONDS - start ))s" >&2; }

# Spatial first, then tags — the usual right order.
time_pass "spatial-then-tags" bash -c "
  osmium extract --polygon '$POLY' -f pbf -o - '$EXTRACT' \
  | osmium tags-filter -F pbf -f pbf -o /dev/null - w/highway"

# Tags first, then spatial — reads and rewrites the whole region first.
time_pass "tags-then-spatial" bash -c "
  osmium tags-filter -f pbf -o - '$EXTRACT' w/highway \
  | osmium extract --polygon '$POLY' -F pbf -f pbf -o /dev/null -"

Step-by-step walkthrough

  1. Declare both formats on a pipe. The input format flag and the output format flag are both needed, because neither end of a pipe carries a file name to infer from.
  2. Use a bare dash for both ends. Osmium reads standard input and writes standard output when given a dash, which is what makes the composition possible at all.
  3. Put the spatial cut first. It is usually the largest reduction and the cheapest to compute, so every later pass reads a far smaller stream.
  4. Materialise after the expensive pass, not before. The point of a checkpoint is to avoid repeating the costly work, so it belongs immediately after it.
  5. Make the checkpoint conditional. Testing for a non-empty existing file makes the script restartable without any further bookkeeping.
  6. Choose the shape from the input size. Below a few hundred megabytes the checkpoint costs more in disk and complexity than it saves; above it, a restart from zero is the expensive outcome.
  7. Measure both orderings once. The rule that spatial comes first is reliable and not universal — a tag filter that removes ninety-nine percent of a file can legitimately come first — and a single timed comparison settles it for your data.
Bytes written to disk by three compositions of the same three passes Four measurements over the same country extract. The naive form writes two full intermediate files plus the output, totalling several times the final result. The piped form writes only the output, since nothing between the passes touches disk. The checkpointed form writes one intermediate after the spatial cut plus the output. A note observes that the piped form is fastest and the checkpointed form is the one that survives a failure at the last pass. Disk written by three compositions Naive: two intermediates baseline Piped: output only about 11% Checkpointed: one file about 38% The final output alone what you wanted The checkpointed form buys restartability with disk, which on a pipeline measured in hours is usually the right purchase.
The naive form's extra writes are pure cost: neither intermediate is ever read more than once.
Three compositions of the same three passes, and what each optimises for Three panels. The naive form runs each pass against a file and writes two intermediates nobody reads twice, which optimises for nothing but is what people write first. The fully piped form passes data between commands in memory and writes only the final output, which optimises for speed and disk and restarts from zero on any failure. The checkpointed form materialises one intermediate immediately after the expensive stage, which costs disk and makes a late failure cheap to recover from. Three compositions, three different optimisations Naive A file between each pass Two intermediates written Neither read twice Optimises for nothing What people write first Fully piped Nothing touches disk Only the output written Fastest and smallest Restarts from zero Right for small inputs Checkpointed One file after the costly pass Disk for restartability Late failure is cheap Skips work on re-run Right for large inputs The choice between the second and third is about input size and failure probability rather than about elegance.
Nobody should be writing the first form, and almost everybody does until they measure the disk it costs.

Verification

  • Piped and naive agree. Both forms must produce byte-identical output; a difference means a format flag is wrong.
  • The pipeline restarts. Delete the output, leave the checkpoint, and re-run; it should skip the expensive pass.
  • Ordering is measured, not assumed. Time both orderings once on real data and keep the faster.
  • No intermediate survives. After a successful run the temporary directory should be empty except for any deliberate checkpoint.
  • Failure propagates. Break a middle command and confirm the script fails rather than producing truncated output.

Common errors and fixes

Symptom Root cause One-line fix
Osmium refuses to read a pipe Input format not declared Pass the input format flag explicitly
Osmium refuses to write a pipe Output format not declared Pass the output format flag explicitly
Pipeline slower than the naive form Tag filter placed before the spatial cut Put the largest cheap reduction first
Failure produces truncated output Pipeline errors not propagated Enable pipefail and errexit in the shell
Restart repeats an hour of work No checkpoint after the expensive pass Materialise once, immediately after it
Disk fills during the run Every pass materialised Pipe the cheap passes together
A command buffers the whole input That command needs random access Materialise before it rather than piping into it

Specification reference

Most osmium-tool commands accept a dash as an input or output file name to read standard input or write standard output, and require the format to be specified explicitly when doing so, since it cannot be inferred from a file name. Some commands require random access to their input and cannot read from a stream. See the osmium-tool manual for which commands stream and the format flags each accepts.

Frequently Asked Questions

Why must the format be declared on a pipe?

Because osmium infers the format from the file name’s extension, and a pipe has no name. Reading standard input without a declared format fails immediately with a clear message, which makes this the one mistake in this area that costs nothing to discover. Both ends need it independently: the input format for what is arriving and the output format for what is leaving.

Should the spatial cut always come first?

Almost always, because it is usually both the largest reduction and the cheapest to compute, so every later pass reads a much smaller stream. The exception is a tag filter so selective that it removes nearly everything — filtering a planet file to a rare key, for instance — where doing that first can win. One timed comparison settles it for your data, and the answer rarely changes afterwards.

When is an intermediate file worth writing?

When the pass before it is expensive and the passes after it might fail. A pipeline that takes an hour and restarts from zero after a failure in the last command has wasted that hour; one with a checkpoint after the expensive stage restarts in minutes. Below a few hundred megabytes of input the calculation reverses, because the expensive stage is not expensive enough to be worth protecting.

Do all osmium commands stream?

No. Commands needing random access over the whole input — sorting, some reference-completing modes — must read a file, and piping into them either fails or silently buffers the entire stream, which defeats the purpose. Checking the manual for the command you are about to pipe into is quicker than discovering it from a memory spike.

Up one level: Choosing an OSM Parser: pyosmium, pyrosm or osmium-tool.