Measuring OSM XML vs PBF Parse Throughput Jump to heading#

Produce a parse-throughput comparison that someone else can reproduce, and that measures the parser rather than your page cache.

Prerequisites Jump to heading#

• osmium-tool, and Python 3.10+ with osmium if benchmarking the library path
• The same extract in each format under test — convert it as in Converting OSM XML to PBF with osmium
• Root, or sudo, if you intend to drop the page cache between runs
• A country-sized extract — a city extract is dominated by process startup

Conceptual minimum Jump to heading#

A benchmark that compares two encodings has to hold everything else still, and there are more things to hold still than there appear to be.

The most common mistake is the page cache. Reading a file once loads it into memory; the second read never touches the disk. Benchmark two formats in sequence without clearing that and the second one is measured against a warm cache, which flatters it by however much the I/O was costing.

The second most common is comparing different work. Counting objects is not the same job as building geometries, and a benchmark where one parser counts while the other assembles ways measures the difference in job, not in format.

Runnable solution Jump to heading#

#!/usr/bin/env bash
# bench.sh — compare parse throughput across encodings of the same extract.
# Fixes the work (count objects), controls the cache, repeats, records context.
set -euo pipefail

RUNS="${RUNS:-7}"
FILES=("$@")
: "${FILES[0]:?usage: bench.sh file1 [file2 ...]}"

drop_caches() {
  sync
  if [[ -w /proc/sys/vm/drop_caches ]]; then
    echo 3 > /proc/sys/vm/drop_caches
  else
    sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  fi
}

context() {
  echo "## context"
  echo "date:        $(date -Is)"
  echo "cpu:         $(grep -m1 'model name' /proc/cpuinfo | cut -d: -f2 | xargs)"
  echo "cores:       $(nproc)"
  echo "governor:    $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo n/a)"
  echo "osmium:      $(osmium --version | head -1)"
  echo "kernel:      $(uname -r)"
  for f in "${FILES[@]}"; do
    echo "input:       $f  $(stat -c%s "$f") bytes  sha256=$(sha256sum "$f" | cut -c1-16)…"
  done
}

context
echo
printf '%-28s %10s %10s %10s %12s\n' file median_s min_s max_s objects

for f in "${FILES[@]}"; do
  # Fix the work: the same command, producing the same counts, for every format.
  objects=$(osmium fileinfo --extended --get data.count.nodes "$f")
  times=()
  # One discarded warm-up run: lets the CPU governor ramp before we measure.
  drop_caches; osmium fileinfo --extended "$f" >/dev/null
  for _ in $(seq "$RUNS"); do
    drop_caches
    start=$(date +%s.%N)
    osmium fileinfo --extended "$f" >/dev/null
    end=$(date +%s.%N)
    times+=("$(echo "$end - $start" | bc)")
  done
  printf '%s\n' "${times[@]}" | sort -n | awk -v f="$f" -v o="$objects" '
    {a[NR]=$1}
    END {printf "%-28s %10.2f %10.2f %10.2f %12d\n", f, a[int((NR+1)/2)], a[1], a[NR], o}'
done

For the library path, where the interesting number is objects per second rather than wall-clock:

#!/usr/bin/env python3
"""Measure pyosmium parse throughput over one file, with the same work per format."""
from __future__ import annotations

import logging
import statistics
import subprocess
import time
from pathlib import Path

import osmium

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
logger = logging.getLogger(__name__)


class CountingHandler(osmium.SimpleHandler):
    """The fixed unit of work: touch every object, build nothing."""

    def __init__(self) -> None:
        super().__init__()
        self.nodes = self.ways = self.relations = 0

    def node(self, n) -> None:
        self.nodes += 1

    def way(self, w) -> None:
        self.ways += 1

    def relation(self, r) -> None:
        self.relations += 1

    @property
    def total(self) -> int:
        return self.nodes + self.ways + self.relations


def drop_caches() -> None:
    subprocess.run(["sync"], check=True)
    subprocess.run(["sudo", "sh", "-c", "echo 3 > /proc/sys/vm/drop_caches"], check=True)


def bench(path: Path, runs: int = 7) -> dict[str, float]:
    durations: list[float] = []
    counts: set[int] = set()
    drop_caches()
    CountingHandler().apply_file(str(path))        # discarded warm-up
    for _ in range(runs):
        drop_caches()
        handler = CountingHandler()
        started = time.perf_counter()
        handler.apply_file(str(path))
        durations.append(time.perf_counter() - started)
        counts.add(handler.total)
    if len(counts) != 1:
        raise RuntimeError(f"{path}: object count varied across runs: {counts}")
    median = statistics.median(durations)
    result = {
        "median_s": median,
        "min_s": min(durations),
        "max_s": max(durations),
        "objects": counts.pop(),
        "objects_per_s": counts_per_s if (counts_per_s := 0) else 0,
    }
    result["objects_per_s"] = result["objects"] / median
    logger.info("%-24s median %6.2f s  spread %5.2f s  %8.0f obj/s",
                path.name, median, max(durations) - min(durations), result["objects_per_s"])
    return result

Step-by-step walkthrough Jump to heading#

drop_caches runs before every timed run, not once at the start. Dropping once and then running seven times measures one cold run and six warm ones, and the median lands on a warm number.

The discarded warm-up run exists for the CPU governor. On a machine with ondemand or schedutil scaling, the first CPU-heavy work after an idle period runs at a lower clock while the governor ramps, which systematically penalises whichever format is measured first.

counts is a set, and a run where it ends up with more than one member is a failed benchmark rather than an interesting result. Object counts must be identical across runs and across formats; if they are not, the two files are not the same data and nothing they are compared on means anything.

Reporting the median plus the spread rather than the best run is the honest choice. The best run is the one with the least interference, which is not the number anyone will reproduce.

Verification Jump to heading#

Prove the controls are working before trusting the numbers.

Check the cache drop actually happens — if it silently fails, every run after the first is warm:

free -m | awk '/Mem:/ {print "cached before:", $6}'
sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
free -m | awk '/Mem:/ {print "cached after: ", $6}'

Check the spread. A median of 9.2 seconds with a min of 9.1 and a max of 9.4 is a controlled measurement; a max of 31 means something else was running and the median is not trustworthy.

Check the work is identical by comparing object counts across formats:

for f in ireland.osm ireland.osm.bz2 ireland.osm.pbf; do
  echo -n "$f  "; osmium fileinfo --extended --get data.count.nodes "$f"
done

All three must print the same number. Any difference means the files are not the same extract, and the benchmark is comparing different data.

Common errors and fixes Jump to heading#

Frequently Asked Questions Jump to heading#

Reading the result honestly Jump to heading#

A benchmark answers exactly the question it measured, and the temptation is to report it as answering a broader one. Three limits are worth stating alongside any numbers you publish.

The measurement is of one machine’s storage and CPU. A parse that is I/O-bound on a spinning disk is CPU-bound on NVMe, which changes the ranking between compressed and uncompressed formats — the bzip2 result above would look considerably better on slow storage, because the bytes it saves would cost more to read.

It is also of one workload. Counting objects touches every record and builds nothing; a job that assembles way geometries spends most of its time on reference resolution rather than on decoding, and the format difference shrinks against that larger constant. Benchmark the shape of work your pipeline actually does before letting a format comparison drive a design decision.

Finally, it is of one file. Extracts differ in tag density, in how many relations they carry and in how well sorted they are, and all three affect decode speed. A ratio measured on a European country extract transfers reasonably to another European country and less well to a dense urban extract or a sparse rural one.

None of this makes the numbers less useful. It makes them a measurement rather than a fact, which is the distinction a published benchmark should be careful to preserve.

Specification reference Jump to heading#

osmium fileinfo --extended performs a full pass over the file, decoding every object to compute counts and the bounding box, and is therefore a reasonable fixed unit of parsing work across formats. --get data.count.nodes prints a single value suitable for scripting.

Related Jump to heading#

• OSM XML vs PBF Comparison — the topic these numbers support.
• Converting OSM XML to PBF with osmium — producing the files under test.
• Benchmarking OSM Parser Memory and Throughput — the same discipline applied to libraries rather than formats.
• Tuning pyrosm Worker Count for PBF Parsing — what happens once you add cores.
• PBF File Structure Deep Dive — why PBF decodes so much faster.

Up one level: OSM XML vs PBF Comparison.