pyosmium vs pyrosm vs osmium-tool: Choosing the Right Parser Jump to heading#

Reaching for the wrong OSM reader is the quietest way to sink a pipeline. A team that prototypes on a city extract with pyrosm, loves how a .osm.pbf drops straight into a GeoDataFrame, and then points the same script at a continental file discovers the failure the hard way: the process climbs past available RAM and the kernel kills it mid-run, hours in, with nothing written. The inverse mistake is just as costly — hand-rolling a pyosmium streaming handler to clip a bounding box that osmium extract would have carved in a single fast pass, or looping a whole-file scan every time you need one object by id. None of these tools is wrong; each was built for a different access pattern, and the skill this guide teaches is matching the task and the scale to the reader before you write the first line. It belongs to the broader Parsing & Tag Normalization Workflows stage, where ingestion sets the ceiling on everything downstream.

The four candidates are not interchangeable layers of one stack; they occupy distinct points on the trade-off between memory and convenience. pyosmium is a thin Python binding over the libosmium C++ core that hands you elements one at a time through callback handlers, so a script’s resident memory stays flat whether the input is a district or the whole planet. pyrosm sits at the opposite corner: it reads a PBF straight into a GeoDataFrame so you can filter, join, and plot in familiar pandas idioms, paying for that convenience with a memory footprint proportional to the result set. osmium-tool is the command-line workhorse for moving whole files around — clipping, merging, deduplicating, applying change files, filtering by time — and it is almost always faster than reimplementing those operations in Python. osmx is the outlier: it expands an extract into an on-disk key-value store so you can fetch a single node, way, or relation by id in roughly constant time, which no streaming reader can do.

Prerequisites Jump to heading#

This guide compares readers of the same byte streams, so a working mental model of those streams pays off. Read the OSM XML vs PBF Comparison first: every tool here prefers .osm.pbf, and knowing why the binary format is 5–10× smaller and block-framed explains why streaming is even possible. The PBF File Structure Deep Dive then details the dense, delta-encoded blocks that libosmium — the engine under both pyosmium and osmium-tool — decodes one at a time. Finally, the Node-Way-Relation Data Model governs the single hardest cross-cutting concern in parser choice: a way is only geometry once its node references are resolved to coordinates, and each tool resolves those references differently. If you already have a concrete how-to in mind rather than a choice to make, the ingestion pattern in async PBF parsing with pyrosm shows one of these tools driven end to end.

Decision Matrix: Tool Against Axis Jump to heading#

The table below is the compressed form of the whole guide. Read down the column that names your binding constraint — memory, output shape, or scale — and the row it selects is your starting default. The prose after it explains the edges where the default flips.

Two axes dominate the choice. The first is whether you need the whole file transformed or a subset examined — the former is osmium-tool’s home, the latter splits between streaming (pyosmium) and materializing (pyrosm). The second is the shape of the output: if the next stage wants a GeoDataFrame, pyrosm saves you the reconstruction code; if it wants counts, a filtered file, or your own records, pyosmium keeps memory flat while you build exactly that. osmx answers a third, orthogonal question — “give me object 240109189 and nothing else” — that the scanning tools answer only by reading everything.

Step-by-Step: What Each Tool Looks Like in Practice Jump to heading#

Seeing the same intent expressed four ways makes the trade-offs concrete. Each sketch below is minimal but runnable against a real extract.

pyosmium — streaming, bounded memory Jump to heading#

pyosmium’s SimpleHandler dispatches a method per primitive type. Nothing accumulates unless you accumulate it, so counting every highway in a planet file costs the same memory as counting them in a town.

import logging

import osmium

logger = logging.getLogger(__name__)


class HighwayCounter(osmium.SimpleHandler):
    """Count highway ways without holding the file in memory."""

    def __init__(self) -> None:
        super().__init__()
        self.count: int = 0

    def way(self, w: osmium.osm.Way) -> None:
        if "highway" in w.tags:
            self.count += 1


handler = HighwayCounter()
handler.apply_file("region.osm.pbf")  # one callback per element; flat RSS
logger.info("highway ways: %d", handler.count)

To turn ways into geometry, pass locations=True to apply_file so pyosmium maintains a node-location store — the same lever that appears throughout memory-efficient chunk processing, where the store choice (flex_mem versus a disk-backed array) is what keeps a continental run inside its budget.

pyrosm — straight to a GeoDataFrame Jump to heading#

pyrosm trades memory for zero reconstruction code. One call yields a GeoDataFrame with geometry already assembled, ready for a spatial join or a plot.

from pyrosm import OSM

osm = OSM("city.osm.pbf")
roads = osm.get_network(network_type="driving")   # GeoDataFrame, geometry built
buildings = osm.get_buildings()                    # another full pass

print(roads[["highway", "maxspeed", "geometry"]].head())

Each get_* call re-reads the file, so extract the classes you need in as few calls as possible. This is the convenient front door for the analytical workflows in async PBF parsing with pyrosm — which exists precisely because that convenience becomes a bottleneck at scale and has to be parallelized behind a bounded queue.

osmium-tool — whole-file operations from the shell Jump to heading#

For anything that reshapes a file rather than analyzing its contents, the CLI is the fast path. These operations stream internally and run in C++, so they routinely beat a Python reimplementation by an order of magnitude.

# Clip to a bounding box (west,south,east,north) — the fastest extraction path
osmium extract -b 13.30,52.42,13.55,52.62 planet.osm.pbf -o berlin.osm.pbf

# Merge several extracts into one sorted file
osmium merge north.osm.pbf south.osm.pbf -o combined.osm.pbf

# Apply a change file to roll an extract forward
osmium apply-changes berlin.osm.pbf 4021.osc.gz -o berlin-updated.osm.pbf

# Keep only objects matching a tag filter
osmium tags-filter planet.osm.pbf w/highway -o highways.osm.pbf

The apply-changes and time-filter subcommands are the entry point to replication work; a pipeline that keeps an extract current is built on exactly these primitives, sequenced and scheduled.

osmx — random access by id Jump to heading#

osmx expands an extract once into an on-disk store, after which any object is a direct lookup. The expand step is the cost; every read afterward is cheap.

# One-time: expand the extract into an on-disk store (osmx ships its own CLI)
osmx expand region.osm.pbf region.osmx

import osmx

env = osmx.Environment("region.osmx")
with osmx.Transaction(env) as txn:
    locations = osmx.Locations(txn)
    node = locations.get(240109189)   # constant-time fetch by node id
    ways = osmx.Ways(txn)
    way = ways.get(4305005)           # random way lookup, no full scan

No scanning reader can serve that pattern: pyosmium and osmium-tool would each read the entire file to reach one object. When your workload is “resolve these 10,000 ids against a fixed extract,” the one-time expand pays for itself immediately.

Validation & Error-Handling Matrix Jump to heading#

Most parser regrets show up as one of a handful of symptoms. Each row pairs a symptom with the tool mismatch that usually causes it and the corrective move.

Performance & Scale Considerations Jump to heading#

The scaling behaviour of each reader is predictable once you know what it holds in memory. pyosmium holds one element plus, optionally, a node-location store whose size is a function of node count, not file size; that ceiling is fixed for a given planet, so a streaming filter’s peak RSS is nearly independent of how much work it does per element. pyrosm holds the reconstructed GeoDataFrame for the feature classes you requested, so its peak scales with the result, which is why a dense urban extract can cost more than a sparse continental one covering ten times the area. osmium-tool streams blocks and buffers only a window of them, so its footprint is bounded regardless of input size — its cost is I/O and CPU, not memory. osmx front-loads all cost into the expand phase and then memory-maps the store, keeping only touched pages resident.

The practical rule of thumb: below roughly a country-sized extract, pyrosm’s convenience usually wins and its memory cost is tolerable; above it, the reconstruction that made pyrosm pleasant becomes the thing that OOMs, and pyosmium streaming (or an osmium-tool pre-clip) takes over. But rules of thumb are no substitute for measurement on your extract and your machine — the sibling reference on benchmarking OSM parser memory and throughput gives a harness that records peak RSS and elements per second for pyosmium and pyrosm on the same file, so the crossover point stops being a guess.

Failure Modes & Gotchas Jump to heading#

• pyrosm’s memory is set by the result, not the file. A 200 MB dense city extract can produce a larger GeoDataFrame than a 2 GB sparse rural one. Size the machine against the expected output, not the input on disk.
• Streaming without a location store yields ways with no geometry. The way callback sees node references, not coordinates, unless pyosmium is resolving locations for you. Forgetting locations=True is the most common “why is my geometry empty” bug.
• osmium-tool’s default extract strategy is not always what you want. complete_ways and smart trade completeness at tile edges against speed and a second read pass; pick deliberately rather than accepting the default silently.
• osmx’s expand cost is real. Building the store reads and rewrites the whole extract to disk. It only pays off when you will do many random lookups; for a single scan it is pure overhead.
• Mixing tools means re-reading. Handing a pyrosm GeoDataFrame to a pyosmium handler, or vice versa, usually means the file is parsed twice. Choose one reader per pass and design the pass to emit everything the next stage needs.
• Version skew in the libosmium core. pyosmium and osmium-tool wrap libosmium but can ship different versions; when a PBF quirk parses in one and not the other, check the underlying library versions before blaming the file.

Integration Points: Where the Choice Feeds the Next Stage Jump to heading#

The parser you pick shapes the interface to everything downstream. A pyosmium streaming pass is the natural producer for the windowed pipelines in memory-efficient chunk processing: the handler emits records, the chunker batches them to a memory budget, and nothing ever materializes the whole file. A pyrosm read, by contrast, hands a GeoDataFrame directly to normalization and analysis, which is why the async pattern in async PBF parsing with pyrosm wraps it in workers and a bounded queue to keep that convenience affordable at scale. osmium-tool sits before the Python stage: it clips, merges, or rolls a file forward so the reader that follows sees a smaller, cleaner input.

import logging

import osmium

logger = logging.getLogger(__name__)


class TagWriter(osmium.SimpleHandler):
    """Stream ways and emit normalized rows the next stage consumes."""

    def __init__(self, sink) -> None:
        super().__init__()
        self.sink = sink
        self.rows: int = 0

    def way(self, w: osmium.osm.Way) -> None:
        if "highway" not in w.tags:
            return
        self.sink.append({
            "id": w.id,
            "highway": w.tags.get("highway"),
            "name": w.tags.get("name"),
        })
        self.rows += 1

That handler is deliberately a producer, not a consumer: it decides what to emit but leaves batching, back-pressure, and writing to the chunk-processing stage, keeping ingestion and transformation cleanly separated.

Explore the Benchmarking Reference in Depth Jump to heading#

• Benchmarking OSM Parser Memory and Throughput — a runnable harness that measures peak RSS and elements-per-second for pyosmium and pyrosm on the same extract, so the crossover between streaming and materializing is evidence, not intuition.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Async PBF Parsing with Pyrosm — the how-to that scales pyrosm’s convenience with processes and a bounded queue once a single pass is too slow.
• Memory-Efficient Chunk Processing — the windowed, budget-bounded discipline a pyosmium streaming pass feeds.
• OSM XML vs PBF Comparison — the format decision that precedes the parser decision.
• PBF File Structure Deep Dive — the block-framed encoding that makes streaming and whole-file transforms possible.
• Node-Way-Relation Data Model — the reference-resolution rules every reader handles differently.
• Benchmarking OSM Parser Memory and Throughput — turn the guidance here into numbers for your own extract.

This guide is part of Parsing & Tag Normalization Workflows — return there to follow the data from ingestion through normalization, error triage, and routing-graph conversion.