Benchmarking OSM Parser Memory and Throughput Jump to heading#

Measure the peak resident memory and parse throughput of pyosmium and pyrosm on the same extract, so the decision between streaming and materializing rests on numbers from your data and your machine rather than a rule of thumb.

Prerequisites Jump to heading#

Tick each box before running the harness; a skipped one is the usual reason two parsers appear to use identical memory or wildly different element counts.

• Python 3.10+ (for the dataclass, | unions, and structural typing used below).
• pyosmium installed (pip install osmium) — the streaming reader described in pyosmium vs pyrosm vs osmium-tool.
• pyrosm installed (pip install pyrosm) for the GeoDataFrame read side of the comparison.
• Optional: psutil (pip install psutil) if you want a live RSS sample rather than only the post-run high-water mark.
• One .osm.pbf extract on local disk — a city or small region so both parsers finish quickly and pyrosm does not OOM.
• A Linux or macOS host — the code normalizes the ru_maxrss unit difference between the two.

Conceptual minimum Jump to heading#

Two numbers decide a parser: how much memory it needs at its worst moment, and how fast it turns bytes into elements. The worst-moment figure is peak resident set size, and the operating system already tracks it for you — resource.getrusage(resource.RUSAGE_SELF).ru_maxrss returns a high-water mark that only ever climbs during a process’s life. That property is a gift and a trap: because it never decreases, running pyosmium and pyrosm in the same process would report the maximum of the two, hiding the very difference you are trying to see. The fix is to run each parser in its own fresh process and read the high-water mark there. Throughput is simpler — wrap the parse in time.perf_counter(), count the elements consumed, and divide. The one subtlety is fairness: the first parser warms the operating system’s page cache, so the second reads from RAM instead of disk and looks faster than it is. This page is the empirical companion to the decision guide in choosing an OSM parser, which explains why the two readers scale differently; here we quantify it.

Runnable solution Jump to heading#

The harness runs each parser in a spawned child process, times the parse, and reports peak RSS and elements per second side by side. It targets pyosmium and pyrosm on Python 3.10+.

from __future__ import annotations

import logging
import multiprocessing as mp
import platform
import resource
import sys
import time
from dataclasses import dataclass

import osmium

logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("osm.parser_benchmark")


def maxrss_bytes() -> int:
    """Peak RSS of *this* process, normalized to bytes.

    ru_maxrss is a high-water mark that only ever increases, so it is only
    meaningful when read in a process that ran exactly one parser.
    """
    raw = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    # Linux reports kibibytes; macOS and the BSDs report bytes.
    return raw * 1024 if platform.system() == "Linux" else raw


@dataclass
class Result:
    parser: str
    elements: int
    seconds: float
    peak_rss_bytes: int

    @property
    def throughput(self) -> float:
        return self.elements / self.seconds if self.seconds else 0.0


class _ElementCounter(osmium.SimpleHandler):
    """Count every primitive so both parsers report comparable work."""

    def __init__(self) -> None:
        super().__init__()
        self.n: int = 0

    def node(self, _n: osmium.osm.Node) -> None:
        self.n += 1

    def way(self, _w: osmium.osm.Way) -> None:
        self.n += 1

    def relation(self, _r: osmium.osm.Relation) -> None:
        self.n += 1


def _run_pyosmium(path: str, q: mp.Queue) -> None:
    handler = _ElementCounter()
    t0 = time.perf_counter()
    handler.apply_file(path)
    elapsed = time.perf_counter() - t0
    q.put(Result("pyosmium", handler.n, elapsed, maxrss_bytes()))


def _run_pyrosm(path: str, q: mp.Queue) -> None:
    from pyrosm import OSM  # imported in the child so spawn stays cheap

    t0 = time.perf_counter()
    osm = OSM(path)
    nodes, edges = osm.get_network(nodes=True, network_type="all")
    count = len(nodes) + len(edges)
    elapsed = time.perf_counter() - t0
    q.put(Result("pyrosm", count, elapsed, maxrss_bytes()))


def benchmark(path: str) -> list[Result]:
    """Run each parser in a fresh process so ru_maxrss reflects one parser."""
    ctx = mp.get_context("spawn")  # a clean interpreter → clean high-water mark
    results: list[Result] = []
    for target in (_run_pyosmium, _run_pyrosm):
        q: mp.Queue = ctx.Queue()
        proc = ctx.Process(target=target, args=(path, q))
        proc.start()
        results.append(q.get())
        proc.join()
    return results


def print_table(results: list[Result]) -> None:
    logger.info("%-10s %13s %9s %14s %14s", "parser", "elements", "sec",
                "peak RSS (MB)", "elem/sec")
    for r in results:
        logger.info("%-10s %13d %9.2f %14.1f %14.0f", r.parser, r.elements,
                    r.seconds, r.peak_rss_bytes / 1e6, r.throughput)


if __name__ == "__main__":
    print_table(benchmark(sys.argv[1]))

Step-by-step walkthrough Jump to heading#

1. Normalize the RSS unit — maxrss_bytes() multiplies by 1024 on Linux, where ru_maxrss is kibibytes, and leaves it alone on macOS/BSD, where it is already bytes. Skip this and your peaks are off by a factor of 1024.
2. Count comparable work — _ElementCounter tallies nodes, ways, and relations, so “elements” means the same thing for both parsers and the throughput divisor is fair.
3. Time only the parse — time.perf_counter() brackets the apply_file call (and pyrosm’s get_network), excluding import and process spin-up so the rate reflects parsing, not startup.
4. Isolate each parser in its own process — mp.get_context("spawn") starts a fresh interpreter per run, so the high-water mark read by maxrss_bytes() belongs to exactly one parser. This is the single most important step: measure both in one process and you get the max of the two, not each.
5. Import pyrosm lazily — the from pyrosm import OSM lives inside _run_pyrosm, so the spawned pyosmium child never pays for pyrosm’s heavy import and vice versa.
6. Report both metrics together — print_table lays peak RSS and elements-per-second side by side, which is the comparison the parser choice actually turns on.

For a live memory trace rather than only the final peak, sample psutil.Process().memory_info().rss on a timer thread inside each child; the getrusage peak is enough to rank the parsers, but a trace shows where pyrosm’s footprint spikes during geometry reconstruction — useful context for the windowed approach in memory-efficient chunk processing.

Verification Jump to heading#

Confirm the harness measured what you think before trusting the ranking:

• Peaks must differ. pyosmium’s peak RSS should be a small fraction of pyrosm’s on the same extract; if they are near-identical, you are almost certainly measuring in one process — check that spawn is in effect.
• Element counts should be the same order of magnitude. pyrosm’s network_type="all" count will not match pyosmium’s total-primitive count exactly (pyrosm filters to the network), but a 1000× gap means the read pulled the wrong feature set.
• Sanity-check the units. A city extract reporting a peak of tens of kilobytes means the Linux KiB-to-bytes conversion was skipped; a realistic pyrosm peak on a city file is hundreds of megabytes to a few gigabytes.
• Repeat and look for stability. Run three times; throughput should cluster within roughly 10–20%. Wild variance points at other load on the machine or cold-versus-warm cache effects.

Common errors and fixes Jump to heading#

Specification reference Jump to heading#

pyosmium processes a file by invoking handler callbacks as it streams elements, which is what keeps its resident memory bounded regardless of file size; see the pyosmium documentation for SimpleHandler and apply_file semantics. The peak-memory metric comes from getrusage(RUSAGE_SELF), whose ru_maxrss field is defined as the maximum resident set size used, in kilobytes on Linux — consult the Python resource documentation for the field list and platform notes.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• pyosmium vs pyrosm vs osmium-tool: Choosing the Right Parser — the decision guide these numbers make concrete.
• Memory-Efficient Chunk Processing — what to do when the benchmark says a single-pass read will not fit in memory.
• Async PBF Parsing with Pyrosm — scaling pyrosm past the single-process throughput this harness measures.
• OSM XML vs PBF Comparison — why every benchmark here uses .osm.pbf as the input.

Up one level: pyosmium vs pyrosm vs osmium-tool: Choosing the Right Parser.