Tuning Pyrosm Worker Count for PBF Parsing Jump to heading#

Pick the number of processes for a parallel pyrosm or pyosmium parse of a tiled OSM extract so throughput climbs with your cores but total resident memory never crosses the RAM budget and triggers an OOM kill — and back the choice with a measured sweep rather than a guess.

Prerequisites Jump to heading#

Work through this list before you trust any worker-count number; the right value is a property of your hardware and your tiles, not a universal constant.

• Python 3.10+ with concurrent.futures.ProcessPoolExecutor available from the standard library.
• pyrosm ≥ 0.6 installed (pip install pyrosm) for the per-tile parse used in the benchmark.
• psutil ≥ 5.9 (pip install psutil) to sample per-process resident-set size during the sweep.
• A set of representative .osm.pbf tiles produced by the ingestion stage in Async PBF Parsing with Pyrosm — not one atypical tile.
• A firm RAM budget in bytes: total machine memory minus headroom for the OS, the parent process, and any downstream sink.
• The core count from os.cpu_count(), and awareness of whether your parse is CPU-bound or I/O-bound (covered below).

Conceptual minimum Jump to heading#

Parallel PBF parsing is embarrassingly parallel at the tile granularity: each worker process opens one extract, parses it independently, and returns a compact result. Adding workers raises throughput — but only until one of two ceilings is hit. The first is the core ceiling: once every physical core is busy on CPU-bound parse work, another worker just time-slices an already-saturated CPU and buys nothing. The second, and the one that actually bites in OSM pipelines, is the memory ceiling. Each pyrosm worker materializes a full GeoDataFrame, so its peak resident-set size (RSS) can run to hundreds of megabytes on a dense tile. Multiply that by too many workers and total RSS crosses the RAM budget; the kernel’s OOM killer reaps a worker, the pool raises BrokenProcessPool, and the whole job dies — usually hours in.

So the correct worker count is the smaller of the two ceilings. If a worker’s peak RSS is peak_RSS_per_worker and you are willing to spend RAM_budget bytes on parsing, then memory admits at most floor(RAM_budget / peak_RSS_per_worker) workers, and cores admit at most cores. The safe pool size is:

$$\text{workers} = \min\!\left(\text{cores},\; \left\lfloor \frac{\text{RAM\_budget}}{\text{peak\_RSS\_per\_worker}} \right\rfloor \right)$$

The two inputs behave differently. cores is fixed and free to read; peak_RSS_per_worker must be measured, because it depends on tile density, whether you request geometry, and which feature classes you pull. The measurement matters more than the formula: guess the RSS low and you reintroduce the OOM you were avoiding. Whether cores or memory wins also depends on the nature of the work — a distinction worth making explicit before benchmarking.

Whether the core knee or the memory ceiling comes first is set by the parse profile. CPU-bound parses — pulling networks and reconstructing geometry — saturate cores, so throughput bends over near cores and you rarely want more workers than physical cores. I/O-bound parses — reading large tiles off a slow disk while doing little per-record work — leave cores idle, so a few extra workers overlap disk latency, but each still costs a full worker’s RSS, so the memory ceiling is unforgiving regardless. When memory is the true constraint, widening the pool is the wrong move; fall back to the streaming approach in Memory-Efficient Chunk Processing instead.

Runnable solution Jump to heading#

The script sweeps worker counts from 1 up to the core count, measuring wall-clock throughput and peak per-worker RSS at each step, then applies the sizing formula to recommend a pool size that fits the RAM budget. It targets Python 3.10+, pyrosm>=0.6, and psutil>=5.9.

from __future__ import annotations

import logging
import os
import time
from concurrent.futures import ProcessPoolExecutor
from pathlib import Path

import psutil
from pyrosm import OSM

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.worker_tuner")

# RAM you are willing to spend on parsing (bytes): leave OS + parent headroom.
RAM_BUDGET = 12 * 1024**3


def parse_tile(tile_path: str) -> tuple[int, int]:
    """Parse one tile; return (row_count, peak_rss_bytes) for this worker."""
    reader = OSM(tile_path)
    gdf = reader.get_network(network_type="driving")
    rows = 0 if gdf is None or gdf.empty else len(gdf)
    # RSS after the GeoDataFrame is materialized is this worker's peak.
    peak_rss = psutil.Process(os.getpid()).memory_info().rss
    return rows, peak_rss


def sweep(tiles: list[Path], max_workers: int) -> None:
    """Benchmark throughput and per-worker RSS across worker counts."""
    cores = os.cpu_count() or 1
    upper = min(max_workers, cores)
    best_peak_rss = 0

    for n in range(1, upper + 1):
        start = time.perf_counter()
        with ProcessPoolExecutor(max_workers=n) as pool:
            results = list(pool.map(parse_tile, [str(t) for t in tiles]))
        elapsed = time.perf_counter() - start

        peak_rss = max(rss for _, rss in results)
        best_peak_rss = max(best_peak_rss, peak_rss)
        throughput = len(tiles) / elapsed
        logger.info(
            "workers=%d  throughput=%.2f tiles/s  peak_rss/worker=%.0f MiB",
            n, throughput, peak_rss / 1024**2,
        )

    recommend(cores, best_peak_rss)


def recommend(cores: int, peak_rss_per_worker: int) -> int:
    """workers = min(cores, floor(RAM_BUDGET / peak_RSS_per_worker))."""
    if peak_rss_per_worker <= 0:
        logger.warning("no RSS measurement; defaulting to 1 worker")
        return 1
    mem_ceiling = RAM_BUDGET // peak_rss_per_worker
    workers = max(1, min(cores, mem_ceiling))
    logger.info(
        "cores=%d  mem_ceiling=%d (budget %.0f GiB / %.0f MiB per worker)  -> workers=%d",
        cores, mem_ceiling, RAM_BUDGET / 1024**3,
        peak_rss_per_worker / 1024**2, workers,
    )
    return workers


if __name__ == "__main__":
    tiles = sorted(Path("tiles").glob("*.osm.pbf"))
    sweep(tiles, max_workers=os.cpu_count() or 1)

Step-by-step walkthrough Jump to heading#

1. Measure RSS inside the worker. psutil.Process(os.getpid()).memory_info().rss, read after the GeoDataFrame is built, captures that worker’s true peak — the number that drives the memory ceiling. Measuring from the parent would miss the per-process footprint entirely.
2. Cap the sweep at core count. min(max_workers, cores) avoids benchmarking pool sizes that only oversubscribe the CPU; past the core knee, CPU-bound throughput flattens and extra workers just add RSS.
3. Take the worst-case worker. max(rss for _, rss in results) uses the densest tile’s footprint, not the average — sizing against the average is how a single urban tile OOM-kills a pool that “looked fine” on rural data.
4. Apply the formula. recommend computes floor(RAM_BUDGET / peak_rss_per_worker) for the memory ceiling and takes the min with cores, clamped to at least one worker so a huge tile never yields a zero-worker recommendation.
5. Read throughput, not just success. The per-n throughput line reveals diminishing returns directly: when going from n to n+1 adds little tiles-per-second, you have passed the knee and the extra RSS is buying nothing.
6. Re-run per dataset. Peak RSS is dataset-specific. A continental sweep and a metro sweep produce different recommendations, so tune against tiles that represent the job you will actually run.

Verification Jump to heading#

Confirm the recommended count is both fast and safe:

• Total RSS stays under budget. During a full run at the recommended worker count, sum resident memory across workers (ps or a psutil monitor); the peak must sit below RAM_BUDGET with headroom to spare. Crossing it means peak_RSS_per_worker was underestimated — re-measure on a denser tile.
• Throughput is near the knee. Plot the sweep’s tiles-per-second against n; the recommended count should land at or just before the point where the curve flattens, not deep into the flat region.
• No BrokenProcessPool. A clean full run with zero OOM-killed workers is the acceptance test. Any such kill means the pool is too wide for the budget.
• Formula agrees with observation. The recommend log line’s mem_ceiling should match where total RSS would actually hit the budget in the sweep; a large discrepancy signals RSS was sampled at the wrong moment.

Common errors and fixes Jump to heading#

Specification reference Jump to heading#

concurrent.futures.ProcessPoolExecutor(max_workers=None) “uses os.cpu_count() on the number of worker processes” by default and runs each submission in a separate process, sidestepping the GIL for CPU-bound work — see the concurrent.futures documentation. For the underlying process model, start method, and why each worker carries its own memory image, consult the multiprocessing documentation; the per-process RSS it describes is precisely the peak_RSS_per_worker term in the sizing formula.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Async PBF Parsing with Pyrosm — the concurrent parsing pattern whose process pool this guide sizes.
• Streaming PBF Blocks Through an Asyncio Queue — bounding in-flight work at the channel so the pool width is not the only memory dial.
• Choosing an OSM Parser: pyosmium, pyrosm, osmium — how each library’s memory profile shifts the worker-count ceiling.
• Memory-Efficient Chunk Processing — the streaming alternative when memory, not cores, is the binding constraint.
• Speed Up OSM Parsing with Multiprocessing in Python — the multiprocessing fan-out this tuning applies to.

Up one level: Async PBF Parsing with Pyrosm.