Profiling Peak Memory of an OSM Parser Jump to heading#

Every OSM pipeline is eventually killed by the operating system for using too much memory, and the debugging that follows is usually guesswork because nobody measured anything before it happened.

Prerequisites Jump to heading#

• Python 3.10+; tracemalloc and resource are in the standard library.
• psutil for sampling resident memory, or a willingness to read the process filesystem.
• A parser to measure, and two differently-sized inputs.
• The strategies in Memory-Efficient Chunk Processing, so the measurement has something to inform.
• Patience for a full run, since peak memory is by definition not visible early.

Conceptual minimum Jump to heading#

Three different numbers get called “memory usage” and they answer different questions.

Resident set size is what the operating system counts and what gets a process killed. It includes memory the allocator has freed but not returned, and pages of memory-mapped files that happen to be resident. It is the number that matters operationally and the least useful for finding a cause.

Allocated bytes, as tracked by the language runtime, attribute memory to the code that requested it. That makes it the number for diagnosis, and it deliberately excludes anything the runtime did not allocate — which includes memory-mapped files and native library allocations.

Peak versus current is the distinction that catches people. A pipeline sitting at two gigabytes may have peaked at twelve during a merge, and only the peak explains why it was killed.

The diagnostic question is almost always does it grow with input size. A structure whose size tracks the file is unbounded and will fail on a larger one; a cache that plateaus is fine however large it looks. Two runs on differently-sized inputs answer that question definitively, and nothing else does.

Runnable solution Jump to heading#

from __future__ import annotations

import logging
import resource
import threading
import time
import tracemalloc
from collections.abc import Callable
from contextlib import contextmanager
from dataclasses import dataclass, field

import psutil

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.profile.memory")

SAMPLE_INTERVAL = 0.25


@dataclass
class MemoryProfile:
    peak_rss_mb: float = 0.0
    final_rss_mb: float = 0.0
    peak_allocated_mb: float = 0.0
    samples: list[float] = field(default_factory=list)
    top_allocators: list[str] = field(default_factory=list)
    seconds: float = 0.0


class Sampler(threading.Thread):
    """Resident memory is sampled, not queried: the peak is a maximum over time."""

    def __init__(self, interval: float = SAMPLE_INTERVAL) -> None:
        super().__init__(daemon=True)
        self.interval = interval
        self.samples: list[float] = []
        self._stop = threading.Event()
        self._process = psutil.Process()

    def run(self) -> None:
        while not self._stop.is_set():
            self.samples.append(self._process.memory_info().rss / 1024 ** 2)
            self._stop.wait(self.interval)

    def stop(self) -> None:
        self._stop.set()
        self.join(timeout=2.0)


@contextmanager
def profile(top: int = 8):
    """Measure resident peak, allocation peak and the top allocating lines."""
    tracemalloc.start(25)
    sampler = Sampler()
    sampler.start()
    started = time.perf_counter()
    result = MemoryProfile()
    try:
        yield result
    finally:
        result.seconds = time.perf_counter() - started
        snapshot = tracemalloc.take_snapshot()
        _current, peak = tracemalloc.get_traced_memory()
        tracemalloc.stop()
        sampler.stop()

        result.samples = sampler.samples
        result.peak_rss_mb = max(sampler.samples) if sampler.samples else 0.0
        result.final_rss_mb = sampler.samples[-1] if sampler.samples else 0.0
        result.peak_allocated_mb = peak / 1024 ** 2
        # The kernel's own high-water mark, as a cross-check on the sampling.
        kernel_peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
        result.top_allocators = [
            f"{stat.size / 1024 ** 2:7.1f} MB  {stat.traceback.format()[-1].strip()}"
            for stat in snapshot.statistics("lineno")[:top]]

        logger.info("peak rss %.0f MB (kernel says %.0f), final %.0f MB, "
                    "peak allocated %.0f MB, %.1fs",
                    result.peak_rss_mb, kernel_peak, result.final_rss_mb,
                    result.peak_allocated_mb, result.seconds)
        for line in result.top_allocators:
            logger.info("  %s", line)


def growth_test(run: Callable[[str], None], small: str, large: str,
                size_ratio: float) -> str:
    """Does memory grow with input size? This is the only question that matters.

    A structure whose footprint tracks the input is unbounded and will fail on
    a larger file; one that plateaus is a cache and is fine however large.
    """
    with profile() as a:
        run(small)
    with profile() as b:
        run(large)

    growth = b.peak_rss_mb / max(a.peak_rss_mb, 1.0)
    logger.info("input grew %.1fx, peak memory grew %.2fx", size_ratio, growth)

    if growth > size_ratio * 0.7:
        return ("UNBOUNDED: memory tracks input size; a structure is "
                "accumulating per element")
    if growth > 1.3:
        return "PARTIAL: something grows with input, but sub-linearly"
    return "BOUNDED: memory is flat; the footprint is a cache or a buffer"


if __name__ == "__main__":
    logger.info("run the growth test before optimising anything")

Step-by-step walkthrough Jump to heading#

1. Sample resident memory rather than querying it. The peak is a maximum over time, and a single reading after the run has finished misses it entirely — often by a factor of five.
2. Cross-check against the kernel’s high-water mark. The operating system tracks a peak of its own, and a large disagreement with the sampled maximum means the sampling interval is too coarse for a short spike.
3. Track allocations separately. Runtime allocation tracking attributes memory to lines of code, which is what turns “it used twelve gigabytes” into “this dictionary used twelve gigabytes”.
4. Take the snapshot before stopping the tracker. Stopping first discards everything, which is an easy mistake to make and produces an empty report.
5. Report peak and final separately. A large gap between them means a transient spike — a merge, a sort, a batch — rather than steady growth, and the two have completely different fixes.
6. Run the growth test first. Comparing peak memory across two input sizes distinguishes an unbounded structure from a large but constant cache, and that distinction determines whether there is a problem at all.
7. Compare growth against the size ratio. Memory growing roughly in proportion to input is the signature of accumulation; growing far less is a cache warming up.

Verification Jump to heading#

• Peak exceeds final. If they are equal, the sampler probably started too late or the interval is too coarse.
• The kernel agrees. The sampled peak and the kernel’s high-water mark should be close; a large gap means a missed spike.
• The growth test is decisive. Two inputs differing by a factor of four should produce an unambiguous verdict.
• Top allocators are plausible. The largest entries should be structures you can name; an unfamiliar line is where to look.
• The profile is reproducible. Two runs on identical input should report peaks within a few percent.

Common errors and fixes Jump to heading#

Specification reference Jump to heading#

Resident set size measures the portion of a process’s memory held in physical RAM, including pages of memory-mapped files and memory freed by the program but not returned to the operating system. Runtime allocation tracking attributes only allocations made through the language runtime and reports both current and peak traced memory. See the Python tracemalloc documentation for the snapshot and peak interfaces.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Memory-Efficient Chunk Processing — the parent topic and the strategies this measures.
• Using an LMDB Node Store for OSM Parsing — a change whose effect this profile confirms.
• Sizing PBF Chunk Batches to a Memory Budget — turning the measured peak into a batch size.
• Bounded LRU Node Cache for OSM Streaming — the structure a plateau usually represents.
• Applying Backpressure in an Asyncio OSM Pipeline — the fix when growth comes from an unbounded queue.

Up one level: Memory-Efficient Chunk Processing.