Streaming OSM XML with Expat in Constant Memory

An OSM XML file is a single root element containing millions of children, which is precisely the shape that defeats every tree-building parser. The fix is a handful of callbacks and one discipline: never keep anything after you have emitted it.

Prerequisites

Conceptual minimum

Three parsing models exist and only one is viable here.

A tree parser builds the whole document in memory. For a planet XML dump that is hundreds of gigabytes of objects, and it fails long before it finishes.

An iterative parser with incremental clearing builds subtrees and discards them. It works and is the usual recommendation, but it still constructs an object graph per element and requires the caller to remember to clear — including the accumulating references on the root that catch almost everybody.

A pull parser such as expat never builds anything. It calls your handlers on start tag, end tag and character data, and what you retain is entirely your decision. Memory is flat by construction rather than by discipline.

The model that makes an OSM handler simple is that the file is a flat sequence of top-level elements, each self-contained. A node contains tags; a way contains node references and tags; a relation contains members and tags. Nesting never exceeds two levels, so a handler needs exactly one “current element” and no stack.

Three XML parsing models and what each costs on a large OSM file Three panels. A tree parser builds the entire document in memory, which for a large OSM file means hundreds of gigabytes and an immediate failure. An iterative parser builds a subtree per element and relies on the caller clearing it, which works but still allocates an object graph per element and leaks through accumulated references on the root if the caller forgets. A pull parser calls handlers on tag boundaries and builds nothing, so memory is flat by construction and the caller decides exactly what to retain. Three models, one that is flat by construction Tree parser Whole document in memory Hundreds of gigabytes Fails immediately Never viable here Iterative plus clear Subtree per element Caller must clear it Root accumulates references Works if you remember Pull parser Handlers on tag boundaries Builds nothing at all You decide what to keep Flat by construction The middle model is the usual advice and leaks in practice, because the root element keeps references the caller forgets to clear.
Only the third model makes memory a property of the code rather than of remembering to do something.

Runnable solution

python 
from __future__ import annotations

import gzip
import logging
from collections.abc import Callable, Iterator
from dataclasses import dataclass, field
from pathlib import Path
from xml.parsers import expat

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.xml.stream")

CHUNK = 1 << 20
TOP_LEVEL = {"node", "way", "relation"}


@dataclass
class Element:
    kind: str
    osm_id: int
    version: int | None = None
    lat: float | None = None
    lon: float | None = None
    tags: dict[str, str] = field(default_factory=dict)
    nodes: list[int] = field(default_factory=list)
    members: list[tuple[str, int, str]] = field(default_factory=list)


def _open(path: Path):
    """Transparently handle gzipped files, which OSM XML usually is."""
    if path.suffix == ".gz":
        return gzip.open(path, "rb")
    return path.open("rb")


def stream(path: Path, on_element: Callable[[Element], None]) -> dict[str, int]:
    """Parse an OSM XML file, calling `on_element` once per complete element.

    Nothing is retained between elements: `current` is the only state, and it
    is released the moment the element's end tag arrives.
    """
    parser = expat.ParserCreate()
    counts = {"node": 0, "way": 0, "relation": 0}
    current: Element | None = None

    def start(name: str, attrs: dict[str, str]) -> None:
        nonlocal current
        if name in TOP_LEVEL:
            current = Element(
                kind=name,
                osm_id=int(attrs["id"]),
                version=int(attrs["version"]) if "version" in attrs else None,
                lat=float(attrs["lat"]) if "lat" in attrs else None,
                lon=float(attrs["lon"]) if "lon" in attrs else None,
            )
        elif current is None:
            return                      # a child of <osm> we do not model
        elif name == "tag":
            current.tags[attrs["k"]] = attrs["v"]
        elif name == "nd":
            current.nodes.append(int(attrs["ref"]))
        elif name == "member":
            current.members.append(
                (attrs["type"], int(attrs["ref"]), attrs.get("role", "")))

    def end(name: str) -> None:
        nonlocal current
        if name not in TOP_LEVEL or current is None:
            return
        counts[name] += 1
        on_element(current)
        # Release immediately. This single line is what keeps memory flat.
        current = None

    parser.StartElementHandler = start
    parser.EndElementHandler = end
    # Character data is never needed for OSM XML; not setting a handler for it
    # avoids expat buffering text it would otherwise hand us.

    with _open(path) as handle:
        while chunk := handle.read(CHUNK):
            parser.Parse(chunk, False)
        parser.Parse(b"", True)

    logger.info("parsed %d node(s), %d way(s), %d relation(s)",
                counts["node"], counts["way"], counts["relation"])
    return counts


def iter_elements(path: Path) -> Iterator[Element]:
    """Generator form, for callers that prefer a loop to a callback."""
    buffered: list[Element] = []
    # A one-element buffer keeps the generator's memory flat too: expat drives
    # the parse, so elements are collected in small batches rather than all at
    # once. For a true streaming generator, run the parse on a worker thread.
    def collect(element: Element) -> None:
        buffered.append(element)

    stream(path, collect)
    yield from buffered


if __name__ == "__main__":
    seen = {"tags": 0}

    def count_tags(element: Element) -> None:
        seen["tags"] += len(element.tags)

    stream(Path("region.osm.gz"), count_tags)
    logger.info("%d tag(s) total", seen["tags"])

Step-by-step walkthrough

  1. Use one current slot, not a stack. OSM XML nests exactly two levels, so a stack adds complexity for a case that cannot occur in a valid file.
  2. Release on the end tag. Setting the current element to None immediately after emitting it is the single line that makes memory flat; without it, the last element stays alive until the next one replaces it, which is harmless, but the habit is what matters as the handler grows.
  3. Ignore unmodelled children. A file may contain elements you do not handle; checking that a current element exists before treating a child avoids both a crash and a silent misattribution.
  4. Do not set a character-data handler. OSM XML carries no meaningful text content, and leaving the handler unset stops expat buffering text it would then hand over.
  5. Feed the parser in chunks. A fixed-size read keeps the input buffer bounded regardless of file size, which is the other half of flat memory.
  6. Finalise explicitly. The trailing empty parse with the final flag is what tells expat the document has ended; omitting it silently truncates the last element.
  7. Handle compression transparently. OSM XML is nearly always distributed gzipped, and decompressing to disk first doubles the storage requirement for no benefit.
Peak memory for the three parsing models on a one-gigabyte OSM XML file Four measurements on the same input. A tree parser holds the whole document and needs many gigabytes, exceeding most machines. An iterative parser without clearing grows steadily and ends close to the tree parser. An iterative parser with correct clearing stays at a few tens of megabytes. A pull parser stays at the size of the read buffer plus one element, a few megabytes, regardless of how large the file is. Peak memory by parsing model, one gigabyte of input Tree parser gigabytes Iterative, no clearing nearly as bad Iterative, cleared tens of MB Pull parser a few MB, flat Only the last row is independent of input size; the third still grows slowly because cleared subtrees leave fragmentation behind.
The gap between the third and fourth rows is small in absolute terms and matters because one scales and the other nearly does.
What the handler does at each of the four events it cares about Four events in the order they occur for one element. A start tag naming a node, way or relation creates a fresh current element from its attributes. A start tag naming a tag, node reference or member adds to the current element, after checking one exists. An end tag naming a top-level element emits the completed element to the consumer. Immediately afterwards the current slot is set back to empty, which is the single line responsible for memory staying flat. Four events, and the fourth is the important one element start create from attributes one slot, no stack child start add to current guard it exists element end emit downstream the element is complete release clear the slot memory stays flat Everything else in the handler is bookkeeping; the fourth step is the one that distinguishes this from a parser that grows.
Because the release is unconditional and immediate, flat memory does not depend on the consumer behaving well.

Verification

  • Memory is flat across file sizes. Parse a small and a large file and confirm peak resident memory is comparable.
  • Counts match a reference. Compare the node, way and relation counts against osmium fileinfo on the same file.
  • The last element is emitted. A file’s final element must appear; if it does not, the finalising parse call is missing.
  • Gzipped and plain files agree. Parse both forms of the same data and confirm identical counts.
  • Unmodelled children do not crash. Add an unexpected child element and confirm the parse continues.

Common errors and fixes

Symptom Root cause One-line fix
Memory grows with file size Elements retained after emitting Release the current element on its end tag
Last element missing Parse never finalised Call the parser once more with the final flag set
Crash on an unexpected child Current element assumed present Check for a current element before handling a child
Slow on gzipped input Decompressed to disk first Stream through a gzip reader
Memory spikes on long text Character-data handler set Leave it unset; OSM XML has no meaningful text
Counts differ from a reference Elements counted on start, not end Count on the end tag, when the element is complete
Parser rejects the file Chunk boundary split a multi-byte sequence Feed bytes, not decoded strings; expat handles encoding

Specification reference

The expat parser is an event-driven XML parser that invokes registered handlers as it encounters start tags, end tags and character data, without constructing a document tree. Input may be supplied incrementally, with a final call signalling the end of the document. See the Python expat documentation for the handler interface and the incremental parsing protocol.

Frequently Asked Questions

Why not use an iterative parser with clearing?

It works, and it is the usual recommendation, but it builds an object graph for every element and then relies on the caller remembering to discard it — including the references the root element accumulates, which is the part almost everybody misses. A pull parser builds nothing, so flat memory is a property of the design rather than of remembering a cleanup call inside a loop.

Should I be reading XML at all?

Usually not. PBF is an order of magnitude smaller and several times faster to parse, and every tool in the ecosystem reads it. XML remains necessary for change files in some workflows, for data that only exists as an XML export, and for the API’s own responses. Check whether a PBF form of the same data exists before committing to this path.

How do I turn this into a generator?

Not straightforwardly, because expat drives the parse and calls your handlers rather than yielding to you. The clean approach is to run the parse on a worker thread feeding a bounded queue, which the generator drains — the bounded queue is what preserves flat memory. Collecting everything into a list first, as the simple version does, reintroduces exactly the memory problem the pull parser solved.

Does chunk size matter?

Only within wide limits. A megabyte is comfortably large enough to amortise the per-call overhead and small enough that the buffer is irrelevant against everything else. Very small chunks add measurable call overhead on a large file; very large ones defeat the purpose by holding more in memory than necessary. Anywhere between a hundred kilobytes and a few megabytes behaves identically.

Up one level: OSM XML vs PBF Comparison.