Full-History .osh.pbf Processing Jump to heading#

A standard .osm.pbf extract answers one question — what does the map look like now — and it discards everything the map used to be. The moment your work needs the past tense, that file is the wrong input, and the failure is quiet rather than loud: a contributor-activity report built from a current-state planet silently under-counts every element that was later deleted or reverted, a temporal join against a 2021 land-use layer matches against 2026 geometry, and an audit of “who touched this building” returns exactly one row because only the newest version survives. The full-history format, distributed as .osh.pbf (the h marks history), fixes this by keeping every version of every node, way, and relation ever committed, along with the changeset, editor, and timestamp that produced each one. This guide sits inside the broader OSM Replication & Diff Sync section, and where the replication pages care about moving the present forward one diff at a time, this one is about reading the whole timeline at rest.

Prerequisites: What You Need in Place First Jump to heading#

Full-history processing rests on the same primitives as any OSM workflow, seen through a temporal lens. You need a working grasp of the node, way, and relation data model, because in a history file each of those primitives is no longer a single object but a chain of versions sharing one id, and a way’s geometry can shift under it when its member nodes are edited independently. You need to be able to read a PBF header’s required-features list, since that is where a file declares itself historical — the walkthrough in how to decode OSM PBF headers in Python shows exactly how to pull the required_features array that carries the HistoricalInformation flag. Finally, a modern pyosmium (osmium ≥ 3.6) and the osmium command-line tool (libosmium ≥ 1.14) must both be installed; the CLI path in this guide relies on the time-filter subcommand, and the Python path relies on the version-aware object attributes that older bindings did not expose.

The .osh.pbf Format: Fields and Rules Jump to heading#

A full-history file is byte-compatible with an ordinary .osm.pbf at the framing level — the same protobuf blob-and-block structure — but it differs in two contractual ways. First, its header’s required_features list contains the string HistoricalInformation, which is a hard signal to any reader that it must expect multiple objects with the same (type, id) and must not assume the last one wins. A consumer that ignores this flag and folds the file into a current-state map will overwrite earlier versions with later ones and quietly lose the history it was handed. Second, every object carries its full version metadata, and deletions are represented not by omission but by an explicit tombstone: a version whose visible attribute is false.

The per-version metadata block is the whole point of the format. Every node, way, and relation version exposes the following fields.

Three rules follow directly from that table and govern every correct history consumer. Versions of one object are strictly ordered by version, and version is a better sort key than timestamp because two rapid edits can share a second-resolution timestamp while their version numbers never collide. A visible=false version terminates the object’s life at its timestamp — but not necessarily forever, since a later visible=true version can resurrect the same id, exactly as v5 does in the diagram above. And the value of any snapshot at instant T is, per object, the single highest-version record whose timestamp is ≤ T; if that record has visible=false, the object simply does not exist at T. That last sentence is the entire algorithm — everything else is making it fast and memory-bounded over a planet-sized file.

Step-by-Step: Streaming Every Version Jump to heading#

The processing pattern is a single sequential pass that hands you each version in file order, from which you either accumulate per-object chains or fold directly toward a snapshot. It uses Python 3.10+ type hints and the project’s logger convention.

1. Confirm the file is historical. Read the header before parsing a byte of payload and assert that it advertises multiple object versions. This is a cheap guard that turns a silent correctness bug — treating a history file as current-state, or vice versa — into a loud failure at startup.
2. Stream all versions with a handler. A SimpleHandler dispatches its node, way, and relation callbacks once per version, in (type, id, version) order within each type block. Capture version, visible, timestamp, changeset, and uid on each call.
3. Key temporal state on (type, id, version). Never key on id alone; that is the collision that collapses a history back into a snapshot. The composite key is unique across the whole file and lets you index, deduplicate, and diff versions without ambiguity.
4. Fold toward the target. For a point-in-time snapshot, retain per (type, id) only the newest version with timestamp ≤ T and drop the object entirely if that survivor is a deletion — the reduction the child guides implement end to end.

import logging
from datetime import datetime

import osmium

logger = logging.getLogger("osm.history.reader")


def assert_history_file(path: str) -> None:
    """Fail fast unless the PBF header advertises HistoricalInformation."""
    header = osmium.io.Reader(path).header()
    if not header.has_multiple_object_versions():
        raise ValueError(
            f"{path} is not a full-history file; its header does not set "
            "HistoricalInformation. Use a .osh.pbf source."
        )
    logger.info("confirmed full-history input: %s", path)


class VersionCollector(osmium.SimpleHandler):
    """Stream every version of every element, keyed on (type, id, version)."""

    def __init__(self) -> None:
        super().__init__()
        # (type, id, version) -> flat metadata record
        self.versions: dict[tuple[str, int, int], dict[str, object]] = {}

    def _record(self, kind: str, obj: osmium.osm.OSMObject) -> None:
        key = (kind, obj.id, obj.version)
        self.versions[key] = {
            "type": kind,
            "id": obj.id,
            "version": obj.version,
            "visible": obj.visible,
            "timestamp": obj.timestamp,  # tz-aware UTC datetime
            "changeset": obj.changeset,
            "uid": obj.uid,
            "user": obj.user,
        }

    def node(self, n: osmium.osm.Node) -> None:
        self._record("node", n)

    def way(self, w: osmium.osm.Way) -> None:
        self._record("way", w)

    def relation(self, r: osmium.osm.Relation) -> None:
        self._record("relation", r)


def load_versions(path: str) -> dict[tuple[str, int, int], dict[str, object]]:
    assert_history_file(path)
    collector = VersionCollector()
    collector.apply_file(path)  # no locations= needed for metadata-only work
    logger.info("collected %d element versions", len(collector.versions))
    return collector.versions

Holding every version in a dictionary is fine for a city or small-region history but will not survive a continental .osh.pbf; the accumulation above is the readable form, and the performance section below explains how to keep only what a given output actually needs.

Validation & Error-Handling Matrix Jump to heading#

History processing has its own failure catalogue, distinct from current-state parsing. Each row below is a defect that produces wrong answers rather than a crash, which is what makes them dangerous.

Performance & Scale Considerations Jump to heading#

A full-history planet is several times larger than a current-state planet — tens of billions of object versions — so the accumulate-everything shape shown above is a teaching form, not a production one. The governing discipline is fold as you stream: never keep two versions of an object once you know which one you need. For a snapshot at T, that means holding a single dictionary keyed on (type, id) and, on each incoming version, overwriting the stored record only when the new version’s timestamp is ≤ T and its version exceeds the stored one. Because a history file emits versions of a given object in ascending version order, you can even drop a stored record the instant you see a version past T, which bounds resident memory to roughly the count of objects alive at T rather than the count of all versions ever.

Two levers dominate throughput. First, choose the CLI when the CLI suffices: osmium time-filter is a compiled single-pass streamer and will out-run any Python accumulation for a straight point-in-time cut, so reserve the pyosmium path for analyses that genuinely need per-version logic, such as provenance or version-to-version diffing. Second, restrict the object types you dispatch on — if you only need node history, omit the way and relation callbacks so libosmium can skip decoding those blocks. For very large jobs, the same block-independence that makes current-state PBF parallelizable applies here, but partition on object-id ranges rather than geography, because a way’s history and its nodes’ histories must land on the same worker to reconstruct geometry.

Failure Modes & Gotchas Jump to heading#

• Resurrection is legal. An id can be deleted and later recreated with a higher version, as v5 shows. Any code that stops processing an object at its first visible=false will miss the resurrection and report the object as permanently gone.
• Timestamp resolution is one second. Two edits inside the same second share a timestamp. version is the tie-breaker; treat timestamp as a coarse filter and version as the authoritative order.
• Geometry drifts under a stable id. A way keeps its id across versions, but its member node references — and those nodes’ own coordinates — change independently. Reconstructing a way at T requires the node versions live at T, not the way’s own version alone.
• Redaction leaves holes. Post-license-change redactions and account deletions can strip user or blank a version’s payload. Aggregate contributor analytics on the numeric uid, which is stable, not on the display name.
• The header flag is a contract, not a hint. Passing a current-state file to history logic (or the reverse) is a silent data error. The has_multiple_object_versions() guard is cheap; run it every time.

Integration Points: Feeding the Next Stage Jump to heading#

A history reader rarely stands alone — its output feeds either a point-in-time export or an audit table. The wiring below folds a stream directly into a snapshot dictionary, the exact reduction the reconstruction guide expands on, and keeps memory proportional to the live object count rather than the version count:

import logging
from datetime import datetime

import osmium

logger = logging.getLogger("osm.history.snapshot")


class SnapshotFolder(osmium.SimpleHandler):
    """Fold a history stream to the live state at instant T."""

    def __init__(self, cutoff: datetime) -> None:
        super().__init__()
        self.cutoff = cutoff
        # (type, id) -> (version, visible) of the newest version <= cutoff
        self._latest: dict[tuple[str, int], tuple[int, bool]] = {}

    def _fold(self, kind: str, obj: osmium.osm.OSMObject) -> None:
        if obj.timestamp > self.cutoff:
            return  # future edit; irrelevant to this snapshot
        key = (kind, obj.id)
        prev = self._latest.get(key)
        if prev is None or obj.version > prev[0]:
            self._latest[key] = (obj.version, obj.visible)

    def node(self, n: osmium.osm.Node) -> None:
        self._fold("node", n)

    def live_ids(self) -> set[tuple[str, int]]:
        """Objects that exist at the cutoff (latest visible version wins)."""
        return {k for k, (_, visible) in self._latest.items() if visible}

Downstream, that snapshot flows into whatever the current-state stages expect — spatial indexing, tag normalization, or a PostGIS load — while the sibling workflow in applying .osc change files with osmium covers the inverse motion of pushing a current extract forward through diffs rather than rewinding a history file backward.

Explore Full-History Processing in Depth Jump to heading#

This section anchors two focused guides that take the format above into concrete tasks:

• Reconstructing OSM Features at a Past Date — materialize the map exactly as it stood at a chosen timestamp, with both the osmium time-filter one-liner and a pyosmium fold that keeps the latest visible version at or before T.
• Extracting Changeset Metadata from History Files — pull per-version provenance (changeset id, uid, user, timestamp) into a table for auditing and contributor analysis, with the metadata-availability caveats spelled out.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• OSM Replication & Diff Sync — the parent section covering diffs, sequence numbers, and keeping an extract current.
• Applying .osc Change Files with osmium — the forward-in-time counterpart that advances a current extract through change files.
• Node-Way-Relation Data Model — the primitives that become version chains in a history file.
• How to Decode OSM PBF Headers in Python — reading the required_features list that carries the HistoricalInformation flag.
• Reconstructing OSM Features at a Past Date — the snapshot-at-T procedure in full.
• Extracting Changeset Metadata from History Files — per-version provenance for auditing and attribution.

This guide is part of the OSM Replication & Diff Sync section — return there for the full path from raw diffs to a continuously updated OSM database.