Extracting Changeset Metadata from History Files Jump to heading#

From a full-history .osh.pbf, collect the provenance of every element version — the changeset that made it, the editor’s uid and name, and the timestamp — into a table you can audit or aggregate by contributor.

Prerequisites Jump to heading#

Confirm each item; the most common surprise is a file that parses cleanly but reports zeros for every metadata field because its metadata was stripped upstream.

• A full-history file (*.osh.pbf) whose header sets HistoricalInformation — the format is covered in Full-History .osh.pbf Processing.
• pyosmium ≥ 3.6 installed (pip install osmium).
• Optional: pandas ≥ 2.1 for the aggregation step (pip install "pandas>=2.1").
• Python 3.10+ for the list[dict[str, object]] typing used below.
• Confirmation that the source retains metadata — a file written with add_metadata=false carries no changeset, uid, or timestamp to extract.
• If you produced the file yourself with osmium extract, that you passed --with-history so version chains and their metadata survived the cut.

Conceptual minimum Jump to heading#

Every version of every OSM element carries a fixed metadata block independent of its tags and geometry: a version counter, the changeset that committed it, the editor’s numeric uid and display user, and a UTC timestamp. In a current-state extract you see this block once per object; in a full-history file you see it once per version, which is exactly what makes the file usable for auditing — you can reconstruct who changed what, when, and under which changeset across an object’s whole life. This is the same per-version stream used to reconstruct features at a past date, read for provenance rather than for state. Extraction is therefore a flat projection: for each version, emit one row of (type, id, version, changeset, uid, user, timestamp), and leave the tags behind.

The one caveat that governs whether this works at all is metadata availability. The OSM PBF and XML schemas treat object metadata as optional, so a producer can strip it to shrink a file — and when it is stripped, pyosmium returns version = 0, changeset = 0, uid = 0, an empty user, and an invalid timestamp rather than raising. History files distributed as .osh.pbf always retain metadata, because the metadata is the history; the risk arises with self-made extracts. Whenever you cut a region from a history planet you must keep history explicitly, and if a downstream tool re-encoded the file with metadata disabled, provenance is simply gone and no code can recover it. The reader below detects that condition instead of silently emitting a table of zeros.

Runnable solution Jump to heading#

The handler below collects one provenance row per version into a list, guarding against stripped metadata, and then aggregates the rows into a per-contributor summary. The aggregation uses pandas when it is available and falls back to the standard library otherwise.

from __future__ import annotations

import logging
from collections import Counter

import osmium

logger = logging.getLogger("osm.provenance")


class ProvenanceHandler(osmium.SimpleHandler):
    """Collect (type, id, version, changeset, uid, user, timestamp) per version."""

    def __init__(self) -> None:
        super().__init__()
        self.rows: list[dict[str, object]] = []
        self.stripped = 0  # versions whose metadata looks absent

    def _row(self, kind: str, obj: osmium.osm.OSMObject) -> None:
        # Stripped metadata surfaces as version 0 and changeset 0, not an error.
        if obj.version == 0 and obj.changeset == 0:
            self.stripped += 1
            return
        self.rows.append(
            {
                "type": kind,
                "id": obj.id,
                "version": obj.version,
                "changeset": obj.changeset,
                "uid": obj.uid,
                "user": obj.user,  # may be "" if redacted
                "timestamp": obj.timestamp.isoformat() if obj.timestamp else None,
                "visible": obj.visible,
            }
        )

    def node(self, n: osmium.osm.Node) -> None:
        self._row("node", n)

    def way(self, w: osmium.osm.Way) -> None:
        self._row("way", w)

    def relation(self, r: osmium.osm.Relation) -> None:
        self._row("relation", r)


def extract_provenance(path: str) -> list[dict[str, object]]:
    """Return one provenance row per element version in *path*."""
    reader = osmium.io.Reader(path)
    if not reader.header().has_multiple_object_versions():
        logger.warning("%s is not a history file; only head versions will appear", path)
    reader.close()

    handler = ProvenanceHandler()
    handler.apply_file(path)
    if handler.stripped:
        logger.warning(
            "%d versions had no metadata (version=0); source may be stripped",
            handler.stripped,
        )
    logger.info("collected %d provenance rows", len(handler.rows))
    return handler.rows


def contributor_summary(rows: list[dict[str, object]]) -> list[tuple[int, str, int]]:
    """Aggregate rows into (uid, user, edit_count), busiest first."""
    try:
        import pandas as pd

        frame = pd.DataFrame(rows)
        grouped = (
            frame.groupby("uid")
            .agg(user=("user", "last"), edits=("version", "size"))
            .sort_values("edits", ascending=False)
            .reset_index()
        )
        return list(grouped.itertuples(index=False, name=None))
    except ImportError:
        counts: Counter[int] = Counter(r["uid"] for r in rows)
        names = {r["uid"]: r["user"] for r in rows}
        return [(uid, names[uid], n) for uid, n in counts.most_common()]


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    provenance = extract_provenance("history.osh.pbf")
    for uid, user, edits in contributor_summary(provenance)[:10]:
        logger.info("uid=%s user=%r edits=%d", uid, user, edits)

Step-by-step walkthrough Jump to heading#

1. Check the header first. extract_provenance opens an osmium.io.Reader and inspects has_multiple_object_versions() before parsing. On a current-state file it warns that only head versions exist, so a caller never mistakes a one-row-per-object result for a full history.
2. Detect stripped metadata, don’t crash on it. A version with both version == 0 and changeset == 0 had its metadata removed upstream; the handler counts these separately rather than writing rows full of zeros that would corrupt any contributor tally.
3. Project, don’t reconstruct. _row copies only the provenance fields and ignores tags and node references, so the handler stays cheap even on a large file — there is no geometry assembly.
4. Keep user as best-effort. The display name can be blank when an account was deleted or a version redacted, while uid remains stable. Rows retain both, and the summary groups on uid so redacted names never split one contributor into several buckets.
5. Aggregate flexibly. contributor_summary uses pandas for a fast groupby when it is installed and otherwise falls back to collections.Counter, so the extraction has no hard dependency on the data-frame stack.
6. Timestamps serialize as ISO-8601. obj.timestamp.isoformat() produces a sortable, timezone-aware string that loads cleanly into a database or a data frame later.

Verification Jump to heading#

• Row count matches version count. For a small region, len(rows) plus the stripped count should equal the total version count reported by osmium fileinfo -e history.osh.pbf.
• No zero uids among real edits. uid = 0 in the output means anonymous or stripped edits leaked past the guard; inspect those rows before trusting the tally.
• Changesets are monotone per object. Within one (type, id), changeset values should generally increase with version; a decrease signals rows sorted or merged incorrectly.
• Spot-check against the API. Pick a changeset id from the table and confirm its editor and timestamp on the live OSM changeset view; they must match the extracted row.
• The stripped-metadata warning fires appropriately. Running against a deliberately metadata-free file should log the versions had no metadata warning and return no rows.

Common errors and fixes Jump to heading#

Specification reference Jump to heading#

Each OSM element carries optional metadata — version, changeset, timestamp, uid, and user — that records the edit provenance; see the OSM Wiki Elements page for the field definitions and Changeset for how edits are grouped. Because the metadata is optional in the PBF and XML schemas, a file can be written without it, in which case these fields are absent; the pyosmium documentation describes the attributes exposed on each object.

Related Jump to heading#

• Full-History .osh.pbf Processing — the format and the version stream this extraction reads.
• Reconstructing OSM Features at a Past Date — the state-oriented sibling that reads the same versions to rebuild geometry at T.
• How to Decode OSM PBF Headers in Python — confirming a file is historical and metadata-bearing before extraction.
• Node-Way-Relation Data Model — the elements whose per-version metadata this table projects.
• Applying .osc Change Files with osmium — the diffs that create the changeset-stamped versions you are auditing.

Up one level: Full-History .osh.pbf Processing.

Frequently Asked Questions Jump to heading#