Recording OSM Data Provenance in a Pipeline

Provenance is the licence obligation everybody knows about and the debugging tool nobody expects. It is also the thing that is nearly free to add on day one and a migration project to add on day eight hundred.

Prerequisites

Conceptual minimum

A provenance record answers one question: what exactly produced this? Answering it completely needs four facts.

The source identity. Not “Geofabrik” but which file, with the checksum actually consumed. A name without a digest identifies a moving target.

The data’s own timestamp. The PBF header’s replication timestamp and sequence number describe the state of the map the file captures, independently of when it was downloaded or processed.

The processing version. Which code produced the output. A result derived by a version of the pipeline with a known bug is distinguishable from one that is not, but only if the version was recorded.

The run identity. A stable identifier for this execution, so several outputs from one run can be recognised as siblings.

Together these form a small record that every output references. The record is stored once; outputs carry the identifier.

The four parts of a provenance record and the question each answers A single record divided into four parts. The source identity holds the file name and the checksum actually consumed, answering which bytes were read. The data timestamp holds the replication timestamp and sequence number from the file header, answering what state of the map the file describes. The processing version holds the pipeline's code revision, answering which logic produced the output. The run identity holds a stable identifier for the execution, answering which outputs are siblings from the same run. Four facts, one small record source file plus digest which bytes were read a name alone is not enough the digest is the identity data time timestamp, sequence from the file header what the map looked like not when you downloaded code version a revision which logic ran distinguishes a known bug from git, not by hand run id one per execution groups sibling outputs survives partial reruns cheap to generate The second part is the one people omit, and it is the only one that says anything about the data rather than about the process.
Outputs carry the record's identifier rather than a copy of it, so the cost per row is a single column.

Runnable solution

python 
from __future__ import annotations

import hashlib
import json
import logging
import subprocess
import uuid
from dataclasses import dataclass, asdict, replace
from datetime import datetime, timezone
from pathlib import Path

import osmium

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.provenance")


@dataclass(frozen=True)
class Provenance:
    run_id: str
    source_path: str
    source_digest: str
    data_timestamp: str | None       # what the map looked like
    sequence_number: int | None      # replication state of the source
    code_version: str
    started_at: str
    stages: tuple[str, ...] = ()

    @property
    def provenance_id(self) -> str:
        """Stable hash over the facts, so identical inputs give one identifier."""
        payload = json.dumps({
            "source_digest": self.source_digest,
            "data_timestamp": self.data_timestamp,
            "sequence_number": self.sequence_number,
            "code_version": self.code_version,
            "stages": list(self.stages),
        }, sort_keys=True)
        return hashlib.sha256(payload.encode()).hexdigest()[:16]

    def then(self, stage: str) -> "Provenance":
        """Return a new record extended by one transformation stage."""
        return replace(self, stages=self.stages + (stage,))


def code_version() -> str:
    try:
        out = subprocess.run(["git", "rev-parse", "--short", "HEAD"],
                             capture_output=True, text=True, check=True)
        dirty = subprocess.run(["git", "status", "--porcelain"],
                               capture_output=True, text=True, check=True)
        # A dirty tree means the recorded revision does not describe the code.
        return out.stdout.strip() + ("-dirty" if dirty.stdout.strip() else "")
    except (OSError, subprocess.CalledProcessError):
        logger.warning("no code version available; provenance will be weaker")
        return "unknown"


def digest_of(path: Path, chunk: int = 1 << 20) -> str:
    hasher = hashlib.sha256()
    with path.open("rb") as fh:
        while block := fh.read(chunk):
            hasher.update(block)
    return hasher.hexdigest()


def header_facts(path: Path) -> tuple[str | None, int | None]:
    reader = osmium.io.Reader(str(path))
    try:
        header = reader.header()
        stamp = header.get("osmosis_replication_timestamp") or None
        seq = header.get("osmosis_replication_sequence_number") or None
    finally:
        reader.close()
    return stamp, int(seq) if seq else None


def ingest(path: Path) -> Provenance:
    """Capture provenance at the moment the input is first read."""
    stamp, seq = header_facts(path)
    if stamp is None:
        logger.warning("%s has no replication timestamp; freshness is unknowable",
                       path.name)
    prov = Provenance(
        run_id=str(uuid.uuid4()),
        source_path=str(path),
        source_digest=digest_of(path),
        data_timestamp=stamp,
        sequence_number=seq,
        code_version=code_version(),
        started_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
    )
    logger.info("run %s reads %s (data of %s, code %s) -> provenance %s",
                prov.run_id[:8], path.name, stamp, prov.code_version,
                prov.provenance_id)
    return prov


def write_record(prov: Provenance, directory: Path) -> Path:
    directory.mkdir(parents=True, exist_ok=True)
    path = directory / f"{prov.provenance_id}.json"
    path.write_text(json.dumps(asdict(prov), indent=2), encoding="utf-8")
    return path


if __name__ == "__main__":
    prov = ingest(Path("/data/poland-latest.osm.pbf"))
    prov = prov.then("tags-filter:highway").then("export:geoparquet")
    logger.info("after %d stage(s), provenance id is %s",
                len(prov.stages), prov.provenance_id)
    write_record(prov, Path("_provenance"))

Step-by-step walkthrough

  1. Capture at ingestion, not at output. The facts are available when the file is opened and become guesswork afterwards.
  2. Hash the file you actually read. A published checksum proves the download was intact; hashing the local file proves you are describing the bytes on disk right now.
  3. Read the header’s replication fields. These describe the map, not the file transfer, and they are what makes a result comparable against another run months later.
  4. Warn when the timestamp is missing. A file without one cannot support any freshness claim, and knowing that up front is better than discovering it during an investigation.
  5. Record a dirty working tree. A revision identifier from a modified checkout does not describe the code that ran, and the suffix says so honestly.
  6. Extend rather than mutate through stages. Each transformation returns a new record with one more stage appended, so the provenance identifier changes when the processing does.
  7. Derive the identifier from the facts. A content hash means two runs over identical inputs with identical code produce the same identifier, which is what makes outputs comparable.
  8. Store the record once, reference it everywhere. Outputs carry a sixteen-character identifier; the full record lives in one place.
Three questions provenance answers, only one of which is about licensing Three panels. The licence question asks what an output is derived from, which attribution and classification both need and which is the reason provenance is usually introduced. The debugging question asks why two runs differ, which is answered instantly when the provenance identifiers differ and requires an investigation when they are absent. The reproducibility question asks how to recreate a result from six months ago, which needs the exact source digest and code revision and is impossible without them. Three questions, three very different audiences Licence What is this derived from? Attribution needs it Classification needs it Why it gets introduced Debugging Why do two runs differ? Compare the identifiers Instant when present An investigation when not Reproducibility Recreate last quarter Needs digest and revision Impossible without both Asked during an audit The second and third uses are why teams that add provenance for licensing reasons end up keeping it for everything else.
One column of sixteen characters answers all three, which is an unusually good return for a design decision.
How a provenance identifier survives a multi-stage pipeline Four stages of one pipeline run. At ingestion the record holds the source digest, the header timestamp and the code revision, producing an initial identifier. After the filtering stage the record gains that stage's name and the identifier changes. After the export stage it gains another and changes again. Every row written by the export carries only the final identifier, from which the whole lineage can be read back by looking up the stored record. The identifier changes with the processing, not just the input ingest digest, stamp, revision first identifier filter stage appended identifier changes export stage appended identifier changes rows carry the final one lineage looks up Two outputs from the same extract processed differently get different identifiers, which is exactly the distinction a debugger needs.
Without the per-stage extension, two very different outputs from one extract would be indistinguishable in the record.

Verification

  • Identical inputs give identical identifiers. Run twice with no changes; the provenance identifier must match.
  • A code change changes the identifier. Commit a change and re-run; it must differ.
  • A dirty tree is visible. Modify a file without committing and confirm the recorded version carries the dirty marker.
  • Every output references a record. Query outputs for a null provenance identifier; the count must be zero.
  • The record resolves. Pick an identifier from an output and confirm the stored record for it exists and describes a real file.

Common errors and fixes

Symptom Root cause One-line fix
Cannot say what a result came from Provenance captured at output, not input Capture the moment the file is opened
Two runs indistinguishable Identifier not derived from the facts Hash source digest, timestamp and code version together
Version recorded but wrong Dirty working tree not detected Append a marker when the checkout has uncommitted changes
Freshness claims unsupportable Header timestamp never read Read the replication fields and warn when absent
Provenance lost after a transformation Record not propagated through stages Extend the record per stage and carry it forward
Every row stores the full record Record copied instead of referenced Store once, reference by identifier
Retrofit is a large project Provenance added late Capture from the first version; it is one column

Specification reference

A PBF file’s OSMHeader may carry osmosis_replication_timestamp, osmosis_replication_sequence_number and osmosis_replication_base_url, describing the replication state the file was produced from. These fields travel with the file through copies and archives, unlike filesystem metadata. See the PBF format documentation for the header fields and Replication Sequence Numbers & State for what the sequence number means.

Frequently Asked Questions

Why record the header timestamp as well as the download date?

Because they answer different questions. The download date says when you fetched the file; the header timestamp says what state of the map the file describes. A file downloaded today can describe a map from last week if a mirror is stale, and only the header field reveals that. It also travels with the file through copies and archives, so a result derived from an archived extract three years ago is still explainable.

Should the provenance identifier be a hash or a sequence?

A hash over the facts, because it makes identical inputs produce identical identifiers. That in turn means two outputs with the same identifier are provably derived from the same bytes by the same code, which a sequence number cannot tell you. It also removes the need for a central counter, which matters when several machines run stages independently.

How do I carry provenance through a transformation?

By extending the record rather than replacing it: each stage appends its name and returns a new record, so the identifier changes when the processing does and the lineage is readable from the record itself. That way an output’s identifier describes not just which extract it came from but which sequence of transformations produced it, which is what makes two similar-looking outputs distinguishable.

Is one column per row too expensive?

A sixteen-character identifier is negligible next to geometry, and it is one column rather than a copy of the record. The alternative — reconstructing which run produced which rows, after the fact — is not a cost comparison at all, because it is usually impossible. The genuinely expensive option is adding provenance to a pipeline that has been running without it, which means every historical output stays unexplainable.

Up one level: OSM Licensing & ODbL Compliance.