Pinning a Reproducible OSM Snapshot by Sequence Number

“The data as of last Tuesday” is not a reproducible input, because the extract you downloaded last Tuesday has been replaced and the planet has moved on. A replication sequence number is, and it costs nothing to record.

Prerequisites

Conceptual minimum

A replication stream is an ordered sequence of change files, each numbered, each carrying a timestamp. A sequence number therefore names a state: the base planet plus every diff up to and including that number. Two runs that reconstruct from the same base to the same sequence see identical data, which is the entire property reproducibility needs.

Three things make this work in practice.

The base must be identified too. A sequence number is meaningless without the state it is counted from. Providers that publish an extract alongside its state.txt make this easy; providers that do not force you to record the file’s own checksum instead, which pins the bytes rather than the position.

Timestamps are not identifiers. A replication timestamp tells you approximately when a diff was cut, and diffs are not evenly spaced. Resolving “2026-09-01T00:00:00Z” to a sequence requires a search through the stream’s state files, and the result is the first sequence at or after that moment — a derived answer worth computing once and then recording as a number.

Retention is finite. Minutely replication history is not kept forever. A pin that names a sequence whose diffs have been expired is not reconstructible, which means a long-lived pin needs either a materialised snapshot or a base close enough to the pin that the intervening diffs still exist.

Three ways of naming an OSM state, and what each guarantees Three panels. Naming a state by date is what people say and it guarantees nothing, because diffs are unevenly spaced, the phrase resolves differently depending on when it is interpreted, and the extract that was current on that date has since been replaced. Naming it by file checksum pins exact bytes and is perfectly reproducible as long as that file still exists somewhere, but it says nothing about position in the stream and cannot be advanced. Naming it by base plus sequence number pins a position, reconstructs deterministically, and can be advanced or compared, but only while the intervening diffs remain within the provider's retention window. Date, checksum, or sequence By date What people actually say Diffs unevenly spaced Resolves differently later Guarantees nothing By checksum Pins exact bytes Perfectly reproducible No position in the stream Cannot be advanced Base plus sequence Pins a stream position Deterministic rebuild Comparable and advanceable Needs diffs retained The third is the right default, and the second is what a long-lived pin degrades into once retention expires the diffs behind it.
Recording both a base checksum and a sequence costs two strings and covers all three cases.

Runnable solution

python 
from __future__ import annotations

import hashlib
import json
import logging
import subprocess
import urllib.request
from dataclasses import dataclass, asdict
from datetime import datetime, timezone
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.replication.pin")

BASE_URL = "https://planet.openstreetmap.org/replication/minute"


@dataclass(frozen=True)
class Pin:
    """Everything needed to rebuild a state, and nothing that varies."""
    base_file: str
    base_sha256: str
    base_sequence: int
    sequence: int
    sequence_timestamp: str
    replication_url: str

    def as_json(self) -> str:
        return json.dumps(asdict(self), indent=1, sort_keys=True)


def sequence_path(sequence: int) -> str:
    text = f"{sequence:09d}"
    return f"{text[0:3]}/{text[3:6]}/{text[6:9]}"


def state_for(sequence: int, base_url: str = BASE_URL) -> dict[str, str]:
    url = f"{base_url}/{sequence_path(sequence)}.state.txt"
    with urllib.request.urlopen(url, timeout=30) as response:
        body = response.read().decode("utf-8")
    state: dict[str, str] = {}
    for line in body.splitlines():
        if line.startswith("#") or "=" not in line:
            continue
        key, _, value = line.partition("=")
        state[key.strip()] = value.strip().replace("\\:", ":")
    return state


def latest_sequence(base_url: str = BASE_URL) -> int:
    with urllib.request.urlopen(f"{base_url}/state.txt", timeout=30) as response:
        body = response.read().decode("utf-8")
    for line in body.splitlines():
        if line.startswith("sequenceNumber="):
            return int(line.split("=", 1)[1])
    raise RuntimeError("no sequenceNumber in state.txt")


def resolve_timestamp(moment: datetime, base_url: str = BASE_URL) -> int:
    """Binary search the stream for the first sequence at or after a moment.

    Diffs are not evenly spaced, so this is a search rather than arithmetic.
    Compute it ONCE and record the integer; never re-resolve at read time.
    """
    moment = moment.astimezone(timezone.utc)
    low, high = 1, latest_sequence(base_url)
    answer = high
    while low <= high:
        mid = (low + high) // 2
        try:
            stamp = datetime.fromisoformat(state_for(mid, base_url)["timestamp"])
        except Exception:                       # expired or missing state file
            low = mid + 1
            continue
        if stamp >= moment:
            answer, high = mid, mid - 1
        else:
            low = mid + 1
    logger.info("%s resolves to sequence %d", moment.isoformat(), answer)
    return answer


def sha256(path: Path, chunk: int = 1 << 20) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        while block := handle.read(chunk):
            digest.update(block)
    return digest.hexdigest()


def make_pin(base: Path, base_sequence: int, sequence: int,
             base_url: str = BASE_URL) -> Pin:
    state = state_for(sequence, base_url)
    return Pin(base_file=base.name, base_sha256=sha256(base),
               base_sequence=base_sequence, sequence=sequence,
               sequence_timestamp=state["timestamp"], replication_url=base_url)


def materialise(pin: Pin, base: Path, out: Path) -> Path:
    """Rebuild the pinned state. Deterministic given the same pin."""
    if sha256(base) != pin.base_sha256:
        raise SystemExit(f"base file does not match pin: {base}")
    subprocess.run(
        ["osmium", "up-to-date", "--server", pin.replication_url,
         "--ending-sequence-id", str(pin.sequence),
         "-o", str(out), "--overwrite", str(base)],
        check=True)
    (out.with_suffix(out.suffix + ".pin.json")).write_text(
        pin.as_json(), encoding="utf-8")
    logger.info("materialised sequence %d to %s", pin.sequence, out)
    return out


if __name__ == "__main__":
    logger.info("resolve once, record the integer, rebuild from base plus pin")

Step-by-step walkthrough

  1. Record the base file’s checksum, not just its name. Filenames are reused; europe-latest.osm.pbf names a different file every day.
  2. Record the base’s own sequence. Without it the pin says where to stop but not where to start, and an osmium up-to-date run from the wrong base silently produces a different state.
  3. Resolve a date to a sequence exactly once. The search costs a handful of requests and the answer is an integer; re-resolving at read time reintroduces the ambiguity the pin was meant to remove.
  4. Use the replication URL as part of the pin. Minutely, hourly and daily streams have independent numbering, and a sequence from one is meaningless against another.
  5. Store the pin beside the output. A pin in a runbook belongs to nobody; a pin in the output’s metadata travels with the thing that depends on it.
  6. Verify the base before applying. A mismatched checksum is a failure, not a warning, because everything downstream of it will be subtly wrong rather than obviously broken.
  7. Materialise long-lived pins. Once the diffs behind a sequence are expired, only a stored snapshot can still reconstruct it.
Retention, and when a pin stops being reconstructible A timeline of four marks. At the moment the pin is created, everything needed to rebuild it exists and reconstruction is a routine command. A few weeks later the minutely diffs behind it begin to be expired by the provider, and reconstruction becomes possible only through a coarser hourly or daily stream with correspondingly less precision. Months later the base extract itself has been replaced at its published location, so the checksum recorded in the pin no longer matches anything downloadable. At that point only a materialised snapshot stored by you can still reproduce the state. How long a pin stays reconstructible At creation everything present a routine command Weeks later minutely diffs expiring coarser streams only Months later base file replaced checksum matches nothing After that only your snapshot nothing upstream helps A pin that must survive a year is a snapshot you have to store, and deciding that at creation time is far cheaper than discovering it later.
The pin stays valid as an identifier throughout; what expires is the ability to rebuild from upstream.
What each field of the pin is for, and what breaks without it A grid of five recorded fields against their role in reconstruction and the failure that follows omitting them. The base filename alone identifies nothing, since providers reuse names daily. The base checksum pins the exact starting bytes, and without it a rebuild from a different extract succeeds while producing different data. The base sequence says where the rebuild starts, and without it the apply may begin mid-stream. The target sequence names the state, and a date in its place resolves differently later. The replication URL distinguishes the minutely, hourly and daily streams, whose numbering is independent. Five fields, five silent failures Role Failure without it Base filename human readability names are reused daily Base checksum pins starting bytes different data, no error Base sequence where rebuild starts apply begins mid-stream Target sequence names the state a date drifts over time Replication URL which stream numbering is independent Every failure in the last column is silent: the rebuild succeeds and the data is wrong, so the checks belong in code, not a runbook.
The whole pin is under two hundred bytes and each field removes one way of being quietly wrong.

Verification

  • Two rebuilds agree. Materialise the same pin twice into different files and compare checksums of the sorted output.
  • A wrong base is rejected. Point the rebuild at a different extract and confirm it fails rather than proceeding.
  • The resolved sequence is stable. Re-resolve the same timestamp and confirm the same integer comes back.
  • Feature counts match expectation. Compare against the counts recorded when the pin was created.
  • The pin travels. Confirm the output carries its pin file and that a consumer can find it without asking you.

Common errors and fixes

Symptom Root cause One-line fix
Reruns produce different results State named by date, not sequence Record the resolved sequence integer
Rebuild fails months later Diffs expired from retention Materialise and store the snapshot
Wrong data from the right sequence Base file differed from the original Record and verify the base checksum
Sequence appears out of range Pin taken from a different stream Record the replication URL in the pin
Cannot trace a result to its input Pin kept outside the output Write the pin beside the output file
Rebuild silently starts mid-stream Base sequence not recorded Store the base’s own sequence number
Timestamp resolves differently over time Re-resolving at read time Resolve once at pin creation and freeze it

Specification reference

Each replication directory contains state.txt files giving sequenceNumber and timestamp for the corresponding change file, with the path derived by splitting the zero-padded nine-digit sequence into three-digit components. The timestamp value escapes colons with backslashes. Minutely, hourly and daily replication directories maintain independent sequence numbering. See the OpenStreetMap replication documentation and osmium up-to-date.

Frequently Asked Questions

Why not just archive the extract instead of pinning a sequence?

Archiving works and costs storage proportional to how many pins you keep, which for a planet file is tens of gigabytes each. A sequence pin costs two strings and an integer, and reconstructs on demand while the diffs survive. The right answer is usually both: pin everything, and materialise only the pins that must outlive retention — a published result, a regulatory submission, a paper’s dataset.

How precise is a sequence pin against a timestamp?

To within one diff interval, which for the minutely stream is about a minute and for the daily stream is a day. That precision is almost always more than enough, because the question being asked is “the same data as last time”, not “the data at exactly this instant”. The risk is not precision but ambiguity, and recording the integer removes that entirely.

Do regional extract providers publish usable sequence numbers?

The major ones do, alongside their extracts, and the number refers to the upstream planet stream rather than a provider-specific one — which is what makes osmium up-to-date work against a regional file. Providers that publish extracts without state files leave you pinning by checksum only, which is reproducible but not advanceable, and that is worth knowing before choosing a provider for work that needs reproducibility.

Should the pin include the tooling version?

Yes, if the output depends on it, which it usually does. The same input processed by different osmium or library versions can produce different geometry in edge cases, so a pin that reproduces the input but not the environment reproduces less than it appears to. Recording the tool versions alongside the sequence is the same two-string cost and closes the gap.

Up one level: Replication Sequence Numbers & State Tracking.