Replication Sequence Numbers & State Tracking Jump to heading#

Every OpenStreetMap diff-sync pipeline lives or dies by a single integer. The replication sequence number is the monotonic counter that names each published change file, and the moment a pipeline loses track of which number it has already applied, correctness collapses in one of two silent ways: it re-applies a diff it already consumed (double-counting edits, resurrecting deleted objects) or it skips one entirely (leaving a permanent hole in the local database that no later diff will ever fill). Neither failure raises an exception — the pipeline keeps running, the row counts keep climbing, and the divergence from upstream only surfaces weeks later when a routing graph has a road that was deleted in reality or a building that upstream never had. This guide is the bookkeeping layer of the OSM Replication & Diff Sync section: it defines exactly what a sequence number is, where the state.txt that carries it lives, how the number encodes a directory path, and how to persist your applied position so a crashed process resumes on the correct diff rather than guessing.

Prerequisites: What You Need in Place Jump to heading#

This reference assumes the surrounding machinery of a diff-sync loop is already understood. The change files that sequence numbers name are the .osc.gz payloads consumed in applying OSC change files with osmium; this page only concerns the numbering and state that decide which of those files to fetch and in what order. You will also want the header-reading skills from how to decode OSM PBF headers in Python, because the replication anchor that seeds a fresh sync lives in the PBF header block rather than in any sidecar file. Finally, a working knowledge of the broader OSM Replication & Diff Sync layer helps place sequence tracking in the context of a full incremental-update pipeline. Everything below is Python 3.10+ with the standard logging pattern.

The state.txt Format Jump to heading#

Each replication interval publishes two files per step: the compressed diff (NNN.osc.gz) and a plain-text state file (NNN.state.txt) that records what that diff represents. The state file is a Java .properties document — the format Osmosis and the replication tooling emit — and it carries exactly two fields that matter to a sync loop, plus a comment line holding the file’s own generation time.

#Sat Jul 11 00:00:02 UTC 2026
sequenceNumber=6123456
timestamp=2026-07-11T00\:00\:00Z

Two encoding details bite the unwary. First, the timestamp is an ISO 8601 instant in UTC, but because the file follows Java properties escaping, the colons in the time portion are prefixed with backslashes (00\:00\:00). A naive line-split-on-= parser leaves those backslashes in place and every downstream datetime.fromisoformat call then raises. Second, the leading # line is a human-readable comment describing when the file was written, which is a second or two after the timestamp value it documents — never parse the comment as authoritative; the timestamp= field is the semantic instant the diff brings the data up to.

The timestamp field answers “what wall-clock moment is this data current as of,” and the sequenceNumber answers “what is the ordinal name of this step.” The two are strictly monotonic together: a higher sequence number always corresponds to a later or equal timestamp. That monotonicity is the invariant every gap check and checkpoint relies on.

The Replication Directory Layout Jump to heading#

OSM publishes diffs at three cadences under planet.openstreetmap.org/replication/ — minute/, hour/, and day/ — and each cadence has its own independent sequence counter. Within a cadence, the sequence number is not used as a flat filename; it is exploded into a three-level directory tree so that no single directory ever holds millions of entries. The rule is fixed: left-pad the integer to nine digits, then cut it into three groups of three.

A sequence like 6123456 becomes the padded string 006123456, which splits into 006, 123, 456, giving the path 006/123/456. Under the minutely stream that resolves to two URLs:

https://planet.openstreetmap.org/replication/minute/006/123/456.state.txt
https://planet.openstreetmap.org/replication/minute/006/123/456.osc.gz

There is also a single top-level state.txt at the root of each cadence directory (…/minute/state.txt) that always reflects the newest published sequence — that is the file you poll to discover how far upstream has advanced. The per-sequence state files are the historical record; the top-level one is the live cursor.

The PBF Replication Anchor Jump to heading#

A freshly downloaded extract — a Geofabrik regional .osm.pbf or a planet dump — is a snapshot, not a stream, so it needs to tell a sync loop where in the replication timeline it sits. It does this through three optional key-value entries in the PBF HeaderBlock, written by whatever tool cut the extract:

• osmosis_replication_sequence_number — the sequence the snapshot was current as of. Start applying diffs from the next sequence.
• osmosis_replication_timestamp — the ISO 8601 instant matching that sequence, the same semantics as the timestamp field in state.txt.
• osmosis_replication_base_url — the replication stream this snapshot descends from, e.g. https://download.geofabrik.de/europe/monaco-updates or the planet minutely URL. This tells you which stream’s counter the sequence number refers to, which matters because a Geofabrik regional extract has its own per-region replication feed with its own numbering, distinct from the planet feed.

Reading these three values is the correct way to seed a pipeline from a base extract; the mechanics of pulling them out of the header live in decoding OSM PBF headers in Python. If a header is missing the sequence anchor entirely — some third-party extracts strip it — you fall back to deriving a starting sequence from the header timestamp, which is exactly the lookup covered in finding the replication sequence for a timestamp.

Step-by-Step: Computing the URL Path in Python Jump to heading#

The path derivation is pure arithmetic on the integer, so it is worth having a single tested function rather than string-formatting URLs inline. The steps:

1. Validate the sequence. Reject negatives and non-integers before formatting; a malformed sequence produces a URL that 404s and the failure is easier to read at the source.
2. Zero-pad to nine digits. f"{seq:09d}" guarantees the three-group split is always well-formed even for small sequences.
3. Slice into thirds. Characters 0–3, 3–6, 6–9 become the two directory levels and the file stem.
4. Join under the cadence base URL. Append .state.txt or .osc.gz to the stem.

import logging
from urllib.parse import urljoin

logger = logging.getLogger("osm.replication.paths")

MINUTE_BASE = "https://planet.openstreetmap.org/replication/minute/"


def sequence_to_fragment(seq: int) -> str:
    """Map a replication sequence number to its 3/3/3 path fragment.

    6123456 -> '006/123/456'. Raises ValueError on invalid input so a bad
    sequence fails at the source rather than as an opaque 404 later.
    """
    if not isinstance(seq, int) or seq < 0:
        raise ValueError(f"sequence must be a non-negative int, got {seq!r}")
    padded = f"{seq:09d}"
    return f"{padded[0:3]}/{padded[3:6]}/{padded[6:9]}"


def sequence_urls(seq: int, base_url: str = MINUTE_BASE) -> dict[str, str]:
    """Return the state.txt and osc.gz URLs for a sequence under a base URL."""
    fragment = sequence_to_fragment(seq)
    base = base_url if base_url.endswith("/") else base_url + "/"
    return {
        "state": urljoin(base, f"{fragment}.state.txt"),
        "change": urljoin(base, f"{fragment}.osc.gz"),
    }


if __name__ == "__main__":
    urls = sequence_urls(6_123_456)
    logger.info("state url: %s", urls["state"])
    logger.info("change url: %s", urls["change"])

Parsing a fetched state.txt back into the two fields is the inverse operation, and it must undo the properties escaping:

from datetime import datetime, timezone


def parse_state_txt(text: str) -> dict[str, object]:
    """Parse a replication state.txt into {'sequence': int, 'timestamp': datetime}."""
    fields: dict[str, str] = {}
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue          # skip the human-readable comment line
        key, sep, value = line.partition("=")
        if sep:
            fields[key.strip()] = value.strip().replace("\\:", ":")

    ts = datetime.fromisoformat(fields["timestamp"].replace("Z", "+00:00"))
    if ts.tzinfo is None:
        ts = ts.replace(tzinfo=timezone.utc)
    return {"sequence": int(fields["sequenceNumber"]), "timestamp": ts}

Validation & Error-Handling Matrix Jump to heading#

Sequence and state handling fails in characteristic ways. Each row is a real fault seen in production sync loops, with how to catch and fix it.

Performance & Scale Considerations Jump to heading#

The cost of sequence tracking is dominated not by the arithmetic but by the network round-trips to discover and fetch state. Polling the root cursor is a single small request; only fetch per-step state.txt files when you actually need the timestamp of a specific historical step, because the cursor plus arithmetic already tells you every intermediate path. When catching up a badly stale extract that is thousands of sequences behind, avoid fetching each step’s state.txt — compute the path fragments directly and request only the .osc.gz payloads in order, reserving state.txt reads for the endpoints of the range. The minutely stream advances roughly 1,440 sequences per day, so a week-stale extract is around 10,000 diffs; at that scale the difference between one request per step and two is a multi-hour swing.

Because the path fragment is a deterministic pure function of the integer, it caches trivially and needs no coordination across workers. If you parallelize the download of a large catch-up range, keep the apply strictly sequential in ascending sequence order — the diffs are not commutative, and applying 456 before 455 corrupts the object version chain. Download concurrently, apply serially.

Failure Modes & Gotchas Jump to heading#

• Cadence counters are independent. The minutely sequence 6123456 and the daily sequence 6123456 name completely different points in time. Store the cadence alongside every sequence you persist.
• Nine-digit padding is not future-proof forever, but it is now. The minutely counter passed six million years into the stream; the split assumes ≤ nine significant digits. Do not hard-code an eight-digit split copied from an old tutorial.
• The top-level state.txt can briefly lag its directory. During publication the root cursor and the newest per-step file update within a small window; if a computed URL 404s on a sequence equal to the cursor, retry with a short backoff rather than treating it as a permanent gap.
• Timestamps are UTC, always. Never localize a replication timestamp for comparison; a timezone-naive datetime compared against an aware one raises, and the fix is to attach timezone.utc, not to strip it.
• Regional feeds renumber from their own origin. A Geofabrik extract’s sequence is meaningless against the planet feed and vice versa. The base_url header is the authority on which counter applies.
• A missed diff is silent. Nothing in the protocol tells you a step was skipped; only your own last-applied bookkeeping detects it. Recovering from that condition is the subject of recovering from a replication sequence gap.

Integration Points: Robust Checkpointing Jump to heading#

The output of this stage is a durable record of the last sequence you have fully applied, written so that a crash between fetching and committing a diff never leaves an ambiguous position. The rule is to advance the checkpoint only after the diff is committed to the target store, and to write it atomically so a torn write cannot corrupt it.

import json
import logging
import os
import tempfile
from pathlib import Path

logger = logging.getLogger("osm.replication.checkpoint")


def write_checkpoint(path: Path, sequence: int, timestamp: str, cadence: str) -> None:
    """Atomically persist the last fully-applied replication position.

    Write to a temp file in the same directory, fsync, then os.replace so a
    reader never sees a half-written checkpoint even across a power loss.
    """
    payload = {"sequence": sequence, "timestamp": timestamp, "cadence": cadence}
    directory = path.parent
    fd, tmp = tempfile.mkstemp(dir=directory, suffix=".tmp")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(payload, handle)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(tmp, path)   # atomic on POSIX; overwrites the old checkpoint
        logger.info("checkpoint advanced to %s sequence %d", cadence, sequence)
    except BaseException:
        Path(tmp).unlink(missing_ok=True)
        raise


def read_checkpoint(path: Path) -> dict | None:
    """Return the last applied position, or None on a fresh pipeline."""
    if not path.exists():
        return None
    return json.loads(path.read_text(encoding="utf-8"))

On startup the loop reads the checkpoint, computes last + 1, and derives that sequence’s URL with sequence_to_fragment; on a fresh pipeline it instead reads the PBF replication anchor to seed the first sequence. That checkpoint is precisely the value the catch-up procedure in catching up a stale OSM extract with pyosmium consumes and advances on every applied step.

Sequence & State Topics in Depth Jump to heading#

Two focused guides extend the mechanics above into the tasks that most often go wrong:

• Finding the Replication Sequence for a Timestamp — given a wall-clock datetime (or a PBF that lacks a sequence anchor), resolve the sequence number to start applying diffs from, with a pyosmium helper and a manual binary-search fallback.
• Recovering from a Replication Sequence Gap — detect that a diff was missed or applied out of order by comparing the local checkpoint against upstream, then replay the missing range safely or reset from a fresh base.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Applying OSC Change Files with Osmium — consuming the .osc.gz payloads that sequence numbers name.
• Catching Up a Stale OSM Extract with Pyosmium — the loop that reads and advances the checkpoint defined here.
• How to Decode OSM PBF Headers in Python — extracting the replication anchor from the header block.
• Extracting Metadata from OSM Planet Files — where the header replication keys sit among other planet metadata.
• Finding the Replication Sequence for a Timestamp — resolving a start sequence when only a datetime is known.
• Recovering from a Replication Sequence Gap — detecting and repairing a missed or out-of-order diff.

This guide is part of the OSM Replication & Diff Sync section — return there for the full incremental-update pipeline from base extract to continuously synced database.