Recovering from a Replication Sequence Gap Jump to heading#

A diff-sync process reports a local applied sequence that is more than one step behind the upstream cursor, or a diff was applied out of order — and you need to tell whether a change file was skipped and, if so, replay the missing range safely without corrupting the object-version history.

Prerequisites Jump to heading#

• pyosmium ≥ 3.6 installed (pip install "osmium>=3.6") for fetching and applying .osc.gz diffs.
• requests ≥ 2.31 for reading the upstream root cursor and per-sequence state.txt files.
• A durable checkpoint recording the last fully applied sequence — the atomic checkpoint from Replication Sequence Numbers & State Tracking.
• Access to the base extract or a way to re-download one, for the reset-from-scratch branch.
• The correct replication base URL for your stream, per OSM Replication & Diff Sync.
• Python 3.10+ for the match statement and union hints used below.

Conceptual minimum Jump to heading#

Replication offers no notification that a step was missed. The protocol guarantees only that if you apply every sequence in ascending order exactly once, your data converges to upstream; violate that — skip 455, or apply 456 before 455 — and the local database silently diverges, because OSM diffs are not commutative. Each .osc.gz encodes creates, modifications, and deletes keyed to specific object versions, so applying them out of order can, for example, try to modify an object a later diff already deleted, or resurrect one an earlier diff removed. The only detector is your own bookkeeping: the gap is the difference between the last sequence you recorded as applied and the sequence upstream reports as current.

Recovery has two shapes. When the missing range is small and still published, the fix is to replay every sequence from checkpoint + 1 up to the upstream cursor, strictly in order — this is a normal catch-up, just triggered by gap detection rather than a schedule. When the local state is already inconsistent (a diff was applied out of order, or the missing range has aged out of the feed’s retained history), no replay can repair it, and the correct move is to reset from a fresh base extract whose PBF header carries a known-good replication anchor, discarding the diverged state entirely. Distinguishing the two is what this page is about; the actual application of diffs is covered in applying OSC change files with osmium, and the numbering it relies on comes from the parent guide, Replication Sequence Numbers & State Tracking.

Runnable solution Jump to heading#

The module reads both positions, classifies the gap, and either replays the missing range in order or signals that a reset is required. It never applies a diff without advancing the checkpoint immediately after, so a crash mid-replay resumes correctly.

from __future__ import annotations

import logging
from dataclasses import dataclass
from datetime import datetime, timezone

import requests
import osmium.replication.server as rserv

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.replication.gap")

MINUTE_URL = "https://planet.openstreetmap.org/replication/minute"


@dataclass(frozen=True)
class GapReport:
    local: int
    upstream: int
    oldest_available: int

    @property
    def gap(self) -> int:
        return self.upstream - self.local

    @property
    def action(self) -> str:
        if self.gap < 0:
            return "reset"          # local ahead of upstream = corrupt checkpoint
        if self.gap == 0:
            return "in_sync"
        if self.local + 1 < self.oldest_available:
            return "reset"          # missing range has aged out of the feed
        return "replay"


def _root_sequence(base_url: str) -> int:
    resp = requests.get(f"{base_url.rstrip('/')}/state.txt", timeout=30)
    resp.raise_for_status()
    for line in resp.text.splitlines():
        if line.startswith("sequenceNumber="):
            return int(line.split("=", 1)[1].strip())
    raise ValueError("no sequenceNumber in root state.txt")


def diagnose_gap(local_seq: int, base_url: str, oldest_available: int) -> GapReport:
    """Compare the local checkpoint against upstream and classify the gap."""
    upstream = _root_sequence(base_url)
    report = GapReport(local=local_seq, upstream=upstream, oldest_available=oldest_available)
    logger.info(
        "local=%d upstream=%d gap=%d -> action=%s",
        report.local, report.upstream, report.gap, report.action,
    )
    return report


def replay_range(report: GapReport, base_url: str, apply_one, save_checkpoint) -> int:
    """Apply every missing sequence in ascending order, checkpointing each step.

    ``apply_one(seq)`` fetches and applies one diff to the target store.
    ``save_checkpoint(seq)`` durably records the last applied sequence.
    Returns the final applied sequence.
    """
    if report.action != "replay":
        raise RuntimeError(f"replay refused: action is {report.action!r}, not 'replay'")

    current = report.local
    for seq in range(report.local + 1, report.upstream + 1):
        apply_one(seq)             # MUST apply strictly in this ascending order
        save_checkpoint(seq)       # advance only after a successful apply
        current = seq
        logger.info("replayed sequence %d (%d remaining)", seq, report.upstream - seq)
    return current

Step-by-step walkthrough Jump to heading#

1. Read the local checkpoint. The last fully-applied sequence comes from the durable checkpoint written by the sync loop; treat a missing checkpoint as “reset required,” not as sequence zero.
2. Read the upstream cursor. _root_sequence fetches the cadence’s root state.txt — a single request — to learn how far upstream has advanced.
3. Classify with GapReport.action. A gap of zero is in-sync; a positive gap whose missing range is still published is replay; a negative gap (local ahead of upstream) means the checkpoint is corrupt; and a missing range older than the feed’s retained history is reset.
4. Refuse to replay a diverged state. replay_range raises unless the action is exactly replay, so an out-of-order or aged-out situation can never be papered over by re-running diffs that will not repair it.
5. Apply strictly ascending, checkpoint per step. The loop applies local + 1 through upstream in order and advances the checkpoint after each successful apply, so a crash resumes at the next unapplied sequence rather than restarting the range.
6. Return the final sequence. The caller compares the returned value against the upstream cursor to confirm the gap is closed.

Verification Jump to heading#

• Recompute the gap after replay. Call diagnose_gap again; action should read in_sync (or apply next if upstream advanced during the replay) and the gap should be zero or one.
• Spot-check a boundary object. Pick an object edited inside the replayed range and confirm its local version matches the value in the last replayed diff — proof the range applied in order.
• Confirm monotonic checkpoints. The checkpoint’s sequence must only ever increase; a log showing it decrement means an out-of-order apply and mandates a reset.
• Reject the reset path silently succeeding. If action is reset, replay must raise rather than log success — verify the RuntimeError fires so a diverged database is never reported as recovered.
• Watch the countdown log. The remaining counter in each replay line should decrease by one every step; a stall or jump signals a fetch failure mid-range.

Common errors and fixes Jump to heading#

Specification reference Jump to heading#

Applying OsmChange diffs out of sequence corrupts state because each diff references specific object versions; the replication model requires strictly ordered, once-only application, as documented on the OSM wiki under Planet.osm/diffs and in the OsmChange format specification. The pyosmium replication client used to fetch and apply ranges is described in the pyosmium replication reference.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Replication Sequence Numbers & State Tracking — the checkpoint and state.txt semantics gap detection depends on.
• Finding the Replication Sequence for a Timestamp — resolving a start sequence when reseeding from a base extract.
• Applying OSC Change Files with Osmium — the mechanics of applying the diffs a replay fetches.
• Catching Up a Stale OSM Extract with Pyosmium — the ordered catch-up loop replay reuses.
• How to Decode OSM PBF Headers in Python — reading the replication anchor from a fresh base extract during a reset.

Up one level: Replication Sequence Numbers & State Tracking.