Catching Up a Stale OSM Extract with pyosmium

You have an .osm.pbf that was current three weeks ago and needs to reach today, and the only trace of where it stands is a timestamp — so you must find the replication sequence that timestamp corresponds to, then fetch and apply every diff from there to the stream head, in order.

Prerequisites

Conceptual minimum

A replication stream is an append-only sequence of numbered diffs, and “catching up” is nothing more than replaying the contiguous run of diffs between where your file stands and the stream’s current head. The one non-obvious part is finding the starting point. Your extract knows a timestamp, but diffs are addressed by integer sequence, so you need a timestamp-to-sequence lookup. pyosmium’s ReplicationServer.timestamp_to_sequence does exactly that: it binary-searches the stream’s state.txt files to find the sequence whose completeness time brackets your timestamp. Once you hold that starting sequence, apply_diffs streams every subsequent change through libosmium’s version-aware merge — the same create/modify/delete semantics detailed in the parent guide, Applying .osc Change Files with osmium — and reports the sequence it reached.

Catch-up cost against how stale the extract is A bar chart of minutes to catch up a country extract from the minutely stream, by staleness. One hour behind is 60 diffs and about 40 seconds. One day is 1440 diffs and about 16 minutes. One week is 10 080 diffs and about 1.9 hours. One month is 43 200 diffs and about 8 hours, at which point re-downloading a fresh 1.2 GB extract takes 4 minutes and is strictly faster. Past about ten days, re-downloading beats replaying wall-clock minutes to bring a 1.2 GB country extract current from the minutely stream 1 hour behind 60 diffs · 42 s 1 day behind 1 440 diffs · 16 min 1 week behind 10 080 diffs · 1.9 h 1 month behind 43 200 diffs · 8 h fresh download instead 1.2 GB over HTTPS · 4 min Compute the crossover for your own link speed once and encode it as a threshold in the catch-up script, so the decision is not made under pressure at 03:00.
There is a crossover, and it is worth computing rather than guessing. Past roughly ten days of staleness the fresh download wins on every axis — time, bandwidth and the risk of hitting a diff that has aged out.

The critical choice is cadence for the catch-up itself. A three-week gap on the minutely stream is roughly 30,000 tiny files; on the hourly stream it is a few hundred; on the daily stream a few dozen. Use the coarsest stream that still lands you close enough to live, catch up on that, then switch to your steady-state stream for ongoing tracking. This is why the diagram below frames catch-up as a bounded loop over a sequence gap, not an open-ended poll.

Catch-up loop closing the gap between a stale sequence and the stream head The stale extract's timestamp maps to a start sequence. A loop fetches the diff at the current sequence, applies it, increments, and repeats until the head sequence is reached, at which point the extract is current. Replay the contiguous gap, one sequence at a time Stale extract timestamp → seq S Fetch diff seq = current Apply + incr seq += 1 seq == head? yes → current no → next sequence

Runnable solution

The script below finds the starting sequence from the extract’s timestamp, then drives apply_diffs to write a current file. apply_diffs handles the fetch-apply loop internally and returns the sequence it stopped at, which you persist for steady-state tracking.

python 
from __future__ import annotations

import datetime as dt
import logging

import osmium
from osmium.replication.server import ReplicationServer

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.catchup")

# Coarse stream for a long gap; switch to minute/ for steady-state after.
REPL_URL = "https://planet.openstreetmap.org/replication/hour/"


def find_start_sequence(server: ReplicationServer, when: dt.datetime) -> int:
    """Resolve the replication sequence current as of a timestamp.

    timestamp_to_sequence binary-searches the stream's state files. Subtract one
    so the first applied diff re-covers the boundary interval rather than skipping
    edits that landed within the extract's last partial window.
    """
    seq = server.timestamp_to_sequence(when)
    if seq is None:
        raise RuntimeError(f"no replication sequence found for {when.isoformat()}")
    start = max(seq - 1, 0)
    logger.info("timestamp %s maps to start sequence %d", when.isoformat(), start)
    return start


def catch_up(base: str, out: str, since: dt.datetime, max_diffs: int = 1000) -> int:
    """Bring a stale extract current and return the sequence reached."""
    server = ReplicationServer(REPL_URL)
    try:
        start = find_start_sequence(server, since)
        writer = osmium.SimpleWriter(out)
        try:
            # apply_diffs fetches and merges every diff from start+1 forward,
            # up to max_diffs, streaming create/modify/delete into the writer.
            reached = server.apply_diffs(
                writer, start, max_size=max_diffs, idx="flex_mem", simplify=True
            )
        finally:
            writer.close()
    finally:
        server.close()

    if reached is None:
        logger.warning("already current; no diffs to apply from sequence %d", start)
        return start
    logger.info("caught up: applied through sequence %d", reached)
    return reached


if __name__ == "__main__":
    # The extract was complete as of this instant (read from its PBF header).
    complete_as_of = dt.datetime(2026, 6, 23, 0, 0, tzinfo=dt.timezone.utc)
    final_seq = catch_up("stale.osm.pbf", "current.osm.pbf", complete_as_of)
    logger.info("record sequence %d as the new steady-state anchor", final_seq)

CLI alternative with pyosmium-get-changes

When you would rather fetch the merged diff separately and apply it with osmium, the pyosmium-get-changes command downloads the gap into a single .osc.gz:

bash 
# Fetch every change since the extract's sequence into one merged diff,
# then apply it to the base with osmium.
pyosmium-get-changes \
  --server https://planet.openstreetmap.org/replication/hour/ \
  --start-date 2026-06-23T00:00:00Z \
  --size 4096 \
  -o catchup.osc.gz

osmium apply-changes stale.osm.pbf catchup.osc.gz \
  --output current.osm.pbf --overwrite

Step-by-step walkthrough

  1. Pick the catch-up stream. REPL_URL targets the hourly stream so a multi-week gap is hundreds of diffs, not tens of thousands; you will re-point at minute/ only once the file is near-live.
  2. Map timestamp to sequence. find_start_sequence calls timestamp_to_sequence, which binary-searches state.txt files, then subtracts one so the boundary window is re-covered rather than skipped — replaying a diff is a safe no-op under version semantics.
  3. Drive apply_diffs. server.apply_diffs(writer, start, ...) fetches each diff after start and streams its objects through libosmium’s merge into the SimpleWriter; simplify=True collapses multiple versions of the same object within the batch to the latest, which is what a current-state output wants.
  4. Bound the batch. max_size caps how far one call advances, so a very large gap is processed in bounded chunks instead of one unbounded download; call catch_up in a loop until reached stops advancing to finish a huge gap.
  5. Close in order. The SimpleWriter is closed before the server so the output file is flushed completely, and the server connection is released in the outer finally.
  6. Record the anchor. The returned sequence becomes your steady-state starting point — persist it exactly as Replication Sequence Numbers and State describes.
The four states of a catch-up loop and the transition each one takes A left-to-right chain of loop states. Read the local sequence from the checkpoint. Compare it against the stream head from state.txt. If behind, fetch and apply the next diff, then write the new checkpoint before looping. When the local sequence equals the head, the extract is current and the loop exits. The checkpoint write is placed after the apply and before the loop, which is what makes a crash resumable rather than corrupting. Apply, then checkpoint — a crash between them must lose work, never invent it read checkpoint local seq S from disk, not memory read stream head state.txt → seq H one HTTP GET apply diff S+1 to the working file atomic rename on success write checkpoint S := S+1 fsync before looping A crash after the apply and before the checkpoint replays one diff. Because application is idempotent by version, replaying one is free; skipping one is not.
The ordering inside the loop is the entire correctness argument: apply, then checkpoint. Reverse those two and a crash between them leaves a checkpoint claiming work that was never done.

Verification

  • Sequence advanced. The final log line reports a sequence far above the start; if reached equals start, the extract was already current or the timestamp resolved past the head.
  • Header is fresh. Run osmium fileinfo -e current.osm.pbf and confirm the reported osmosis_replication_timestamp is within one diff interval of now.
  • Object counts moved. Compare osmium fileinfo node/way/relation counts before and after; a three-week catch-up on an active region changes counts by a visible margin.
  • Re-run is a no-op. Running catch_up again immediately should return the same sequence and add nothing, proving idempotency.

Common errors and fixes

Symptom Root cause One-line fix
timestamp_to_sequence returns None Timestamp older than the stream’s retention Re-anchor from a fresh base extract instead of catching up
Catch-up never finishes Minutely stream over a multi-week gap Use the hourly or daily stream for the bulk of the gap
Output missing recent edits Timestamp resolved one window too late Subtract one from the resolved start sequence
HTTP 404 on a diff Wrong replication base URL for the region Point --server / REPL_URL at the matching stream
Memory climbs on a large batch In-memory location index on a big region Keep idx="flex_mem" bounded via smaller max_size chunks
Duplicate versions in output simplify=False on a current-state target Set simplify=True to collapse to latest per object

Specification reference

The pyosmium replication client resolves timestamps to sequence numbers and applies change files through libosmium. See the official pyosmium documentation for osmium.replication.server.ReplicationServer covering timestamp_to_sequence and apply_diffs, and the OSM Wiki “Planet.osm/diffs” page for the replication directory layout and state.txt format the client reads.

Frequently Asked Questions

How do I find the sequence my stale extract corresponds to?

Read the extract’s completeness timestamp from its PBF header, then call ReplicationServer.timestamp_to_sequence with that timestamp. It binary-searches the stream’s state.txt files and returns the sequence whose completeness time brackets your timestamp. Subtract one from the result so the boundary window is re-applied rather than skipped.

Should I catch up over the minutely stream?

Not for a large gap. A multi-week gap on the minutely stream is tens of thousands of tiny files. Use the hourly or daily stream to cover the bulk cheaply, then switch to minutely for steady-state tracking once the file is within a day or two of live.

What if the timestamp is older than the stream retains?

timestamp_to_sequence returns None because the diffs no longer exist to fetch. You cannot catch up incrementally past the retention window; download a fresh base extract with a recent header sequence and resume steady-state tracking from there.

Is it safe to re-run the catch-up script?

Yes. Applying diffs is version-aware whole-object replacement, so replaying diffs already merged is a no-op. Re-running from the same start sequence produces identical output and simply reports the same reached sequence, which is why the boundary-window overlap is harmless.

Up one level: Applying .osc Change Files with osmium.