OSM Replication & Diff Sync Jump to heading#

An OpenStreetMap extract is a photograph of a moving subject. The moment osmium extract or a Geofabrik download writes a .osm.pbf to disk, thousands of contributors begin editing the objects it contains — a road is split, a building’s footprint is corrected, a closed shop is deleted. Within a day a regional file already disagrees with the live database in tens of thousands of places; within a month it is stale enough that routing, geocoding, and completeness metrics built on top of it quietly drift out of true. This section is the reference for closing that gap: it explains how OSM publishes its edit history as a stream of diffs, how you consume that stream in the correct order, and how you apply it to a local .osm.pbf or PostGIS database so your copy tracks upstream to the minute instead of aging in place. It builds directly on the format and model groundwork in OSM Data Fundamentals & Architecture; if the byte layout of a PBF blob is unfamiliar, start there first.

The scope is deliberately the update loop — the mechanics of turning a fixed snapshot into a living dataset without re-downloading a planet file every night. Full re-imports are simple but wasteful: a planet PBF is roughly 80 GB and an osm2pgsql import of it takes many hours, whereas a minute of global change is a few hundred kilobytes that applies in seconds. The engineering payoff of replication is that update cost tracks change volume, not database size. Getting it right, though, demands discipline around ordering, state tracking, and provenance, because a single skipped or reordered diff corrupts state silently and is often not noticed until a downstream query returns a road that was deleted weeks ago.

The OsmChange Diff Format Jump to heading#

The unit of replication is the OsmChange document, distributed gzip-compressed with the extension .osc.gz. It is an XML file whose root <osmChange> element contains three kinds of operation block — <create>, <modify>, and <delete> — each wrapping full OSM elements (<node>, <way>, <relation>) exactly as they appear in a normal OSM XML file, with their id, version, timestamp, changeset, and, for modifications and deletions, a visible attribute. A <modify> block carries the entire new state of the element, not a field-level patch; consumers replace the prior version wholesale. A <delete> block carries the element stub with visible="false" at its new version number. This whole-object replacement semantics is what makes application idempotent-by-version: applying the same diff twice yields the same result, provided you respect version numbers.

Two properties of the format govern correct application. First, version monotonicity: every operation names a version, and an element’s version increments by one on each edit. A diff that modifies a node to version 8 is only valid against a base holding version 7; applying it to a base still at version 5 means you have missed versions 6 and 7. Second, intra-file ordering is not spatial or type-grouped in a way you can rely on — a single .osc may create a node, then modify a way that references it, so a naive two-pass loader that resolves references eagerly can trip over forward references within one file. Robust appliers process creates, then modifies, then deletes, or defer reference resolution until the whole diff is ingested. The detailed field reference and the create/modify/delete application rules live in Applying .osc Change Files with osmium.

Replication Streams and Cadence Jump to heading#

OSM publishes its change stream at three cadences, each a directory tree of numbered diffs on a replication server (the canonical one being planet.openstreetmap.org/replication/, mirrored by Geofabrik for regional extracts):

The three streams are not independent datasets — the hourly and daily diffs are aggregations of the same edits carried by the minutely stream, so you choose exactly one cadence per pipeline and never mix them for a single database. A common design error is to catch up a weeks-behind extract from the minutely stream, which would mean fetching tens of thousands of tiny files; the daily stream covers the same span in a few dozen large diffs and is far cheaper for a long catch-up, after which you switch to minutely for steady-state tracking. Geofabrik additionally publishes region-scoped .osc.gz streams so you can update a country extract without processing global change, though those streams update less frequently and carry their own sequence numbering.

Sequence Numbers and state.txt Jump to heading#

Ordering in OSM replication is enforced by a single monotonically increasing integer: the sequence number. Every diff in a stream is identified by its sequence, and the server publishes a companion state.txt alongside each diff and at the stream root. A state.txt is a tiny key-value file:

#Wed Jul 14 09:03:02 UTC 2026
sequenceNumber=6543210
timestamp=2026-07-14T09\:00\:00Z

The sequenceNumber is the identity of the most recent diff, and timestamp is the instant up to which that diff is complete — meaning every edit with a timestamp at or before that value is reflected once the diff is applied. The sequence number also maps directly onto the diff’s path: OSM formats the integer as a nine-digit, zero-padded, slash-segmented path, so sequence 6543210 lives at 006/543/210.osc.gz with its own 006/543/210.state.txt. Your pipeline’s core state is therefore just one integer — the last sequence you successfully applied. Recording it durably (alongside the file checksum and applied-at timestamp) is what makes the loop resumable and auditable. A PBF that has itself been kept current carries this integer in its header as osmosis_replication_sequence_number, which you can read back with the technique in How to decode OSM PBF headers in Python rather than tracking it in a sidecar. The full mechanics — deriving the starting sequence, mapping timestamps to sequences, and detecting gaps — are covered in Replication Sequence Numbers and State.

Applying Diffs: osmium and pyosmium Jump to heading#

Once you hold an ordered list of .osc.gz files, application is a merge of the diff stream into the base data. The workhorse for file-to-file updates is the osmium command-line tool. Its apply-changes subcommand consumes a base .osm.pbf and one or more change files and writes a new PBF reflecting the merged state:

# Apply a single minutely diff to a base extract, writing a new PBF.
osmium apply-changes base.osm.pbf 006543211.osc.gz \
  --output updated.osm.pbf

# Apply several ordered diffs in one pass (osmium sorts internally by object,
# but you must pass them in ascending sequence order for correct versioning).
osmium apply-changes base.osm.pbf \
  006543211.osc.gz 006543212.osc.gz 006543213.osc.gz \
  --output updated.osm.pbf

For programmatic control — fetching diffs, applying them, and reacting to failures inside one process — pyosmium exposes the same machinery. Its osmium.replication.server.ReplicationServer class talks to a replication URL, resolves your last sequence into a set of diffs, and streams their changes into a handler, while apply_diffs drives the whole catch-up. The concrete recipe for bringing a stale file current in Python is Catching Up a Stale OSM Extract with pyosmium. Whichever path you take, the invariant is identical: diffs apply in ascending sequence order, and each successful application advances your recorded sequence by exactly the span consumed. Wrapping application in the defensive-decoding and quarantine discipline from Error Handling in Large OSM Extracts keeps a single malformed change from aborting an otherwise healthy run.

Full-History Files and Temporal Snapshots Jump to heading#

Replication keeps the current state current, but a whole class of work needs the past: what did this neighbourhood look like a year ago, when was this building first mapped, how has road coverage grown. For that OSM distributes full-history files with the .osh.pbf extension. Where a standard PBF holds only the latest visible version of each element, an .osh.pbf retains every version — each carrying its version, timestamp, changeset, and visible flag — and declares the HistoricalInformation feature in its header so parsers know to expect multiple versions per (type, id). The file is, in effect, the accumulated diff stream folded back into a single object store keyed on (type, id, version).

Reconstructing the map as it stood at an instant is then a filtering operation: for each element, select the highest version whose timestamp is at or before the target instant and whose visible flag is true. The osmium tool packages this as time-filter:

# Reconstruct the visible state of the map as of a past instant.
osmium time-filter history.osh.pbf 2025-01-01T00:00:00Z \
  --output snapshot-2025.osm.pbf

Full-history processing is memory- and CPU-heavy because the version dimension multiplies object counts several-fold, and it interacts with the spatial layer — reconstructing geometry at a past date means resolving node versions, not just node IDs, so the incremental index-maintenance patterns in Spatial Indexing for OSM Extracts need a temporal key. The dedicated reference Full-History .osh.pbf Processing covers version selection, changeset attribution, and the snapshot-reconstruction algorithm in depth.

Building an Automated Update Pipeline Jump to heading#

A production replication pipeline is a small state machine wrapped in a scheduler, and its correctness rests on a few non-negotiable properties. It must be resumable: interrupted mid-cycle, the next run continues from the last durably recorded sequence, never re-applying or skipping. It must be atomic at the state boundary: the recorded sequence advances only after the corresponding data write has been committed, so a crash between apply and record leaves the data behind the sequence, not ahead of it — behind is safe (the diff re-applies harmlessly under version semantics), ahead is silent data loss. And it must be idempotent: re-running a cycle over already-applied diffs produces identical state, which the version-numbered whole-object replacement of OsmChange guarantees as long as you never skip.

The end-to-end assembly — a fetch-apply-record loop, a lock to prevent overlapping runs, checksum verification, and a scheduler to fire it every minute — is the subject of Building a Minutely Update Pipeline. Two failure modes deserve naming here because they define the pipeline’s hard edges. A sequence gap occurs when your recorded sequence is so far behind that the stream has rotated the intervening diffs out of retention, or when a run advanced state without a matching data commit; recovery means re-anchoring against a fresh base rather than trying to fetch missing diffs. A poison diff — one that fails to apply because the base is at the wrong version — signals that your state and data have diverged and must be reconciled before the loop resumes. Both branch out of the steady-state loop into the gap-recovery path drawn in the overview diagram above.

Provenance and ODbL for Derived State Jump to heading#

Replication changes the licensing picture in a way a static extract does not: a continuously updated database is a derivative database under the Open Database License (ODbL), and its share-alike and attribution obligations attach to the state at every point in time, not just at initial import. That makes provenance an engineering requirement, not a footnote. Every update cycle should append to an immutable ledger: the sequence number applied, the source stream URL, the state.txt timestamp, and the checksum of the diff file. This ledger is simultaneously your reproducibility record (any past state can be rebuilt by replaying the ledger from a known base) and your compliance record (you can prove, for any published extract, exactly which upstream edits it incorporates and attribute “© OpenStreetMap contributors” against a dated source).

The keep-open clause of the ODbL means that if you redistribute the updated database you cannot layer technical restrictions on it, and the share-alike clause means adaptations you publish inherit the licence — both of which are far easier to satisfy when provenance is stamped automatically at each cycle than when reconstructed after the fact. The authoritative obligations are the official OpenStreetMap Copyright & License terms; pin your interpretation to a dated copy in the same ledger that records your sequences, so licence state and data state are auditable together.

Explore Replication & Diff Sync in Depth Jump to heading#

Each reference below drills into one stage of the update loop introduced above:

• Applying .osc Change Files with osmium — the OsmChange format field-by-field and how apply-changes and pyosmium merge create/modify/delete blocks into a base.
• Replication Sequence Numbers and State — how sequence numbers, state.txt, and header anchors track exactly how current your data is and how to detect gaps.
• Full-History .osh.pbf Processing — retaining every element version and reconstructing the map at any past instant with time-filter.
• Building a Minutely Update Pipeline — assembling a resumable, atomic, scheduled fetch-apply-record loop for steady-state tracking.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• OSM Data Fundamentals & Architecture — the data model, PBF format, and validation gates that replication builds upon.
• PBF File Structure Deep Dive — the block and blob layout that both base extracts and applied output are written in.
• How to Decode OSM PBF Headers in Python — reading the replication sequence number stored in a PBF header.
• Spatial Indexing for OSM Extracts — incremental index maintenance as diffs mutate geometry.
• Error Handling in Large OSM Extracts — quarantine and defensive decoding for the diffs an update loop consumes.

This section anchors the diff-sync side of the OSM knowledge base; return to the site home to explore the fundamentals, parsing, and quality-assurance pipelines that a continuously updated dataset feeds.