Applying .osc Change Files with osmium Jump to heading#

Imagine a nightly job that merges yesterday’s edits into a country extract, and one morning a downstream router starts sending drivers down a road that was deleted last week. The diff that deleted it was fetched but applied against a base that was already a version ahead, so the delete silently no-op’d and the ghost road survived. That is the failure this guide exists to prevent: applying an OsmChange file is not a blind append but a version-aware merge, and getting the version arithmetic wrong corrupts state without raising an error. This reference is part of the broader OSM Replication & Diff Sync section, which frames why an extract must track upstream at all; here the focus narrows to the single operation of taking one or more .osc.gz files and a base .osm.pbf and producing a correct, updated PBF.

Prerequisites Jump to heading#

Applying diffs assumes a few pieces are already in place. You need a base extract whose replication sequence you know — read it from the header per How to Decode OSM PBF Headers in Python, or from a sidecar you maintain. You need the ordered set of .osc.gz files that follow that sequence, which the sibling reference Replication Sequence Numbers and State explains how to resolve. And because the output is another PBF, a working knowledge of the container’s block structure from the PBF File Structure Deep Dive helps when you inspect or validate what apply-changes writes.

The tooling itself: install osmium-tool (the osmium CLI, version 1.11 or later is comfortable) for file-level application, and pyosmium (3.6+) when you want to drive application from Python. Both wrap the same C++ libosmium merge logic, so their semantics are identical; the choice is between a shell one-liner and in-process control.

The OsmChange Format Reference Jump to heading#

An .osc file is XML with a root <osmChange version="0.6"> and a generator attribute. Inside it, operation blocks group elements by the action to perform. The exact fields that govern application are:

Three semantic rules follow from this layout. First, a <modify> block contains the element’s complete new state, never a partial patch — every tag and reference is present, and application replaces the stored object wholesale. Second, a <delete> carries only enough of the element to identify it plus its new visible="false" version; the merge engine marks the object absent rather than physically erasing history. Third, the three action blocks may appear in any number and in any order within a file, and a single file may create an element and reference it from another element created in the same file — so a correct applier cannot assume a create always precedes the reference to it in document order.

Ordering and Version Semantics Jump to heading#

The rule that makes application correct is blunt: process diffs in ascending sequence order, and within the merge let versions decide. When apply-changes reads a base object at version v and a change at version v+1, it accepts the change. If the change is at v+2 — because you skipped the diff carrying v+1 — libosmium still applies it (the tool does not have the intervening state to object), which means the guardrail is yours, upstream of the tool: never hand apply-changes a diff set with a gap. Conversely, re-applying a diff you already merged is safe, because replacing version v with an identical version v is a no-op; this is what makes the operation idempotent and lets a crashed pipeline retry a cycle without harm.

Whole-object replacement also means the merge is last-writer-wins by document position when two changes to the same object appear across the ordered input. Because you feed diffs in sequence order and each diff carries strictly increasing versions for a given object, the last occurrence is by construction the newest — which is exactly what you want. The danger is only ever a gap or a reorder in the input you supply, not a conflict the engine resolves incorrectly.

Step-by-Step: Applying Diffs with osmium Jump to heading#

The apply-changes subcommand is the direct path. It reads a base and any number of change files and writes a merged output.

1. Confirm the base sequence. Know the sequence number your base is at; the first diff you apply must be that number plus one.
2. Order the change files. Sort the .osc.gz paths by their numeric sequence, ascending, with no gaps.
3. Run apply-changes. Pass the base first, then the ordered diffs, then the output path.
4. Verify and record. Confirm the output opens and advance your recorded sequence to the last diff applied.

# Steady-state: apply one minutely diff, producing a new sequence.
osmium apply-changes region.osm.pbf 006543211.osc.gz \
  --output region-updated.osm.pbf --overwrite

# Catch-up: apply a contiguous run of diffs in one pass. Shell brace order is
# not guaranteed by osmium — it re-sorts objects internally — but supplying them
# in ascending sequence keeps the version chain intact and auditable.
osmium apply-changes region.osm.pbf \
  006543211.osc.gz 006543212.osc.gz 006543213.osc.gz \
  --output region-updated.osm.pbf --overwrite

When your base is itself a full-history file, add --with-history so apply-changes retains prior versions instead of collapsing to the latest — the two modes write structurally different output and are not interchangeable:

# Update a full-history file, retaining every version rather than the latest only.
osmium apply-changes --with-history history.osh.pbf 006543211.osc.gz \
  --output history-updated.osh.pbf --overwrite

Step-by-Step: Applying a Diff with pyosmium Jump to heading#

When application must live inside a larger process — fetching, applying, and recording in one transaction — pyosmium exposes the merge through a writer and a diff-reading handler. The pattern below reads a base and a diff and writes an updated PBF, logging what it merged.

from __future__ import annotations

import logging

import osmium

logger = logging.getLogger(__name__)


class DiffApplier(osmium.SimpleHandler):
    """Merge one .osc.gz change file into a base PBF, writing an updated PBF.

    libosmium resolves create / modify / delete by object version; this handler
    simply forwards the merged object stream to a writer and tallies operations.
    """

    def __init__(self, writer: osmium.SimpleWriter) -> None:
        super().__init__()
        self.writer = writer
        self.counts: dict[str, int] = {"node": 0, "way": 0, "relation": 0}

    def node(self, n: osmium.osm.Node) -> None:
        self.writer.add_node(n)
        self.counts["node"] += 1

    def way(self, w: osmium.osm.Way) -> None:
        self.writer.add_way(w)
        self.counts["way"] += 1

    def relation(self, r: osmium.osm.Relation) -> None:
        self.writer.add_relation(r)
        self.counts["relation"] += 1


def apply_one_diff(base: str, diff: str, out: str) -> dict[str, int]:
    """Apply a single change file to a base extract and write the result."""
    writer = osmium.SimpleWriter(out)
    applier = DiffApplier(writer)
    try:
        # apply_file with a base then a diff merges by version; the change file
        # is read after the base so its versions win where ids collide.
        applier.apply_file(base)
        applier.apply_file(diff)
    finally:
        writer.close()
    logger.info("applied %s onto %s -> %s: %s", diff, base, out, applier.counts)
    return applier.counts

For the higher-level replication driver that fetches and applies a whole span of diffs rather than one file, see Catching Up a Stale OSM Extract with pyosmium, which uses pyosmium’s ReplicationServer.apply_diffs instead of the manual writer loop shown here.

Validation and Error Matrix Jump to heading#

Applying diffs surfaces a recognizable set of failures; each has a specific detection and fix rather than a generic retry.

Route genuinely defective change files — truncated downloads, gzip CRC failures — to the quarantine discipline in Error Handling in Large OSM Extracts rather than aborting a multi-diff catch-up on the first bad file.

Performance and Scale Considerations Jump to heading#

The cost of apply-changes is dominated by rewriting the base, not by the diff. Because a PBF is a compressed, block-structured container, merging a 200 KB minutely diff into an 8 GB regional file still means decompressing, merging, and recompressing the whole base — so a naive minutely loop that rewrites a large PBF every 60 seconds spends nearly all its time on I/O for a handful of changed objects. Two mitigations matter. For file-based workflows, batch: accumulate several minutely diffs and apply them in one apply-changes pass so the base is rewritten once per batch rather than once per diff. For high-frequency tracking, apply into a database rather than a file, where only the changed rows are touched — the subject of Applying Minutely Diffs to a PostGIS Database.

libosmium reads and writes streams, so peak memory is bounded by the location cache for reference resolution, not by file size; on continental bases, point osmium at a disk-backed index (--index-type / a location store) so way reconstruction during history-aware application does not exhaust RAM.

Failure Modes and Gotchas Jump to heading#

• Gaps are invisible to the tool. apply-changes cannot know a sequence is missing; it merges whatever versions you give it. Validate contiguity before you invoke it.
• --with-history is not a flag you can add later. A history base and a current base produce different output structures; choosing the wrong mode silently discards or duplicates versions.
• Document order within one .osc is not dependency order. A file may reference an object created later in the same file; never resolve references eagerly on the first pass.
• Deletions have no geometry. Treat a missing lat/lon on a delete as expected, not as corruption.
• Output overwrite is opt-in. Omitting --overwrite fails the run; a scheduled loop must write to a temp path and rename, or pass the flag explicitly.
• Timestamps do not order diffs. Sequence number orders the stream; the element timestamp is for temporal reconstruction, not for deciding which diff comes first.

Integration Points Jump to heading#

The output of this stage is an updated base ready for the next cycle, and its sequence number is the hand-off. Record that number and feed it back as the starting point of the following fetch — the loop the OSM Replication & Diff Sync pillar describes. A minimal wiring that applies a diff and then stamps the new sequence for the next run:

import logging

logger = logging.getLogger(__name__)


def apply_and_advance(base: str, diff: str, seq: int, out: str) -> int:
    """Apply one diff, then return the sequence the pipeline should record next."""
    counts = apply_one_diff(base, diff, out)  # from the pyosmium section above
    next_seq = seq + 1
    logger.info("advanced to sequence %d (%s)", next_seq, counts)
    return next_seq  # persist this only after the output write has committed

Explore This Topic in Depth Jump to heading#

Two focused guides extend this reference to concrete recovery and database scenarios:

• Catching Up a Stale OSM Extract with pyosmium — discover the starting sequence for a weeks-behind extract and apply every intervening diff with pyosmium’s replication API.
• Applying Minutely Diffs to a PostGIS Database — keep an osm2pgsql or imposm database current by appending diffs incrementally instead of rewriting a file.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Replication Sequence Numbers and State — resolving the ordered set of diffs this guide applies.
• Full-History .osh.pbf Processing — the history-aware base that --with-history targets.
• How to Decode OSM PBF Headers in Python — reading the base sequence number before applying.
• PBF File Structure Deep Dive — the container format the merged output is written in.
• Error Handling in Large OSM Extracts — quarantining truncated or corrupt change files.

Up one level: OSM Replication & Diff Sync.