Automating Geofabrik Extract Downloads with Checksums

Download a regional .osm.pbf on a schedule so that the file your parser opens is provably complete, provably current, and never a half-finished transfer.

Prerequisites

Conceptual minimum

Three independent properties have to hold before a downloaded extract is safe to parse, and each needs its own check.

Completeness is proved by comparing a computed digest against the published one. This catches truncated transfers and corrupted mirrors, and nothing else does — a PBF file that lost its last blob still decompresses and still parses right up to the point where the data stops.

Currency is proved by reading the data’s timestamp, not the file’s. The PBF header carries an osmosis_replication_timestamp describing the state of the map the extract was cut from. Filesystem modification time changes when you copy a file and tells you nothing.

Atomicity is a property of how you write, not of what you downloaded. If the pipeline’s input path is also the download target, a process killed mid-transfer leaves a truncated file exactly where a parser expects a good one, and the next run parses it happily until it stops.

Four stages between a provider URL and a file a parser may open Four stages. The fetch stage streams the extract and its published checksum to a temporary path, computing the digest as bytes arrive so no second read is needed. The verify stage compares the computed digest against the published one and aborts on a mismatch, deleting the temporary file. The gate stage reads the PBF header replication timestamp and aborts if it is older than the declared tolerance. The swap stage renames the verified file over the pipeline's input path, which is atomic and leaves no window where a partial file is visible. Fetch, verify, gate, swap — in that order fetch stream to a temp path hash as bytes arrive verify digest must match delete on mismatch gate header timestamp fail if too old swap rename into place atomic, no window Reversing the last two stages would publish a stale but intact file, which is the failure the gate exists to prevent.
Each stage can only reject; none of them repairs anything, which is what makes the sequence easy to reason about.

Runnable solution

python 
from __future__ import annotations

import hashlib
import logging
import re
import sys
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from pathlib import Path

import osmium
import requests

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.extract.download")

UA = "osm-pipeline-example/1.0 (contact@example.org)"
CHUNK = 1 << 20


class IntegrityError(RuntimeError):
    """The downloaded bytes do not match the published digest."""


class StaleExtractError(RuntimeError):
    """The extract is intact but describes a map older than we tolerate."""


@dataclass(frozen=True)
class Source:
    url: str
    md5_url: str
    max_age: timedelta


def published_digest(md5_url: str) -> str:
    """Providers publish '<hex>  <filename>' beside each extract."""
    response = requests.get(md5_url, headers={"User-Agent": UA}, timeout=60)
    response.raise_for_status()
    match = re.match(r"([0-9a-f]{32})\s", response.text.strip())
    if not match:
        raise IntegrityError(f"unparseable digest file at {md5_url}")
    return match.group(1)


def download_to(tmp: Path, url: str) -> str:
    """Stream to disk, hashing as we go so the file is read exactly once."""
    digest = hashlib.md5()
    total = 0
    with requests.get(url, headers={"User-Agent": UA},
                      stream=True, timeout=600) as response:
        response.raise_for_status()
        with tmp.open("wb") as handle:
            for chunk in response.iter_content(chunk_size=CHUNK):
                handle.write(chunk)
                digest.update(chunk)
                total += len(chunk)
    logger.info("downloaded %.1f MiB", total / (1 << 20))
    if total < (1 << 20):
        # An error page served with HTTP 200 is small; a real extract is not.
        raise IntegrityError(f"suspiciously small download: {total} bytes")
    return digest.hexdigest()


def header_timestamp(path: Path) -> datetime:
    """Read the replication timestamp the provider baked into the PBF header."""
    reader = osmium.io.Reader(str(path))
    try:
        stamp = reader.header().get("osmosis_replication_timestamp")
    finally:
        reader.close()
    if not stamp:
        raise StaleExtractError(f"{path} has no replication timestamp in its header")
    return datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)


def refresh(source: Source, target: Path) -> Path:
    tmp = target.with_suffix(target.suffix + ".part")
    expected = published_digest(source.md5_url)

    actual = download_to(tmp, source.url)
    if actual != expected:
        tmp.unlink(missing_ok=True)
        raise IntegrityError(f"digest mismatch: got {actual}, expected {expected}")
    logger.info("digest verified: %s", actual)

    age = datetime.now(timezone.utc) - header_timestamp(tmp)
    if age > source.max_age:
        tmp.unlink(missing_ok=True)
        raise StaleExtractError(
            f"extract describes a map {age} old, tolerance is {source.max_age}")
    logger.info("freshness ok: extract is %s old", age)

    tmp.replace(target)          # atomic within a filesystem
    logger.info("installed %s", target)
    return target


if __name__ == "__main__":
    base = "https://download.geofabrik.de/europe/poland-latest.osm.pbf"
    try:
        refresh(Source(base, base + ".md5", timedelta(days=2)),
                Path("/data/poland-latest.osm.pbf"))
    except (IntegrityError, StaleExtractError) as exc:
        logger.error("refresh refused: %s", exc)
        sys.exit(1)

Step-by-step walkthrough

  1. Fetch the digest first. Knowing what the file should hash to before downloading it means the verification is a comparison rather than a decision about whether to bother.
  2. Hash while streaming. The digest is updated chunk by chunk as bytes land, so a multi-gigabyte file is read from the network once and never re-read from disk.
  3. Reject implausibly small downloads. A provider error page returned with HTTP 200 is a few kilobytes. A minimum-size assertion catches that class before the digest comparison even runs, with a clearer message.
  4. Delete on failure. Both failure paths remove the temporary file. Leaving a failed .part behind invites a later process to find it and guess.
  5. Read the header, not the filesystem. osmosis_replication_timestamp is the state of the map the extract was cut from. It travels with the file through copies, mirrors and archives, which is exactly what makes it trustworthy.
  6. Fail on staleness, do not warn. A stale extract produces perfectly valid output describing an old map, so the only effective response is to stop the pipeline.
  7. Swap by rename. Path.replace is atomic within a filesystem, so the target path holds either the previous verified file or the new verified file and never anything in between. Keep the temporary file on the same filesystem or the rename degrades into a copy.
Three download failures that produce no error without explicit checks Three panels. A truncated transfer leaves a file that still decompresses and parses until the data simply stops, and is caught only by comparing the computed digest against the published one. An error page served with HTTP 200 is a small text file where a multi-gigabyte extract was expected, and is caught by a minimum-size assertion before any parsing. A stale mirror serves a complete, correct file describing a map from weeks ago, and is caught only by reading the header replication timestamp. Three silent failures, three different checks Truncated transfer File decompresses fine Parses until data stops Counts look low, not wrong Caught by: digest comparison Nothing else detects it Error page as data HTTP 200, a few kilobytes Not a PBF at all Parser error is cryptic Caught by: minimum size Check before hashing Stale mirror Complete, valid, correct file Describes an old map Every gate downstream passes Caught by: header timestamp Never by a checksum Only the first of these three is what people mean by a corrupt download, and it is the least damaging of the three.
The third failure is the dangerous one because every other check in the pipeline reports success.
Where the temporary file lives and why it matters Four layers describing the file lifecycle. The remote file is the provider's published extract with its own digest. The temporary path is a sibling of the target on the same filesystem, holding bytes that have not yet been verified. The verified state is reached once the digest matches and the header timestamp is within tolerance. The installed path is the pipeline's input, reached by an atomic rename that never exposes a partial file. Four states, one atomic transition between the last two Remote Provider file plus published digest the only source of truth Temporary Sibling of the target, unverified same filesystem, always Verified Digest matched, freshness passed still not installed Installed The pipeline's input path reached by rename only Putting the temporary file elsewhere turns the final rename into a full copy, reintroducing the partial-file window it was there to remove.
The whole design is one rule: the input path is only ever written by a rename from a file that already passed both gates.

Verification

  • A deliberately corrupted file is rejected. Truncate a downloaded extract and re-run; the digest comparison must fail and the temporary file must be gone.
  • A stale file is rejected. Set the tolerance to an hour and re-run against a daily extract; the freshness gate must fail.
  • The target is never partial. Kill the process mid-download repeatedly; the target path must always hold a complete, previously verified file.
  • The temporary file shares a filesystem with the target. Check that the rename is instant rather than taking as long as a copy.
  • A rerun with an unchanged remote file is cheap. Compare the published digest against the installed file’s digest first and skip the download when they match.

Common errors and fixes

Symptom Root cause One-line fix
Parser stops mid-file Truncated transfer accepted Compare the computed digest before installing
Cryptic parse error on a tiny file Error page saved as the extract Assert a plausible minimum size first
Output describes an old map No freshness gate Read the header timestamp and fail on excess age
Rename takes minutes Temporary file on a different filesystem Put the .part file beside the target
Stale .part files accumulate Failure paths do not clean up Unlink the temporary file on every failure
Digest file unparseable Provider format differs from the assumption Match a hex digest with a regular expression, not by splitting
Freshness gate always passes Filesystem mtime used instead of header Read osmosis_replication_timestamp from the PBF header

Specification reference

A PBF file’s OSMHeader block may carry optional metadata including osmosis_replication_timestamp, osmosis_replication_sequence_number and osmosis_replication_base_url, which describe the replication state the file was produced from. Extract providers populate these for their published regional files. See the PBF file format documentation for the header fields and their semantics.

Frequently Asked Questions

Why not just check the file size?

Because size only catches the gross failures. A transfer that dropped the last few blobs is within a fraction of a percent of the expected size and passes any tolerance loose enough not to produce false alarms as the region grows. The published digest is exact and costs one small extra request. Keep a minimum-size assertion as well, but as a fast check for the error-page case rather than as the integrity test.

Should a stale extract fail the run or just warn?

Fail it. The entire problem with a stale extract is that everything downstream succeeds: the parser is happy, the validation rules pass, the output looks normal. A warning on a successful run is not read by anybody. Failing is also easy to override deliberately when you genuinely want to reprocess an archived file, which is the case a warning is usually trying to accommodate.

What tolerance should I set?

A little over the provider’s publication cadence, tight enough that a mirror which stopped updating is caught within a day or two. For a daily extract, two days is a sensible default: it survives a single missed publication without alarming, and it catches a mirror that has genuinely stalled. Declare it next to the job’s schedule so an inconsistency between the two is visible when somebody reads the configuration.

Can I skip the download when nothing changed?

Yes, and you should. Fetch the published digest first and compare it against the digest of the file you already have; if they match, the remote file is byte-identical and there is nothing to do. That turns a daily job into one small request on days when the provider has not published, which matters when many jobs share a volunteer-funded mirror.

Up one level: OSM Extract Providers & Automated Downloads.