OSM Extract Providers & Automated Downloads

The cheapest, fastest and most reproducible way to get OpenStreetMap data is almost always a file somebody else has already cut for you. It is also the source most often consumed carelessly, because downloading a file feels like a solved problem in a way that querying an API does not — and the failures that follow are correspondingly quiet.

How a published extract reaches a pipeline and where integrity is established The planet file is cut by a provider into regional extracts on a regular cadence, published alongside a checksum and in some cases a signature. A download step fetches both the file and its checksum. A verification step compares the computed digest against the published one and refuses to continue on a mismatch. A freshness gate compares the file's publication timestamp against the expected cadence and fails if the file is older than the pipeline tolerates. Only then does the extract reach the parser. Two gates stand between a download and a parser provider cut planet to region download file plus checksum verify digest refuse on mismatch check freshness refuse if stale parser only verified input The second gate is the one nobody adds, and a mirror that silently stopped updating is the failure it exists to catch.
A corrupt download fails loudly at the parser; a stale one produces perfectly valid output derived from last month's map.

The Problem This Topic Solves

Your pipeline needs OSM data for a region, on a schedule, reproducibly. Extract providers publish exactly that: pre-cut regional files, refreshed daily, with checksums and — depending on the provider — replication directories that let you keep a local copy current instead of re-downloading.

The failure scenario that motivates this whole topic is not a corrupt file. It is a stale one. A mirror stops updating, or a scheduled download starts failing silently and the pipeline keeps using the copy it already has. Nothing errors. The parser is happy, the validation rules pass, the output looks exactly as it always does — and every number it produces describes a map from three weeks ago. A checksum does not catch this, because the file is intact. Only an explicit assertion on the file’s publication date does.

Prerequisites

Know the format landscape from OSM XML vs PBF Comparison, because providers publish both and the choice matters more than it looks. Understand cut strategies from OSM Extract Clipping & Boundaries, since a provider’s choice of strategy determines what you get at the region’s edge. And read the parent Querying OSM: Overpass, Nominatim & APIs overview for why a file usually beats a query.

What Providers Actually Differ On

Extract providers are not interchangeable, and the differences that matter are rarely the ones listed on a comparison page.

Coverage granularity. Some providers publish a strict administrative hierarchy — continent, country, state — while others publish city-sized cuts around arbitrary urban areas. If your area of interest is a metropolitan region that straddles two states, those two models give you very different amounts of surplus data to filter.

Cut strategy at the boundary. A region’s edge has to do something about ways and relations that cross it. Different providers make different choices about whether to include the referenced nodes outside the boundary, whether to include partially-covered relations, and whether to clip geometry. This determines whether a road that leaves your region ends cleanly at the border or dangles with missing nodes.

Cadence and replication. Daily is typical for regional extracts. More important is whether the provider also publishes a replication directory for that region, because that is what lets you apply diffs rather than re-downloading — the machinery in OSM Replication & Diff Sync.

Integrity artefacts. A published checksum is the minimum. Some providers also publish signatures, which additionally prove the file came from them rather than from whatever answered the DNS query.

How the main kinds of OSM extract source differ on the properties that matter A grid comparing three source kinds across four properties. A country and state provider offers an administrative hierarchy, daily cadence, per-region replication directories and published checksums. A city-extract provider offers arbitrary urban cuts, varied cadence, usually no replication directory, and checksums. The full planet file offers complete coverage, a weekly cadence, the canonical replication stream, and both checksums and signatures. Three source kinds, and replication is the dividing line Country/state City extracts Planet file Coverage model administrative arbitrary urban everything Cadence daily varies weekly Replication per region usually none canonical Integrity checksums checksums checksum + sig Without a replication directory you are re-downloading the whole region forever, which is the hidden cost of a convenient city cut.
The replication row is the one that decides your operational model for the next two years.

Integrity: Checksums, Signatures and Atomic Writes

Three mechanisms, doing three different jobs, and they are frequently confused.

A checksum proves the bytes you have are the bytes the provider published. It catches truncated transfers, corrupted storage and the occasional bad mirror. It is the minimum bar and it costs one extra small download.

A signature proves the file came from the provider, not from someone who intercepted the connection or compromised a mirror. Where a provider publishes one, verifying it is strictly better than verifying a checksum alone — a checksum served from the same compromised host proves nothing.

An atomic write is the local half of the problem. Downloading directly to the path your pipeline reads means a process killed mid-transfer leaves a truncated file exactly where the parser expects a complete one. Download to a temporary path, verify, then rename. The rename is atomic on any sane filesystem, so the target path only ever holds a verified file. Automating Geofabrik Extract Downloads with Checksums implements all three.

Freshness as a Gate, Not a Log Line

The stale-extract failure deserves its own discipline because it is invisible by construction. The defence is a single assertion at the top of the pipeline: the input file’s publication timestamp must be within an explicitly declared tolerance, and the pipeline must fail if it is not.

Two details make it work. First, use the file’s timestamp, not the filesystem’s modification time, which changes when you copy the file and tells you nothing about the data. PBF files carry a header timestamp; providers also publish per-file dates. Second, declare the tolerance in the pipeline’s configuration next to the schedule, so a daily job asserting a three-day tolerance is visibly inconsistent and gets noticed.

The same assertion also catches a subtler failure: a download that silently fell back to a cached copy because the provider returned an error the client swallowed.

Validation and Error Handling

Condition Root cause Detection Remediation
Parser fails at a blob boundary Truncated transfer Checksum mismatch, or a parse error mid-file Verify the digest before the file is moved into place
Output describes an old map Mirror stopped updating File timestamp older than the cadence Assert freshness; fail the run, do not warn
Roads end abruptly at the border Provider’s cut strategy Dangling way references at the region edge Choose a provider strategy, or take a larger region
Download succeeds but is HTML Provider returned an error page with HTTP 200 File is kilobytes, not gigabytes Assert a minimum size and the PBF magic bytes
Different results on two machines Each downloaded a different daily file The two files’ timestamps differ Pin one file and share it, rather than downloading twice
Provider outage stops every job No local cache of the last good file All jobs fail simultaneously Keep the previous verified file and fall back explicitly
Checksum verified, file still wrong Checksum served from the same bad mirror Signature verification fails Verify the signature where one is published

Performance, Scale and the Shared Cache

A fleet of jobs each downloading the same country extract is the most common waste in an OSM pipeline. Ten jobs downloading the same six-gigabyte file is sixty gigabytes of transfer, a heavy load on a volunteer-funded mirror, and — worse — ten copies that may not be the same file if the provider published a new one partway through.

The fix is a single fetch into shared storage, keyed on the provider’s published version, with every job reading from there. That change gives reproducibility for free: jobs that read the same key are provably reading the same bytes. Mirroring OSM Downloads Behind a Local Cache covers the cache and the fallback behaviour when the provider is unreachable.

Beyond that, the biggest lever is not downloading at all. A region kept current with replication diffs transfers a few megabytes a day instead of gigabytes, which is the whole argument of Building a Minutely Update Pipeline.

Daily bytes transferred under four ways of keeping a country extract current Four approaches compared on daily transfer volume for a hypothetical fleet of ten jobs using one country extract. Every job downloading its own copy daily transfers the most by a wide margin. One shared download per day transfers a tenth of that. Downloading only when the published checksum changes saves a little more. Applying replication diffs instead of re-downloading transfers a tiny fraction of the file size each day. Four ways to stay current, three orders of magnitude apart Ten jobs, ten downloads baseline One shared download a tenth Download on checksum change slightly less Apply replication diffs a hundredth The first bar is what most pipelines actually do, and the gap to the second one is a configuration change rather than an engineering project.
Only the last approach changes the operational model; the first improvement is simply not fetching the same file ten times.

Failure Modes and Gotchas

  • An error page is a valid HTTP 200. Assert a plausible minimum size and check the file’s magic bytes before trusting it.
  • Filesystem mtime is not data freshness. Copying a file updates its mtime and tells you nothing about the map it describes.
  • “Latest” is not a version. A URL ending in -latest.osm.pbf names different bytes on different days. Record the checksum you actually used.
  • Boundary effects are provider choices. A way crossing the region edge behaves differently depending on the cut strategy, and that difference propagates into routing graphs.
  • Two machines, two files. Concurrent downloads around a publication boundary can fetch different versions. Fetch once, share the result.
  • Decompression is not verification. A file that decompresses is not necessarily complete; the checksum is the only proof.
  • A provider is a dependency. Its outage is your outage unless you keep the last good file and fall back deliberately.

Integration Points

Downstream, a verified extract is the input to every parsing workflow on this site — the parser choice in Choosing an OSM Parser: pyosmium, pyrosm or osmium-tool assumes a complete, trustworthy file. If your region of interest is smaller than the smallest published extract, the clipping workflow in Clipping an OSM Extract with a .poly Boundary takes over.

Sideways, the provenance of the download — which provider, which file, which checksum, which date — is exactly what the licensing obligations in Recording OSM Data Provenance in a Pipeline need recorded.

Guides in This Topic

Frequently Asked Questions

Is verifying a checksum enough?

It is enough to prove the bytes are intact, which catches truncation and corruption. It does not prove the file came from the provider, because a compromised mirror can serve a matching checksum alongside a modified file. Where a provider publishes a signature, verify that instead — it establishes origin as well as integrity. And neither mechanism says anything about whether the file is current, which needs a separate freshness assertion.

How do I detect a stale extract?

Assert on the data’s own timestamp rather than on the file’s modification time. A PBF file carries a header timestamp describing the state of the map it was cut from, and providers publish per-file dates alongside the download. Compare that against an explicitly declared tolerance and fail the run when it is exceeded. Logging a warning is not enough, because the pipeline will succeed and nobody reads warnings on a successful run.

Should every job download its own extract?

No. A fleet of jobs downloading the same regional file wastes bandwidth, loads a volunteer-funded mirror unnecessarily, and — the part that actually bites — can leave different jobs working from different daily files if a publication happens mid-run. Fetch once into shared storage keyed on the published version, and have every job read from there, which makes reproducibility a property of the design rather than a hope.

What happens at the edge of a regional extract?

That depends on the cut strategy the provider used, and providers differ. A way crossing the boundary may be included complete with its nodes outside the region, included but truncated with dangling references, or excluded entirely. The consequences show up downstream as roads that stop at the border or geometry that cannot be assembled. If edge behaviour matters — it always does for routing — either choose a provider whose strategy you have verified or take a larger region and clip it yourself.

When should I use the planet file instead of a regional extract?

When your area of interest spans several countries, when you need the canonical replication stream rather than a per-region one, or when you are cutting your own regions and want to control the boundary strategy. The planet file is a large commitment in storage and processing time, so it is rarely the right answer for a single country — but for a multi-region pipeline that already needs its own cutting step it removes a whole class of provider dependency.

Up one level: Querying OSM: Overpass, Nominatim & APIs.