Mirroring OSM Downloads Behind a Local Cache

Give a fleet of jobs one shared, verified copy of each OSM extract, so ten workers transfer one file rather than ten — and so two workers can never end up silently reading different days of the map.

Prerequisites

Conceptual minimum

The cache has one job: make “the extract for region R at version V” a name that resolves to the same bytes for every job, forever.

That points straight at content addressing. Keying the cache on the provider’s published digest rather than on a filename or a date gives three properties at once. Two jobs asking for the same version provably get the same bytes. A provider republishing under the same filename produces a different key rather than silently changing the file underneath a running job. And an archived result can name the exact input it used, which is what makes a run reproducible months later.

The second requirement is single flight. Ten jobs starting simultaneously must produce one download, not ten. A cross-process lock around the fetch, with a re-check of the cache after acquiring it, is all that takes.

The third is a deliberate outage policy. When the provider is unreachable, the cache still holds the last good version. Serving it is usually right and occasionally catastrophic, so the choice must be explicit and must be visible in the logs when it is exercised.

How ten jobs asking for one extract become one download Ten concurrent jobs all request the same region. A cache lookup keyed on the published digest resolves nine of them immediately once the file is present. The tenth, arriving first, takes a cross-process lock, re-checks the cache, performs the verified download and installs the file under its digest key. All ten then read the same local path, so no two jobs can be working from different published versions of the map. Ten requests, one lock, one download ten jobs same region, same moment digest lookup resolve the version single-flight lock one fetch, others wait verified download digest and freshness one shared path identical bytes for all Without the lock, ten workers each download the file and up to ten of them can land on different published versions across a boundary.
The lock is not a performance optimisation first — it is what stops a fleet from disagreeing about which day it is.

Runnable solution

python 
from __future__ import annotations

import logging
import shutil
from dataclasses import dataclass
from datetime import timedelta
from pathlib import Path

import requests
from filelock import FileLock

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.extract.cache")

UA = "osm-pipeline-example/1.0 (contact@example.org)"


class ProviderUnavailable(RuntimeError):
    """The provider could not be reached and no fallback was permitted."""


@dataclass(frozen=True)
class CacheConfig:
    root: Path
    max_age: timedelta
    allow_stale_on_outage: bool = False    # opt in, never the default
    keep_versions: int = 3


def _region_dir(config: CacheConfig, region: str) -> Path:
    path = config.root / region
    path.mkdir(parents=True, exist_ok=True)
    return path


def _published_digest(md5_url: str) -> str:
    response = requests.get(md5_url, headers={"User-Agent": UA}, timeout=30)
    response.raise_for_status()
    return response.text.strip().split()[0]


def _newest_cached(region_dir: Path) -> Path | None:
    files = sorted(region_dir.glob("*.osm.pbf"), key=lambda p: p.stat().st_mtime)
    return files[-1] if files else None


def _prune(region_dir: Path, keep: int) -> None:
    files = sorted(region_dir.glob("*.osm.pbf"), key=lambda p: p.stat().st_mtime)
    for old in files[:-keep] if len(files) > keep else []:
        old.unlink()
        logger.info("pruned %s", old.name)


def fetch(config: CacheConfig, region: str, url: str, md5_url: str,
          download) -> Path:
    """Return a path to a verified extract, downloading at most once per version.

    `download(url, md5, dest)` is the verifying client: it must check the digest
    and the freshness gate and write `dest` atomically.
    """
    region_dir = _region_dir(config, region)

    try:
        digest = _published_digest(md5_url)
    except requests.RequestException as exc:
        cached = _newest_cached(region_dir)
        if cached and config.allow_stale_on_outage:
            # Loud on purpose: this run is NOT using current data.
            logger.warning("provider unreachable (%s) — serving cached %s",
                           exc, cached.name)
            return cached
        raise ProviderUnavailable(f"cannot reach provider and no fallback: {exc}")

    target = region_dir / f"{digest}.osm.pbf"
    if target.exists():
        logger.info("cache hit for %s at %s", region, digest[:12])
        return target

    # Single flight: the first arrival downloads, everyone else waits and re-checks.
    with FileLock(str(region_dir / ".fetch.lock"), timeout=3600):
        if target.exists():
            logger.info("another worker fetched %s while we waited", digest[:12])
            return target
        logger.info("fetching %s for %s", digest[:12], region)
        download(url, md5_url, target)
        _prune(region_dir, config.keep_versions)
    return target


def resolve_for_job(config: CacheConfig, region: str, url: str, md5_url: str,
                    download, workdir: Path) -> Path:
    """Give the job a stable local name pointing at the shared cached file."""
    source = fetch(config, region, url, md5_url, download)
    link = workdir / f"{region}.osm.pbf"
    link.unlink(missing_ok=True)
    try:
        link.symlink_to(source)
    except OSError:
        shutil.copy2(source, link)   # filesystems without symlinks
    logger.info("job input %s -> %s", link, source.name)
    return link


if __name__ == "__main__":
    logger.info("wire `download` to the verifying client from the sibling guide")

Step-by-step walkthrough

  1. Ask for the digest first. The published checksum is a tiny request and it is the version identity. Everything else follows from it.
  2. Key the file on the digest. A file named for its own content cannot be ambiguous, cannot be silently replaced, and can be named in a provenance record that still means something in a year.
  3. Return immediately on a hit. The common path does one small HTTP request and a filesystem check. No lock is taken when the file is already present, so a fleet of hundreds of jobs costs hundreds of tiny requests rather than any contention.
  4. Re-check inside the lock. Between deciding to fetch and acquiring the lock, another worker may have finished. Checking again is two lines and removes the duplicate download entirely.
  5. Delegate the actual download. The verifying client is passed in. This cache adds sharing and single-flight; it deliberately does not re-implement digest and freshness checking.
  6. Make the outage path opt-in and loud. Serving a stale file during a provider outage is a policy decision. The default refuses; enabling it logs a warning naming the file being served, so the decision is visible in the run’s output.
  7. Give each job a stable local name. Jobs should not embed digests in their configuration. A symlink from a predictable name to the content-addressed file gives both stability and traceability, with a copy fallback where symlinks are unavailable.
  8. Prune inside the lock. Retention runs while the lock is held, so no other worker can be reading a file as it is removed.
Three cache key choices and what each one allows to go wrong Three panels comparing cache key strategies. Keying on the filename alone means a republished file silently replaces the old one under the same key, so two jobs in the same run can read different data. Keying on the date is better but still ambiguous when a provider republishes within a day, and it cannot express that two dates carry identical bytes. Keying on the published digest makes identity exact, makes a republication a new key, and makes a provenance record verifiable long afterwards. Three cache keys, only one of them is an identity Filename Latest name, reused daily Republish overwrites in place Two jobs, two different files No provenance value at all Never adequate Date One key per calendar day Republication within a day hides Identical bytes, two keys Provenance is approximate Better, still ambiguous Published digest Key is the content itself Republication is a new key Identical bytes, one key Provenance verifiable later The only real identity The date key looks sufficient until the first time a provider republishes a corrected file a few hours after the original.
Content addressing is not sophistication here; it is the minimum needed for two jobs to prove they read the same map.
What happens to a fleet across a provider publication boundary Four moments in time. Before the publication all jobs resolve the same digest and share one cached file. At the publication moment the provider swaps the file behind the same filename and publishes a new digest. Immediately after, a newly started job resolves the new digest, takes the lock and fetches, while jobs already running keep their existing symlink target and finish against the version they started with. Once the fetch completes both versions exist in the cache and retention decides when the older one goes. A publication mid-run must not change a running job's input before one digest all jobs agree publish provider swaps new digest appears after new jobs fetch running jobs unaffected settled both versions held retention decides Keeping the older version is what lets a job that started before the boundary finish against a consistent input instead of failing.
Content addressing plus a short retention window is the whole mechanism; nothing here needs coordination between jobs.

Verification

  • Ten concurrent jobs produce one download. Launch them together against a cold cache and count provider requests for the file itself; it must be one.
  • A warm cache takes no lock. Instrument the lock acquisition; on a warm run it should never be reached.
  • Two jobs resolve to the same file. Compare the symlink targets from two workers started either side of a publication boundary; they must match.
  • The outage path refuses by default. Block network access and run; the job must fail rather than quietly using an old file.
  • Pruning never removes an in-use file. Run a long job while another triggers a fetch and prune; the running job’s input must survive.

Common errors and fixes

Symptom Root cause One-line fix
Ten downloads for one file No cross-process lock Take a lock around the fetch and re-check inside it
Two jobs disagree about the data Cache keyed on filename Key on the published digest instead
Deadlock under load Lock held across the whole job Hold the lock only around fetch and prune
Job breaks when a version is pruned Retention ignores readers Prune inside the lock and keep several versions
Silent use of stale data Outage fallback enabled by default Make the fallback opt-in and log it as a warning
Symlink fails on the target filesystem Filesystem without symlink support Fall back to a copy, explicitly
Cache grows without bound No retention policy Prune to a fixed number of versions per region

Specification reference

Content-addressed storage names an object by a cryptographic digest of its contents, so a name resolves to exactly one byte sequence and any change to the contents produces a different name. Applied to published OSM extracts, the provider’s own published digest serves as that name without recomputation. See the OSM planet and extract documentation for the digest files published alongside each extract.

Frequently Asked Questions

Why key the cache on a digest rather than on the date?

Because a date is not an identity. Providers occasionally republish a corrected file within the same day, and a date key silently overwrites the earlier one — so two jobs in the same run can read different data while both believing they used “today’s extract”. A digest key makes a republication a new entry, makes identical bytes resolve to one entry, and lets a stored provenance record be verified long afterwards.

Should the cache serve a stale file when the provider is down?

Only when somebody has decided in advance that it should, and only loudly. For a dashboard that tolerates yesterday’s data, continuing is clearly better than failing. For a pipeline computing figures somebody will act on, silently substituting old data is worse than an outage. Make it a configuration flag that defaults to refusing, and log a warning naming the file whenever the fallback is used.

How many versions should I keep?

Enough to cover your longest-running job plus a margin for reproducing a recent result — three is a reasonable default for a daily extract. The cost is storage; the benefit is that a job started before a refresh keeps a valid input and that last week’s output can be re-derived. Prune inside the same lock the fetch uses so a file is never removed while a fetch or another prune is in flight.

Does the cache replace the checksum and freshness checks?

No, it composes with them. The cache’s job is sharing and single-flight; the verifying client’s job is proving the bytes are complete and the data is current. Keeping them separate means a cache hit is trustworthy because nothing was ever installed in the cache without passing both gates, and it keeps each piece small enough to reason about.

Up one level: OSM Extract Providers & Automated Downloads.