Running OSM Diff Sync in Docker with a Persistent State Volume

Containerise a minutely update loop so a container restart costs a minute, not a full reimport — and so two containers can never apply diffs to the same file at once.

Prerequisites

Conceptual minimum

Containers are ephemeral by design and a diff-sync loop is defined by its state. Reconciling those is a matter of being explicit about which is which.

Which parts of a diff-sync deployment belong in the image and which in a volume A grid of five items. The sync script belongs in the image because code is immutable and rebuilt on deploy. The working PBF belongs in a volume because re-downloading it takes hours. The sequence checkpoint belongs in a volume above all, because losing it means a rebuild. The osmium node cache may optionally live in a volume; it is rebuildable but slow. Logs belong in neither and should go to stdout for the runtime to collect. What must survive a container restart, and where it lives in the image in a volume why the sync script yes no code is immutable, rebuilt on deploy the working .osm.pbf no yes re-downloading it is hours the sequence checkpoint no yes — this one above all losing it means a rebuild the osmium node cache no optional rebuildable, but slow logs no no — stdout let the runtime collect them One row is the whole reason this needs care: a checkpoint written inside the container filesystem is gone the moment the container is replaced.
The checkpoint is the one piece of state that cannot be reconstructed from anything else, which makes the volume mount a correctness requirement rather than a convenience.

The checkpoint is the piece that matters. The working extract is expensive to lose and can be re-downloaded; the checkpoint cannot be reconstructed from anything, and losing it means either a rebuild from a fresh extract or, worse, a guess that silently skips or replays diffs — the failure the sequence discipline in Replication Sequence Numbers & State Tracking exists to prevent.

The startup and locking sequence for a containerised diff-sync A four-stage chain. The container starts and reads the checkpoint from the mounted volume. It takes an flock on the volume rather than on a container-local path, because the lock must live where the state lives. It applies the diff and writes the checkpoint in that order, with an fsync before releasing. It exits zero, and either a restart policy re-runs it or an internal sleep loop continues. One container, one lock, one owner of the state container starts reads the checkpoint from the mounted volume flock on the volume not on /tmp the lock must share the state apply + checkpoint in that order fsync before releasing exit 0 restart policy re-runs it or a sleep loop inside Putting the lock inside the container filesystem makes it invisible to a second container, which is exactly the case it exists to prevent.
A lock on a container-local path is invisible to every other container, which makes it a lock that cannot do its job.

The lock deserves the same care as the checkpoint, and for the same reason. flock on a path inside the container’s own filesystem is invisible to any other container, so two replicas each take “the lock” successfully and both apply diffs to the same file.

Three scheduling models for a containerised diff-sync loop Three panels. A long-running loop sleeps between iterations in-process, keeping one container always up with the restart policy handling crashes; it is simplest to reason about but accumulates memory leaks. Restart-on-exit runs one iteration per process with restart always re-running it, giving a fresh process each minute and no leak accumulation, at the risk of restart storms if it fails fast. An external scheduler such as a Kubernetes CronJob or systemd timer owns the cadence and gives overlap prevention free through a forbid concurrency policy, at the cost of another moving part. Three ways to schedule the loop in a container Long-running loop Sleep between iterations, in-process One container, always up Restart policy handles crashes Simplest to reason about Memory leaks accumulate Restart-on-exit Script does one iteration, exits restart: always` re-runs it Fresh process every minute No leak accumulation Restart storms if it fails fast External scheduler Kubernetes CronJob, systemd timer The runtime owns the cadence concurrencyPolicy: Forbid Overlap prevention comes free One more moving part The middle option needs a backoff or a failing script becomes a restart loop; the third gets overlap prevention from the scheduler instead of from flock.
All three work. What must not vary is that exactly one of them applies a diff to the state volume at a time.

Runnable solution

dockerfile 
# Dockerfile
FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
        osmium-tool python3 python3-pyosmium ca-certificates curl \
    && rm -rf /var/lib/apt/lists/*

# Run as a fixed non-root uid so the volume's ownership is predictable across hosts.
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin osm
WORKDIR /app
COPY --chown=osm:osm sync.py /app/sync.py

USER osm
ENV STATE_DIR=/state \
    REPL_BASE=https://planet.osm.org/replication/minute/ \
    PYTHONUNBUFFERED=1

# stdout only: let the container runtime collect and rotate the logs.
ENTRYPOINT ["python3", "/app/sync.py"]
yaml 
# compose.yaml
services:
  osm-sync:
    build: .
    restart: unless-stopped
    volumes:
      - osm-state:/state          # the extract, the checkpoint and the lock
    environment:
      EXTRACT: /state/ireland.osm.pbf
      SLEEP_SECONDS: "60"
    healthcheck:
      # Unhealthy once the checkpoint stops advancing, not merely when the
      # process dies — a wedged loop keeps its process alive.
      test: ["CMD", "python3", "/app/sync.py", "--healthcheck"]
      interval: 120s
      timeout: 10s
      retries: 3
      start_period: 300s
    stop_grace_period: 120s       # let an in-flight apply finish rather than be killed
    deploy:
      resources:
        limits: { memory: 4G }

volumes:
  osm-state:
python 
#!/usr/bin/env python3
"""Containerised OSM diff-sync. All durable state lives under $STATE_DIR."""
from __future__ import annotations

import fcntl
import json
import logging
import os
import signal
import subprocess
import sys
import time
from dataclasses import dataclass
from pathlib import Path

logging.basicConfig(level=logging.INFO, stream=sys.stdout,
                    format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger("osm-sync")

STATE_DIR = Path(os.environ.get("STATE_DIR", "/state"))
EXTRACT = Path(os.environ.get("EXTRACT", str(STATE_DIR / "data.osm.pbf")))
CHECKPOINT = STATE_DIR / "checkpoint.json"
LOCK_PATH = STATE_DIR / "sync.lock"      # on the volume, not in the container
SLEEP_SECONDS = int(os.environ.get("SLEEP_SECONDS", "60"))
STALE_AFTER = int(os.environ.get("STALE_AFTER", "900"))

_shutdown = False


def _request_shutdown(signum, _frame) -> None:
    """SIGTERM sets a flag; the loop finishes the current iteration and exits."""
    global _shutdown
    logger.info("signal %d received — finishing the current iteration", signum)
    _shutdown = True


@dataclass(frozen=True)
class Checkpoint:
    sequence: int
    updated_at: float

    @classmethod
    def read(cls) -> "Checkpoint | None":
        if not CHECKPOINT.exists():
            return None
        data = json.loads(CHECKPOINT.read_text())
        return cls(sequence=int(data["sequence"]), updated_at=float(data["updated_at"]))

    def write(self) -> None:
        """Temp file plus atomic rename, both on the volume, then fsync the dir."""
        tmp = CHECKPOINT.with_suffix(".tmp")
        tmp.write_text(json.dumps({"sequence": self.sequence,
                                   "updated_at": self.updated_at}))
        with tmp.open("rb") as handle:
            os.fsync(handle.fileno())
        tmp.replace(CHECKPOINT)
        dir_fd = os.open(STATE_DIR, os.O_RDONLY)
        try:
            os.fsync(dir_fd)          # the rename itself must reach the disk
        finally:
            os.close(dir_fd)


def healthcheck() -> int:
    """Healthy only while the checkpoint is advancing."""
    checkpoint = Checkpoint.read()
    if checkpoint is None:
        logger.warning("no checkpoint yet")
        return 1
    age = time.time() - checkpoint.updated_at
    if age > STALE_AFTER:
        logger.error("checkpoint is %.0f s old (limit %d s)", age, STALE_AFTER)
        return 1
    logger.info("healthy: sequence %d, %.0f s old", checkpoint.sequence, age)
    return 0


def apply_next(sequence: int) -> bool:
    """Apply one diff. Returns False when already current."""
    result = subprocess.run(
        ["pyosmium-up-to-date", "--verbose", "--size", "100",
         "--server", os.environ["REPL_BASE"], str(EXTRACT)],
        capture_output=True, text=True)
    if result.returncode not in (0, 3):        # 3 = already up to date
        logger.error("apply failed (%d): %s", result.returncode, result.stderr.strip())
        raise RuntimeError("apply-changes failed")
    return result.returncode == 0


def iteration() -> None:
    checkpoint = Checkpoint.read()
    sequence = checkpoint.sequence if checkpoint else 0
    if apply_next(sequence):
        # Work first, checkpoint second — a crash between them replays, never skips.
        Checkpoint(sequence=sequence + 1, updated_at=time.time()).write()
        logger.info("applied; checkpoint now %d", sequence + 1)
    else:
        Checkpoint(sequence=sequence, updated_at=time.time()).write()
        logger.info("already current at %d", sequence)


def main() -> int:
    if "--healthcheck" in sys.argv:
        return healthcheck()

    STATE_DIR.mkdir(parents=True, exist_ok=True)
    signal.signal(signal.SIGTERM, _request_shutdown)
    signal.signal(signal.SIGINT, _request_shutdown)

    with LOCK_PATH.open("w") as lock:
        try:
            fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
        except BlockingIOError:
            logger.error("another container holds the state lock — exiting")
            return 1

        logger.info("lock acquired; syncing %s", EXTRACT)
        while not _shutdown:
            started = time.monotonic()
            try:
                iteration()
            except Exception:
                logger.exception("iteration failed; retrying after the interval")
            elapsed = time.monotonic() - started
            for _ in range(int(max(0.0, SLEEP_SECONDS - elapsed))):
                if _shutdown:
                    break
                time.sleep(1)          # 1 s granularity so SIGTERM is responsive
    logger.info("shut down cleanly")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Step-by-step walkthrough

The lock file lives at /state/sync.lock, on the volume. That is the single most important line in the file: a lock on the volume is shared by every container that mounts it, which is precisely the set of processes that could collide. flock also releases automatically when the process dies, so a hard kill does not leave a stale lock behind.

Checkpoint.write does temp file, fsync, rename, then fsync of the directory. The directory fsync is the step usually missed — without it the rename can be lost in a power failure even though the data survived, leaving the checkpoint at its previous value and replaying one diff. That is the safe direction, which is why the ordering also puts the apply first.

The USER with a fixed uid matters more in containers than elsewhere. A volume created by one image and mounted by another with a different uid gives permission errors that look like corruption; pinning 10001 makes the ownership predictable across rebuilds and hosts.

The healthcheck asserts that the checkpoint is advancing, not that the process is alive. A loop wedged on a socket read keeps its process, its port and its liveness, and only the checkpoint’s age reveals it — the same distinction drawn in Replication Monitoring & Lag Alerting.

stop_grace_period: 120s with a SIGTERM flag lets an in-flight apply finish. The default ten seconds is far too short for a diff apply on a country extract, and a SIGKILL partway through leaves a partially rewritten file.

The sleep is a one-second loop rather than one long time.sleep, so a shutdown signal is acted on within a second instead of up to a minute.

Verification

Prove the state survives the container, which is the whole point:

bash 
docker compose up -d && sleep 180
docker compose exec osm-sync cat /state/checkpoint.json
docker compose down && docker compose up -d          # container replaced
docker compose logs --tail 5 osm-sync                # expect it to resume, not restart

The second checkpoint read must show a sequence at or above the first. A sequence of zero means the volume is not mounted where the code expects.

Prove the lock actually excludes a second container:

bash 
docker compose up -d
docker compose run --rm osm-sync                     # a second one, same volume
# expect: "another container holds the state lock — exiting", exit 1

If the second container starts syncing, the lock is on a container-local path.

Prove graceful shutdown does not truncate an apply:

bash 
docker compose stop -t 120 osm-sync
docker compose logs --tail 3 osm-sync                # expect "shut down cleanly"
osmium fileinfo /var/lib/docker/volumes/.../ireland.osm.pbf   # must still parse

Common errors and fixes

Symptom Root cause Fix
Full reimport after every restart Checkpoint written inside the container Put it on the mounted volume
Two containers applying diffs together Lock on a container-local path flock a file on the volume
Permission denied on the volume uid differs between image builds Pin a fixed uid in the Dockerfile
Healthy container, stale data Healthcheck only tests liveness Assert the checkpoint’s age
Corrupt PBF after a deploy SIGKILL mid-apply Raise stop_grace_period; handle SIGTERM
Container OOM-killed No memory limit, or too low for the node cache Set a limit above the cache size
Restart storm Fail-fast script with restart: always Add a backoff, or run a long-lived loop

Frequently Asked Questions

Named volume or bind mount?

A named volume for anything managed by the container runtime, because it handles ownership and lifecycle. A bind mount when the extract must be readable by processes outside the container — a tile server, a PostGIS import — which is common enough that bind mounts are the pragmatic default in a mixed deployment. The locking and checkpoint discipline is identical either way.

Should the loop be inside the container or a CronJob?

If you already run Kubernetes, a CronJob with concurrencyPolicy: Forbid gives overlap prevention from the scheduler and one fewer thing to get right. On a single host, an in-container loop with a flock is simpler and has fewer moving parts than adding a scheduler. Both are correct; what must not happen is two mechanisms both thinking they own the cadence.

Can several containers sync different regions on one volume?

Yes, with a lock file per region rather than one global lock. Each container locks /state/<region>.lock and touches only its own extract and checkpoint. What must not be shared is the extract: two containers applying diffs to one file corrupt it regardless of how careful each one is on its own.

How do I back this up?

Snapshot the volume with the container stopped, or at minimum while the lock is held by your backup process rather than by the sync. Copying the extract and the checkpoint at different moments produces a backup whose checkpoint does not describe its data, which restores as a silently wrong state — the checkpoint-ahead-of-data case that Recovering from a Replication Sequence Gap shows cannot be repaired by replay.

Specification reference

flock(2) advisory locks are associated with the open file description and are released automatically when the last descriptor is closed, including on process death. Locks are visible to any process that opens the same file through the same filesystem, which for containers means the lock file must live on a shared mount rather than inside a container’s own writable layer.

Up one level: Building a Minutely Update Pipeline.