Running OSM Diff Sync on Kubernetes with a StatefulSet

A replication loop is stateful, single-writer and long-running, which is nearly the opposite of what a Deployment assumes — and two pods applying the same diff to the same database is a corruption, not a race you can retry.

Prerequisites

Conceptual minimum

Three properties of the workload decide the shape of the manifest.

It must not run twice. A Deployment during a rolling update deliberately runs old and new pods together, and a Deployment whose node becomes unreachable may start a replacement while the original is still running and still writing. A StatefulSet with replicas: 1 gives at-most-one semantics: Kubernetes will not create the replacement until the original is confirmed gone, which is exactly the trade — availability sacrificed for the guarantee you need.

Its state is on disk. The sequence marker, the partially downloaded diffs and the working files all live on a volume. A StatefulSet’s volumeClaimTemplates binds the same claim to the pod each time it is scheduled, so a restart resumes rather than restarts.

It is not a request handler. The default readiness and liveness semantics assume a server. For a loop, “alive” means it is making progress, and “ready” is meaningless — there is no traffic to gate. Wiring a liveness probe to data freshness rather than process existence is what turns a silently stalled loop into a restart.

The single most consequential setting is terminationGracePeriodSeconds. A loop killed mid-apply leaves the database and the sequence marker disagreeing, and recovering from that costs far more than waiting for the current diff to finish.

Why a Deployment is the wrong controller for a replication loop Three panels. A Deployment performs rolling updates by design, starting the new pod before terminating the old one, so two replication loops write to the same database during every deploy. A Deployment also replaces a pod on an unreachable node without confirming the original has stopped, since from the control plane an unreachable node is indistinguishable from a slow one. A StatefulSet with one replica gives at-most-one semantics, never starting a replacement until the original is confirmed terminated, and binds the same persistent volume to the pod on every reschedule so the sequence marker survives. Deployment, node failure, StatefulSet Rolling update New pod before old stops Two loops, one database Happens every deploy By design, not a bug Unreachable node Replacement started Original may still write Cannot be distinguished Corruption, not a race StatefulSet, one replica At most one, guaranteed Waits for confirmed exit Same volume every time Availability traded away The third panel is less available than the other two, and that is the point: a stalled loop is recoverable, a double-applied diff is not.
Choose the controller for the guarantee it gives, not the uptime it promises.

Runnable solution

yaml 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: osm-diff-sync
spec:
  replicas: 1                      # more than one corrupts the database
  serviceName: osm-diff-sync
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate            # with 1 replica this is stop-then-start
  selector:
    matchLabels: { app: osm-diff-sync }
  template:
    metadata:
      labels: { app: osm-diff-sync }
    spec:
      # Long enough for the current diff to finish. A loop killed mid-apply
      # leaves the database and the sequence marker disagreeing.
      terminationGracePeriodSeconds: 900
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        fsGroup: 10001
      containers:
        - name: sync
          image: registry.example.net/osm-diff-sync:2026.09.17
          args: ["--state-dir", "/var/lib/osm", "--interval", "60"]
          env:
            - name: REPLICATION_URL
              value: https://planet.openstreetmap.org/replication/minute
            - name: PGHOST
              valueFrom:
                secretKeyRef: { name: osm-db, key: host }
          ports:
            - { name: metrics, containerPort: 9187 }
          volumeMounts:
            - { name: state, mountPath: /var/lib/osm }
          resources:
            requests: { cpu: "1", memory: 4Gi }
            limits:   { memory: 8Gi }        # no CPU limit: throttling stalls apply
          # "Alive" means making progress, not "the process exists". A loop
          # whose lock is stuck answers a TCP check perfectly while doing
          # nothing at all.
          livenessProbe:
            httpGet: { path: /healthz/freshness, port: metrics }
            initialDelaySeconds: 300
            periodSeconds: 60
            failureThreshold: 10             # 10 minutes of staleness
          startupProbe:
            httpGet: { path: /healthz/started, port: metrics }
            failureThreshold: 60
            periodSeconds: 10
  volumeClaimTemplates:
    - metadata:
        name: state
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests: { storage: 200Gi }
python 
"""The freshness endpoint the liveness probe reads.

Reporting process existence is worthless here: the failure mode is a loop
that runs, logs and exits zero every minute while applying nothing.
"""
from __future__ import annotations

import json
import logging
import time
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.k8s.health")

STATE_DIR = Path("/var/lib/osm")
MAX_STALE_S = 600


def freshness() -> tuple[bool, dict]:
    marker = STATE_DIR / "applied.json"
    if not marker.exists():
        return False, {"reason": "no marker yet"}
    data = json.loads(marker.read_text(encoding="utf-8"))
    age = time.time() - data["applied_at"]
    return age < MAX_STALE_S, {"sequence": data["sequence"],
                               "age_seconds": round(age, 1)}


class Handler(BaseHTTPRequestHandler):
    def do_GET(self) -> None:                      # noqa: N802
        if self.path == "/healthz/started":
            ok, body = STATE_DIR.exists(), {"state_dir": str(STATE_DIR)}
        elif self.path == "/healthz/freshness":
            ok, body = freshness()
        else:
            ok, body = False, {"reason": "unknown path"}
        payload = json.dumps(body).encode("utf-8")
        self.send_response(200 if ok else 503)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, *args) -> None:          # probes are noisy
        return


if __name__ == "__main__":
    logger.info("serving freshness on :9187")
    HTTPServer(("", 9187), Handler).serve_forever()

Step-by-step walkthrough

  1. Set replicas: 1 and mean it. Scaling this workload is not a capacity lever, and an autoscaler pointed at it is a corruption waiting for load.
  2. Use volumeClaimTemplates, not a shared claim. The template binds storage to the pod’s identity, which is what makes a reschedule a resume.
  3. Give a long grace period. Fifteen minutes lets the current diff complete; the alternative is a partially applied change and a marker that disagrees with the database.
  4. Handle SIGTERM in the loop. The grace period only helps if the process finishes the current diff and exits rather than dying at the first signal.
  5. Probe freshness, not liveness. The characteristic failure is a loop that appears healthy by every process-level measure while applying nothing.
  6. Set a startup probe. Initial catch-up can take hours, and a liveness probe without a startup probe restarts the pod repeatedly during it.
  7. Omit the CPU limit. CPU throttling during an apply extends it past the probe threshold, which restarts a pod that was merely slow.
  8. Pin the image by digest or dated tag. A loop restarting onto an unexpected version mid-incident is a second problem during the first.
What a graceful termination has to accomplish, in order Four steps. Kubernetes sends SIGTERM and starts the grace-period countdown. The loop stops accepting new work, meaning it will not begin the next diff, but continues the one in progress. The in-flight diff completes its apply and the sequence marker is written, which is the atomic point that makes the state consistent. The process exits zero well inside the grace period, and Kubernetes proceeds. If the grace period expires first, SIGKILL arrives mid-apply and leaves the database and the marker disagreeing, which requires manual reconciliation. SIGTERM to a consistent stop SIGTERM countdown begins grace period running stop taking work no new diff started current one continues finish and mark apply completes marker written, atomic exit zero inside the grace period state consistent If the countdown expires at the third step, SIGKILL leaves the database ahead of the marker and reconciliation becomes manual.
The grace period must exceed the slowest realistic diff, not the average one.
Six manifest settings and the specific failure each one prevents A grid of six settings against the failure they prevent and what happens at the default. One replica prevents two loops writing the same database, where the default of scaling freely corrupts the sequence. Volume claim templates prevent state loss on reschedule, where an emptyDir restarts the catch-up from nothing. A long termination grace period prevents a mid-apply kill, where the thirty-second default leaves the marker and database disagreeing. A freshness liveness probe prevents a silently stalled loop, where a process check reports health forever. A startup probe prevents restart loops during the initial catch-up. Omitting the CPU limit prevents throttling from extending an apply past the probe threshold. Setting, failure prevented, and the default Prevents At the default replicas: 1 two writers sequence corrupted volumeClaimTemplates state loss catch-up from nothing grace period 900s mid-apply kill 30s, marker disagrees freshness liveness silent stall healthy forever startupProbe restart loop killed during catch-up no CPU limit throttled apply false restarts Every default in the right column is the correct choice for a request handler, which is what the defaults were written for.
None of these are tuning. Each removes one specific way this workload breaks.

Verification

  • Only one pod ever runs. Trigger a rolling update and watch; the old pod should terminate before the new one starts.
  • State survives a delete. Delete the pod and confirm the replacement resumes from the recorded sequence.
  • Freshness gates liveness. Pause the loop artificially and confirm the probe fails and the pod restarts.
  • Termination is graceful. Delete the pod mid-apply and confirm the diff completes before exit.
  • Startup does not thrash. On an empty volume, confirm the initial catch-up completes without restart loops.

Common errors and fixes

Symptom Root cause One-line fix
Duplicate or skipped diffs Deployment rolling two pods together Use a StatefulSet with one replica
State lost on reschedule emptyDir or a shared claim Use volumeClaimTemplates
Marker disagrees with the database Pod killed mid-apply Raise terminationGracePeriodSeconds, handle SIGTERM
Pod healthy while data is stale Liveness probing the process Probe a freshness endpoint instead
Restart loop on first deploy No startup probe during catch-up Add a startupProbe with a long threshold
Random restarts under load CPU limit throttling the apply Remove the CPU limit; keep the memory limit
Unexpected version after a restart Mutable image tag Pin by digest or a dated tag

Specification reference

A StatefulSet maintains at most one pod with a given identity at any time. Unlike a Deployment, the controller will not create a replacement pod until the previous one is confirmed terminated; when a node becomes unreachable, the pod is not deleted until the node object is removed or the pod is force-deleted. Volumes provisioned from volumeClaimTemplates are bound to the pod identity and reattached on reschedule. See the Kubernetes StatefulSet documentation, and the pod-lifecycle documentation for termination and probe semantics.

Frequently Asked Questions

Would a CronJob be simpler than a long-running loop?

It looks simpler and it is not. A CronJob gives no mutual exclusion — a run that overshoots its schedule overlaps the next one unless concurrencyPolicy: Forbid is set, and even then the semantics around missed schedules and a stuck job need care. A long-running loop with an internal interval makes the exclusion structural rather than configured, and the StatefulSet supplies the at-most-one guarantee the CronJob would need to be told about.

What about force-deleting a pod on an unreachable node?

Only when you have independently confirmed the original is not writing — by checking the database’s connection list, or by confirming the node is genuinely powered off. Force deletion tells Kubernetes to stop waiting for confirmation, which is exactly the guarantee protecting the database. Doing it reflexively to clear a stuck pod is how two loops end up running, and the symptom is a corrupted sequence rather than an error message.

How large should the persistent volume be?

Large enough for the working extract plus several times the largest diff, plus headroom for osmium temporary files during an apply, which can approach the size of the file being rewritten. Undersizing produces a failure mid-apply that looks like corruption, and disk is cheap relative to the diagnosis. Alerting on volume utilisation is worth more here than in most workloads because the failure is destructive rather than merely inconvenient.

Should the database run in the same cluster?

It can, but the replication loop should not assume it. The loop’s correctness depends on its own state volume and the database’s transactional guarantees, not on their co-location, and keeping them independent means a cluster upgrade affects availability rather than consistency. What does matter is that the loop’s sequence marker and the database write land in the same transaction where possible, which is a database-side design question rather than a Kubernetes one.

Up one level: Building a Minutely Update Pipeline.