Measuring OSM Replication Lag in Seconds

Answer the question “how old is my copy of OpenStreetMap” as a number of seconds, correctly, including the week the clocks change.

Prerequisites

Conceptual minimum

Four clocks are involved and only two of them are available to you.

The four timestamps involved in a replication lag calculation A four-stage chain of clocks. The diff timestamp records when the edits were cut and is carried in the applied state.txt. The publish time is when the file appeared upstream and is not recorded anywhere. The apply time is when you ingested it, from your own clock. Now is the moment the question is asked, always as an aware UTC value. Four clocks, and only one of them is yours diff timestamp when the edits were cut in the applied state.txt publish time when the file appeared not recorded anywhere apply time when you ingested it your own clock now the moment you ask aware UTC, always Data age is now minus the first clock. Pipeline delay is the third minus the second. Confusing the two is how a healthy pipeline gets blamed for an upstream stall.
Data age is now minus the diff timestamp; pipeline delay is apply time minus publish time. Only the first is computable from what the stream gives you.

The number worth reporting is data age: now, minus the timestamp of the newest diff you have applied. It answers what a consumer cares about and it degrades under every failure — a stopped loop, a slow loop, and an upstream stall alike. The number people often compute instead is the gap between sequence numbers, which is useful for a different question and reads as zero during an upstream stall.

The timestamp itself comes from the state.txt that accompanies the diff you applied, and that file has one quirk worth knowing before writing any parsing code: it is a Java properties file, so the colons inside the ISO timestamp are backslash-escaped.

text 
#Mon Aug 11 00:00:02 UTC 2026
sequenceNumber=6123456
timestamp=2026-08-11T00\:00\:00Z
Three timestamp handling mistakes in a lag calculation Three panels. A naive datetime from datetime.now with no timezone compared against a UTC stamp is off by the UTC offset and changes by an hour twice a year; the fix is datetime.now with timezone.utc. Unescaped colons: state.txt backslash-escapes the colons in its timestamp, so fromisoformat raises or a regex mis-parses; the fix is to unescape first. Wall clock for elapsed time: time.time deltas go backwards across an NTP step, giving negative ages or an alert that never fires; the fix is time.monotonic. Three ways the same calculation goes wrong Naive datetime datetime.now()` — no tzinfo Compared against a UTC stamp Off by the UTC offset Changes by an hour twice a year Fix: `datetime.now(timezone.utc) Unescaped colons state.txt writes `2026-08-11T00\:00\:00Z Colons are backslash-escaped fromisoformat` raises on it Or worse, a regex silently mis-parses Fix: replace `\\:` with `:` first Wall clock for elapsed time.time()` deltas for heartbeat NTP steps make it go backwards Negative ages, or never alerting Only affects elapsed, not absolute Fix: `time.monotonic() All three produce a number. None raises in the common case, and two of them are only wrong for part of the year.
Two of the three are seasonal. They work through testing, through the first deployment, and then produce a wrong number the week the clocks change.

Runnable solution

python 
#!/usr/bin/env python3
"""Compute OSM replication lag for a locally applied sequence."""
from __future__ import annotations

import logging
import time
import urllib.request
from dataclasses import dataclass
from datetime import datetime, timezone

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
logger = logging.getLogger(__name__)

BASE_URL = "https://planet.osm.org/replication/minute/"


@dataclass(frozen=True)
class State:
    sequence: int
    timestamp: datetime          # always aware, always UTC


def parse_state(text: str) -> State:
    """Parse an Osmosis state.txt. Colons in the timestamp are backslash-escaped."""
    fields: dict[str, str] = {}
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, _, value = line.partition("=")
        fields[key.strip()] = value.strip().replace("\\:", ":")
    stamp = datetime.fromisoformat(fields["timestamp"].replace("Z", "+00:00"))
    if stamp.tzinfo is None:                      # belt and braces: never return naive
        stamp = stamp.replace(tzinfo=timezone.utc)
    return State(sequence=int(fields["sequenceNumber"]), timestamp=stamp)


def sequence_path(sequence: int) -> str:
    """6123456 → '006/123/456' — the path is computed, never listed."""
    padded = f"{sequence:09d}"
    return f"{padded[0:3]}/{padded[3:6]}/{padded[6:9]}"


def fetch_state(url: str, timeout: float = 10.0) -> State:
    with urllib.request.urlopen(url, timeout=timeout) as response:
        return parse_state(response.read().decode("utf-8"))


def lag_seconds(applied_sequence: int, base_url: str = BASE_URL) -> dict[str, float]:
    """Data age, upstream age and sequence lag, computed from one pair of fetches."""
    head = fetch_state(f"{base_url}state.txt")
    applied = fetch_state(f"{base_url}{sequence_path(applied_sequence)}.state.txt")
    now = datetime.now(timezone.utc)
    result = {
        "data_age_seconds": (now - applied.timestamp).total_seconds(),
        "upstream_age_seconds": (now - head.timestamp).total_seconds(),
        "sequence_lag": float(head.sequence - applied.sequence),
    }
    logger.info(
        "applied seq %d (%s) · head seq %d (%s) · data age %.0f s",
        applied.sequence, applied.timestamp.isoformat(),
        head.sequence, head.timestamp.isoformat(),
        result["data_age_seconds"],
    )
    return result


class Heartbeat:
    """Elapsed time since the last success, measured on a clock that cannot go backwards."""

    def __init__(self) -> None:
        self._last = time.monotonic()

    def beat(self) -> None:
        self._last = time.monotonic()

    @property
    def age_seconds(self) -> float:
        return time.monotonic() - self._last


if __name__ == "__main__":
    print(lag_seconds(applied_sequence=6_123_456))

Step-by-step walkthrough

parse_state does the unescaping before anything else touches the value, which is the only place that quirk needs to be known. Everything downstream receives an aware UTC datetime and cannot reintroduce the problem. The defensive tzinfo is None branch exists because a state file with a timestamp lacking its Z — rare, but present on some mirrors — would otherwise return a naive value that poisons every comparison made with it.

sequence_path reproduces the three-level directory arithmetic described in Replication Sequence Numbers & State Tracking. Fetching the applied sequence’s own state.txt rather than remembering its timestamp locally is deliberate: it removes any possibility of the recorded timestamp disagreeing with the sequence, at the cost of one HTTP request.

lag_seconds returns three numbers from one consistent moment. Computing them from a single now matters more than it looks — calling datetime.now() separately for each would let the values disagree slightly and, on a slow link, noticeably.

Heartbeat uses time.monotonic rather than time.time. Elapsed-time measurements taken from the wall clock go backwards when NTP steps the clock, which yields a negative age and, depending on the comparison, either a spurious alert or an alert that can never fire again.

Four lag readings from one day of a healthy pipeline A bar chart of observed timestamp lag in seconds. The median is 47 seconds, one diff cut and applied. During a nightly reindex the lag reaches 210 seconds, a known scheduled dip. After a twenty-minute outage the lag peaks at 1240 seconds and returns to normal within six minutes. An upstream publishing gap shows 900 seconds with nothing wrong locally. What the number looks like when it is right one minutely-synced country extract, values sampled every minute for a day median lag 47 s — a diff, cut and applied during a nightly reindex 210 s — a known, scheduled dip after a 20-minute outage 1 240 s peak, back to normal in 6 min upstream publish gap 900 s — nothing wrong locally Knowing the shape of your own normal is the whole point of measuring: three of these four are healthy, and only the sustained version of any of them is not.
Three of these four readings come from a pipeline behaving correctly. Thresholds that do not account for them page someone on a Tuesday for nothing.

Verification

Run it against a pipeline you know to be current and expect a data age comparable to the cadence — under two minutes on a minutely stream. Then verify the failure directions rather than only the happy path:

bash 
# Data age should rise roughly one second per second while the loop is stopped.
systemctl stop osm-diff-sync.timer
sleep 300 && python3 lag.py     # expect ~300 s more than before
systemctl start osm-diff-sync.timer

Two assertions are worth keeping as tests. The parsed timestamp must be timezone-aware, which catches the naive-datetime regression the moment someone simplifies the parser. And data age must never be negative: a negative value means either a clock skew on your host or a checkpoint recording a sequence that has not been applied, and both deserve to fail loudly.

Common errors and fixes

Symptom Root cause Fix
Lag off by exactly one or two hours Naive datetime.now() compared with a UTC stamp Use datetime.now(timezone.utc)
ValueError: Invalid isoformat string Colons still backslash-escaped Unescape \: before parsing
Lag changes by an hour overnight in spring Local time used somewhere in the chain Keep everything aware and in UTC
Occasional negative heartbeat age time.time() used for elapsed Use time.monotonic()
Lag reads zero while data is stale Sequence lag reported instead of data age Report the timestamp difference
404 fetching the applied state Sequence padded to the wrong width Pad to nine digits, split 3/3/3

Frequently Asked Questions

Why fetch the applied sequence's state.txt instead of storing its timestamp?

Because the stored value can drift from the sequence it claims to describe — after a manual intervention, a partial restore, or a bug in the checkpoint write. Fetching it means the timestamp is by definition the one belonging to that sequence. The cost is one small HTTP request per measurement, which is negligible next to the diff fetches the loop already makes.

What is a normal lag on the minutely stream?

Between about thirty and ninety seconds for a healthy pipeline, with a long right tail from publishing jitter. Anything below thirty seconds means you are fetching faster than the stream publishes, and anything sustained above a few minutes means the loop is not keeping up. Measure your own distribution over a month before choosing thresholds.

Should the lag include the time my own processing takes?

That depends on what you promise consumers. Data age as computed here measures the age of the edits in your database. If downstream artefacts — tiles, a routing graph — are rebuilt on a slower cadence, their age is later still, and a consumer of those needs the age of the artefact, not of the database row. Measure both if both are published.

Can I compute lag without any network access?

Partly. Now minus the applied diff’s timestamp is computable from local state alone, provided the timestamp was recorded alongside the sequence at apply time. What needs the network is the upstream head, and without it you cannot distinguish your pipeline falling behind from the stream having stopped.

Where to compute it

The measurement can live in three places and the choice affects what it can detect.

Inside the loop, computed at the end of each iteration, it is cheapest and most accurate — the applied sequence is already in hand and no extra state file read is needed. Its weakness is that it stops being computed exactly when the loop stops, which is the moment you most want a number. A gauge frozen at a healthy value is indistinguishable from a healthy pipeline unless the alerting rule also treats staleness of the metric itself as a fault.

In a sidecar process reading the same checkpoint, it keeps reporting when the loop dies, which turns a silent freeze into a visibly rising number. The cost is a second reader of the checkpoint, which is safe for a file written by atomic rename and is not safe for one written in place — another reason the checkpoint-writing discipline in Building a Minutely Update Pipeline matters beyond crash safety.

In a remote prober that only knows the public artefact — the timestamp embedded in a published file, or an endpoint your service exposes — it measures what a consumer actually experiences, including any delay between the database being current and the artefact being rebuilt. It cannot distinguish which stage is behind, so it complements the other two rather than replacing them.

Most pipelines want the sidecar as the primary source and, if anything is published externally, a remote prober as the check that the promise made to consumers is being kept.

Up one level: Replication Monitoring & Lag Alerting.