Continuous QA for OSM Pipelines

Validation that runs when somebody remembers is not validation; it is an occasional audit that tells you a problem has existed for an unknown length of time. The alternative is a gate that runs on every execution, knows what normal looks like, and stops the pipeline when the output stops resembling it.

The technical parts of this are straightforward — the rules from Writing Custom OSM Validation Rules in Python run perfectly well in a CI job. What makes continuous QA hard is everything around them: choosing thresholds that survive ordinary variation, deciding which failures should stop a deployment, and producing a result that a person reading it at eight in the morning can act on rather than dismiss. A gate people override every week is worse than no gate, because it consumes attention while providing no protection.

This topic covers the shape of that system. It assumes the rule-authoring material in this section and the pipeline structures in Parsing & Tag Normalization Workflows.

Three places validation can run, and what each one can still prevent Three panels. Validation on a pull request runs against a sample or a fixture and catches logic errors in the pipeline code before they are merged, but it cannot see production data volumes or real upstream changes. Validation after a build, before publication, runs against the full output and can still stop a bad dataset from reaching consumers, which is the only position where a gate genuinely gates. Validation after publication runs against live data and catches what the earlier stages missed, but by then consumers are already reading it and the remedy is a rollback rather than a prevention. Where a check sits decides what it can prevent On a pull request Sample or fixture data Catches code logic errors Blind to real volumes Cheap and fast Before publication Full production output Can still stop the release The only real gate Costs build minutes After publication Live data, real usage Catches what slipped Consumers already read it Remedy is rollback All three are worth having, but only the middle one prevents anything, and it is the one most often skipped for build-time reasons.
A check that runs after publication is monitoring. Only a check before it is a gate.

The Problem This Topic Solves

An OSM pipeline’s output degrades in ways that no exception reports. A parser update silently drops a tag that thirty thousand features depended on. An upstream extract arrives truncated and the pipeline processes it happily, producing a third of the usual rows. A normalisation rule that was correct for one region produces nonsense in another the first time that region is included. None of these throw; all of them produce a dataset that loads, queries and looks entirely plausible.

The only defence is to state what the output should look like and check it, every time. That statement has two halves: absolute invariants that must always hold — no null geometries, every identifier unique, coordinates within the world — and relative expectations derived from the last known-good run, which is where the interesting failures live. A drop from 4.1 million to 2.7 million buildings violates no invariant and is obviously wrong.

The reason this is not simply solved is that relative expectations are noisy. OSM genuinely changes; extracts genuinely vary; a mapping party genuinely adds forty thousand features in a weekend. A threshold tight enough to catch a truncated extract is loose enough to be tripped by a good week of mapping, and resolving that tension is most of the design work.

Prerequisites

What to Check, and in What Order

Checks are worth ordering by cost and by how early they can fail, because a cheap check that catches a truncated input should never run after an expensive one.

Input checks run first and cost nothing. Does the extract exist, is its size within the expected range, does its checksum differ from last time, does its internal timestamp advance? A truncated or stale input is the single most common upstream failure and it is detectable before any processing.

Structural checks run on the output and assert invariants. Every geometry valid, every identifier unique, no coordinates outside the world, no required column null. These are cheap, they never produce false positives, and they should fail hard with no threshold at all.

Statistical checks compare the output against history. Feature counts by type, tag coverage rates, geometry-area distributions, null rates per column. These are where truncated extracts and dropped tags are caught, and where thresholds have to be chosen rather than asserted.

Semantic checks run last because they are expensive. Routing graph connectivity, boundary closure, address completeness against a reference. They catch the problems that matter most to consumers and they cost enough that they are often sampled rather than run exhaustively.

A useful discipline is that each layer must pass before the next runs. Statistical checks against an output that failed structural checks produce noise, and a report full of consequential failures buries the one that caused them.

Choosing Thresholds That Survive an Ordinary Day

The temptation is to pick round numbers — alert on a ten percent change — and the result is a gate that fires on ordinary variation in small categories and never fires on a catastrophic change in large ones. Thresholds should come from the data.

The workable approach is to record each metric on every successful run, and derive the threshold from the observed distribution: a band of several standard deviations around a rolling median, or a percentile range from the last few weeks. This adapts to how variable each metric actually is, which differs by orders of magnitude between “number of buildings in Germany” and “number of ferry routes in a small region”.

Two refinements matter. Absolute floors catch the case where a metric’s history is itself wrong — after a series of degraded runs, a rolling band happily accepts continued degradation. A hard floor that says “never fewer than two million buildings” is crude and catches exactly that. Directional asymmetry reflects that in most OSM pipelines a sudden drop is far more likely to be a defect than a sudden rise, so the band should usually be tighter downward than upward.

The four check layers in execution order, with where each fails A timeline of four marks. Input checks run first, cost almost nothing, and catch a truncated, stale or duplicate extract before any processing happens. Structural checks run next against the output, assert invariants that admit no threshold, and fail hard on an invalid geometry or a duplicate identifier. Statistical checks compare the output against recorded history and catch dropped tags and partial data, which is where thresholds must be chosen from observed variation. Semantic checks run last because they are the most expensive, covering connectivity and completeness, and are often sampled rather than run exhaustively. Cheapest first, most expensive last Input free, runs first truncated or stale Structural invariants, no threshold invalid or duplicate Statistical compared to history dropped tags, partials Semantic expensive, often sampled connectivity, coverage Each layer must pass before the next runs, or a single structural failure fills the report with consequences that hide it.
Ordering by cost also orders by specificity: the cheapest checks give the clearest diagnoses.

Making the Gate Trustworthy

A gate is only useful if people believe it, and belief is built from two properties.

It must be actionable. A failure that says “statistical check failed” sends somebody on a hunt. A failure that says “building count in the Bavaria extract fell 34 percent against a 14-day median of 4.1 million, first observed in run 8821” tells them where to look and roughly what happened. The cost of producing the second message is a few lines in the check; the cost of not producing it is measured in the hours it takes somebody to re-derive it.

It must be rare. A gate that fires weekly gets overridden weekly, and the override becomes reflexive long before the failure that mattered arrives. If a check fires often, the correct response is to fix the check — widen the band, split the metric, exclude the volatile category — not to train people to ignore it.

The related decision is what a failure should do. Blocking publication is right for structural failures and for statistical failures on critical metrics. For everything else, publishing with a recorded warning is usually better, because a pipeline that refuses to publish over a minor anomaly teaches people to bypass it. The distinction should be explicit in the check definition rather than implied by severity labels nobody reads.

Four failure classes that raise no exception, and what catches each A grid of four silent failures against the check layer that detects them and the signal that layer sees. A truncated upstream extract is caught by input checks, which see a file size well below the recorded range. A parser regression that drops a tag is caught by statistical checks, which see a null rate jumping for one column while row counts hold steady. A normalisation rule wrong for a newly included region is caught by statistical checks segmented by region, which see the anomaly confined to one area. Topology damage from a geometry change is caught by semantic checks, which see routing connectivity fall while every feature remains individually valid. Silent failures and what sees them Caught by The signal Truncated extract input checks size below the range Dropped tag statistical checks null rate jumps, rows flat Rule wrong in a region segmented statistics anomaly in one area Topology damage semantic checks connectivity falls Every row produces an output that loads, queries and looks plausible, which is why none of them reaches anybody as an error.
The third row is the argument for segmenting metrics rather than aggregating them globally.

Validation and Error Handling

Check What it catches Response
Input size and checksum Truncated, stale or unchanged extract Fail before processing; no build minutes wasted
Geometry validity and identifier uniqueness Structural corruption in the output Block publication; no threshold applies
Feature counts against a rolling band Dropped tags, partial data, parser regressions Block on critical metrics, warn on the rest
Null-rate per column A normalisation rule that stopped matching Warn, and block above an absolute ceiling
Routing connectivity sample Topology damage from a geometry change Block; consumers cannot work around it
Check runtime trend A gate slowly becoming too slow to run Reassess sampling before it gets skipped

Performance and Scale

The practical constraint on continuous QA is that a gate people wait for gets skipped. If validation adds forty minutes to a twenty-minute build, somebody will propose running it nightly instead, and the gate stops being a gate.

Three techniques keep it affordable. Sampling applies to semantic checks, where validating a random ten thousand features gives a statistically sound estimate of a rate at a fraction of the cost — with the caveat that sampling detects rates, not individual catastrophes, so a rare but critical condition still needs an exhaustive check. Incremental checking validates only what changed, which pairs naturally with the affected-set work in Incremental Updates for Derived Datasets. Metric reuse avoids a separate scan per check by computing every count, rate and distribution in a single pass over the output and evaluating all thresholds against those aggregates afterwards.

That last one is the largest available saving and the most commonly missed. Twenty checks each scanning a continental dataset is twenty scans; one scan producing a metrics document that twenty checks then evaluate is one.

Failure Modes and Gotchas

A gate with no history. Relative checks need recorded metrics from previous runs, and a pipeline that does not store them can only assert invariants. Storing metrics is cheap and it is the prerequisite for everything statistical.

History poisoned by bad runs. A rolling band computed over runs that included a degradation quietly accepts that degradation as normal. Only record metrics from runs that passed, and keep an absolute floor as a backstop.

Thresholds on metrics nobody chose. It is easy to generate checks for every column automatically and end up with three hundred checks, most of which nobody understands, several of which fire regularly. A dozen chosen metrics that somebody can explain beat three hundred generated ones.

Blocking on the wrong things. A gate that blocks on cosmetic anomalies gets bypassed, and the bypass path then covers real failures too.

No owner. A failing gate with no named owner becomes a red mark people route around. This is an organisational failure mode and it destroys technically perfect systems routinely.

Integration Points

Continuous QA sits at the end of the pipeline and touches everything before it. It consumes the rules from Writing Custom OSM Validation Rules in Python and the topology checks in Routing Graph Topology QA. Its input checks overlap with the replication monitoring in Replication Monitoring & Lag Alerting, and the two should share thresholds rather than disagree about what stale means.

Downstream, the gate’s decision is what Incremental Updates for Derived Datasets and every export consumer depends on, which is the argument for placing it before publication rather than after.

Guides in This Topic

Frequently Asked Questions

Should quality checks run against the source data or the output?

Both, for different reasons. Checking the source catches upstream problems before you spend an hour processing them, and it distinguishes “the extract was bad” from “our pipeline broke”, which is the first question anybody asks. Checking the output is what actually protects consumers, because a pipeline can damage perfectly good input. The input checks are cheap enough that there is no reason to choose.

How do you validate data you have no reference for?

Against itself over time, which is what the statistical layer does. You may have no authoritative count of buildings in a region, but you have last week’s count, and a thirty percent move is informative without any external truth. Where an external reference does exist — a national address file, an official boundary set — it is worth using, but the absence of one is not a reason to skip validation.

What belongs in the gate versus in monitoring?

The gate answers “should this output be published”, and it must be fast enough to run inline and decisive enough to act on. Monitoring answers “is the system behaving”, runs continuously, and tolerates signals that are informative without being decisive. Trend detection belongs in monitoring; a single run’s pass or fail belongs in the gate. Confusing the two produces either a gate too slow to run or monitoring too coarse to be useful.

How many checks is the right number?

Few enough that somebody can explain every one of them, which in practice means somewhere between ten and thirty for a typical pipeline. The failure mode of too few is obvious; the failure mode of too many is that the report becomes unreadable and the regularly firing checks train people to skim it. Each check should have a name, an owner and a sentence explaining what its failure means.

Should a failing gate block a deployment or just warn?

Block for structural failures and for statistical failures on metrics the business actually depends on; warn for everything else. The decision belongs in the check’s definition, stated explicitly, rather than being inferred from a severity label. A gate that blocks on everything gets a bypass mechanism, and that bypass will eventually be used on the failure that mattered.

Up one level: OSM Data Quality & Validation.