Conflation QA & Rollback

Conflation is the one area of this site where the reviewer is often somebody who did not build the pipeline and has no reason to trust it. That shapes what quality assurance has to produce: not a green build, but evidence.

The four artefacts a conflation run should produce before anything ships Four artefacts in order. The labelled sample is a few hundred human-judged pairs, maintained over time, against which every matcher change is scored. The metrics are precision and recall computed against that sample, reported per region and per feature class rather than only in aggregate. The evidence pack is a set of representative before-and-after examples and counts that a reviewer can check without running anything. The revert plan describes how the upload is structured so that undoing one decision does not disturb the others. Four artefacts, none of them optional labelled sample a few hundred pairs maintained over time metrics precision and recall per region, per class evidence pack examples and counts checkable without code revert plan one decision per change tested, not assumed The fourth artefact is the one teams write after they need it, which is exactly when it is most expensive to produce.
Each artefact answers a different reviewer question, and none of them can be reconstructed after the upload.

The Problem This Topic Solves

You have a conflation run ready and you need to know whether it is good enough to act on — and, if it turns out not to be, how to undo it. Both questions have concrete answers, and both are much easier to answer before the run than after.

The failure scenario is a run that nobody can evaluate. The matcher reports 94 percent of records matched, which sounds excellent until somebody asks what fraction of those matches are correct and there is no way to say. A sample is taken by hand, 12 percent turn out to be wrong, and the run has already been applied. Undoing it is impossible because the changes were applied in place with no record of which came from which match. Every part of that is preventable with artefacts produced before the run, not after.

Precision and Recall, Measured Rather Than Estimated

Two numbers describe a matcher’s quality, and both need a labelled sample to compute.

Precision is the fraction of proposed matches that are correct. It answers “when this says match, how often is it right?” — the question that matters when a wrong match creates bad data.

Recall is the fraction of true matches that were proposed. It answers “how much did this miss?” — the question that matters when a missing match leaves a gap.

They trade against each other, and which one to favour depends entirely on what happens downstream. For an import, precision dominates: a wrong match creates a duplicate or a bad edit in the public map, while a missed match simply leaves a feature unmapped for now. For an internal enrichment, recall may matter more, since a missing attribute is a hole in a dataset and a slightly wrong one may be tolerable.

The labelled sample that makes both computable is a few hundred pairs judged by a human, maintained over time, and re-scored on every matcher change. That is a day of work once and half an hour per change afterwards. Measuring Conflation Precision and Recall covers the sampling design, which matters more than the arithmetic.

Aggregate Numbers Hide the Failures That Matter

A single precision figure over a national run is almost useless, because conflation quality varies enormously with context. Three breakdowns are worth reporting every time.

By density. Dense urban areas are where distance stops discriminating and precision falls. A run with 96 percent precision overall may be at 99 in rural areas and 78 in city centres.

By feature class. Named, categorised features match well; unnamed generic ones do not. Reporting per class shows which parts of the dataset are trustworthy.

By source region. Where the external dataset was assembled differently in different regions — a common situation with national datasets built from municipal contributions — quality follows those boundaries.

How one aggregate precision figure decomposes across contexts A grid showing a run reporting ninety four percent precision overall broken down three ways. By density, rural areas reach ninety nine percent while dense urban areas fall to seventy eight. By feature class, named and categorised features reach ninety seven while unnamed generic ones fall to seventy one. By region, the region where the external source was well maintained reaches ninety six while the one assembled from older municipal data falls to eighty three. A note observes that all three breakdowns come from the same labelled sample at no extra cost. The same run, read three different ways Best slice Worst slice Overall 94% precision 94% precision By density rural: 99% dense urban: 78% By feature class named: 97% unnamed: 71% By region maintained: 96% legacy data: 83% Every one of these breakdowns comes free from the same labelled sample; only reporting the first row is a choice.
The aggregate is a weighted average of slices that behave nothing like each other, which is why it explains nothing.

The Evidence Pack

A reviewer who did not build the pipeline needs something they can check without running anything. That is the evidence pack, and it has four parts.

Counts, broken down as above: how many records, how many matched, how many routed to review, how many left unmatched.

Representative examples: twenty or so before-and-after pairs sampled across the score range, including a few deliberately near the threshold, so a reviewer can see what a marginal decision looks like.

The decision rules: the thresholds, the precedence rules and the cardinality rule, stated in plain words rather than as code.

The known limitations: where the run is weakest and why. A pack that admits its weak spots is enormously more credible than one that does not, and a reviewer will find them anyway.

For an upload this pack is what accompanies the import plan; for an enrichment it is what accompanies the dataset. Auditing a Conflation Run Before Upload assembles one.

Reviewing the Review Queue

The group routed to human review is a deliverable in its own right, and how it is presented determines whether it gets worked through or quietly abandoned.

Size it for the people you have. A queue of two hundred pairs is an afternoon; a queue of twenty thousand is a project nobody will start. If the matcher produces a queue larger than the available attention, the answer is to improve the signals rather than to raise the thresholds — raising them moves uncertain pairs into the confident bucket without making them any less uncertain.

Order it by value, not by score. The pairs worth a human’s time first are the ones where a decision unlocks the most: a record matching a feature that many other records also touch, or a pair whose resolution would settle an ambiguity affecting a whole area. Ordering strictly by score puts the most marginal cases first, which is the least rewarding place to start.

Show the evidence, not the number. A reviewer deciding a pair needs the component scores, the names as they actually appear, the distance, and the runner-up — not a combined figure. Every one of those is already computed; presenting them is a formatting decision that changes review speed by a large factor.

Feed the decisions back. A reviewed pair is a labelled pair, and labelled pairs are exactly what the measurement in Measuring Conflation Precision and Recall needs. A review queue whose decisions are discarded after use throws away the most expensive data the project generates.

Rollback: Structure It Before You Need It

A revert is only as easy as the structure of what was applied.

For an enrichment, reverting means re-deriving from stored inputs, which works only if nothing was overwritten in place. The layered design in Attribute Enrichment from Authoritative Sources makes this nearly free.

For an import, reverting means undoing changesets — and that is easy when each changeset was small and single-purpose, and extremely painful when one changeset carried thousands of unrelated objects. The structure decision is made at upload time and cannot be changed afterwards.

Two things make a revert tractable regardless of direction. A record of which source record produced which change, so a partial revert is possible. And a rehearsal: reverting one small changeset on the development instance, before the real rollout, proves the path works. Rolling Back a Bad OSM Import works through the real case, including what to do when other mappers have edited the affected objects since.

What a revert costs under three different upload structures Three panels. Small single-purpose changesets let a reviewer revert exactly the decision that was wrong, costing minutes and affecting only the objects involved. A few large changesets force an all-or-nothing choice, so undoing one mistake destroys every correct change uploaded alongside it. In-place changes with no record of previous values cannot be reverted at all, and recovery means reconstructing the earlier state from a historical extract if one exists. The revert cost is decided at upload time Small, single purpose Revert exactly one decision Minutes of work Only the objects involved The structure to aim for A few large changesets All or nothing per changeset Good changes destroyed too Reviewer picks the lesser evil Avoidable by splitting In place, no record Cannot be reverted at all Previous values are gone Reconstruct from history If a history file exists Nothing about the third panel can be improved after the fact, which is why the structure is a design decision rather than an operational one.
All three take the same effort to upload; only the first takes a reasonable effort to undo.

The Signal That Something Has Drifted

A conflation pipeline that ran well six months ago and runs badly now usually gives three warnings before anybody notices, and all three are cheap to watch.

The no-match rate moves. A sudden rise means either the external source changed shape or the OSM side stopped being found — a projection change, a filter that started excluding a feature class, a candidate radius now wrong for a refreshed source. A sudden fall is just as suspicious, because it usually means the matcher started accepting pairs it previously rejected.

The review queue changes size. The band between the two thresholds should hold a roughly stable share of records. A queue that doubles without a threshold change means the score distribution moved, which means a signal changed — often a name field that started arriving empty, or a category vocabulary that was silently renamed upstream.

The component mix changes. Tracking which signal contributed most to each confident match, aggregated per run, is the most sensitive alarm available. A pipeline where name similarity used to carry most decisions and now leans on distance has lost a signal without losing a number, and the precision cost of that will show up weeks later in complaints rather than immediately in a metric.

None of these needs a labelled sample; all three come free from the run’s own output. Recording them per run and comparing against the previous one turns a slow, invisible degradation into a line on a chart somebody can point at.

Validation and Error Handling

Condition Root cause Detection Remediation
Quality cannot be stated No labelled sample No precision figure exists Label a few hundred pairs once; maintain them
Good aggregate, bad in practice Only aggregate reported Complaints concentrate in one area Report per density, class and region
Reviewer cannot evaluate No evidence pack Questions require running the pipeline Publish counts, examples, rules and limitations
Revert impossible Changes applied in place No original values remain Never overwrite; derive the resolved value
Partial revert impossible One enormous changeset Undoing one thing undoes everything Small, single-purpose changesets
Revert fails on conflicts Objects edited since the upload Version conflicts during the revert Re-read, and route conflicting objects to a human
Metrics drift unnoticed Sample scored once, never again Quality falls with no signal Re-score the sample on every matcher change

Guides in This Topic

Frequently Asked Questions

Why is a match rate not a quality measure?

Because it counts proposals rather than correct proposals. A matcher that pairs every record with its nearest feature achieves a match rate near 100 percent and a precision that may be dreadful. The two numbers that matter are precision — of the matches proposed, how many are right — and recall — of the true matches available, how many were found. Neither can be computed without a human-labelled sample.

How large does the labelled sample need to be?

A few hundred pairs is enough for a useful precision figure and, more importantly, enough to break down by density and feature class. Stratifying matters more than raw size: a sample drawn uniformly from a national dataset is dominated by rural areas where matching is easy, and will report a flattering number that says nothing about the city centres where the failures live.

Should I optimise for precision or recall?

For what happens downstream. An import should favour precision heavily, because a wrong match puts bad data on a public map while a missed match leaves a feature unmapped for a while longer. An internal enrichment can often favour recall, because a missing attribute is a visible hole and a slightly wrong one may be acceptable. Decide explicitly and record the decision, because the thresholds follow from it.

What makes a revert hard?

Two things, both decided before the revert is needed. Changes applied in place, with no record of the previous value, cannot be undone at all. And changes grouped into a few enormous changesets cannot be undone selectively, so a reviewer who finds one problem must choose between accepting it and destroying everything else. Both are structural decisions made at upload time and neither can be fixed afterwards.

What if somebody has edited the objects since my upload?

Then the revert conflicts on those objects, and that is the correct behaviour rather than an obstacle. Somebody looked at the data and changed it, which means a blind revert would discard their work. Re-read the conflicting objects, revert the ones still in the state you created, and route the rest to a human. A revert that forces its way through conflicts is a second bad edit on top of the first.

Up one level: OSM Conflation & Data Enrichment.