Measuring Conflation Precision and Recall

Two numbers, an afternoon of labelling, and a habit of re-scoring — that is the whole of conflation quality measurement, and it is the difference between a matcher you can defend and one you merely hope about.

Prerequisites

Conceptual minimum

Precision is correct matches divided by proposed matches. Recall is correct matches divided by true matches that exist. Computing precision is easy: sample the proposals and judge them. Computing recall is harder, because it needs the denominator — pairs that should have matched, including ones the matcher never proposed.

That asymmetry drives the sampling design. A precision sample is drawn from the matcher’s output. A recall sample must be drawn from the external records, then each one investigated to determine whether a true counterpart exists at all, which is slower per item and cannot be avoided if recall is to mean anything.

The second idea is stratification. A uniform sample of a national dataset is dominated by easy rural cases, and the resulting figure is a flattering average over slices that behave nothing alike. Sampling within strata — by density, by feature class — and reporting per stratum is what makes the numbers informative.

Why the two metrics need two different samples Three panels. A precision sample is drawn from the matches the system proposed and asks of each whether it is correct, which is quick to judge because both sides are in front of the labeller. A recall sample is drawn from the external records regardless of outcome and asks whether a true counterpart exists in OSM at all, which requires searching and is several times slower per item. The stratification panel explains that both samples must be drawn within strata, because a uniform draw is dominated by the easy cases and produces an average that describes no real slice. Two metrics, two samples, one stratification Precision sample Drawn from proposals Is this match correct? Both sides in front of you Fast to judge Sample a few hundred Recall sample Drawn from all records Does a counterpart exist? Requires searching OSM Several times slower Sample fewer, stratified Stratification Both samples, within strata Density and feature class Uniform favours easy cases Report per stratum Never only the aggregate Teams measure precision and skip recall because the sampling is harder, and then cannot say what their matcher is missing.
Recall is the expensive number and the one that reveals whether the candidate radius was ever right.

Runnable solution

python 
from __future__ import annotations

import logging
import math
import random
from collections import defaultdict
from dataclasses import dataclass

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.conflate.metrics")


@dataclass(frozen=True)
class Labelled:
    record_id: str
    stratum: str
    proposed: bool          # did the matcher propose a match for this record?
    correct: bool | None    # was the proposal right? None when none was made
    truth_exists: bool      # does a true counterpart exist in OSM at all?


def wilson(successes: int, total: int, z: float = 1.96) -> tuple[float, float]:
    """Wilson score interval: honest at small n, unlike the naive one.

    A naive interval around 19/20 extends above 1.0, which is not a
    probability; Wilson stays inside the unit interval and is the right
    default for the sample sizes hand labelling can afford.
    """
    if total == 0:
        return (0.0, 0.0)
    p = successes / total
    denom = 1 + z * z / total
    centre = (p + z * z / (2 * total)) / denom
    spread = z * math.sqrt(p * (1 - p) / total + z * z / (4 * total * total)) / denom
    return (max(0.0, centre - spread), min(1.0, centre + spread))


def stratified_sample(records: list[dict], per_stratum: int,
                      stratum_key: str = "stratum",
                      seed: int = 20260917) -> list[dict]:
    """Draw an equal number from each stratum, reproducibly."""
    by_stratum: dict[str, list[dict]] = defaultdict(list)
    for record in records:
        by_stratum[record[stratum_key]].append(record)

    rng = random.Random(seed)       # fixed seed: the sample must be re-drawable
    sample: list[dict] = []
    for stratum, members in sorted(by_stratum.items()):
        take = min(per_stratum, len(members))
        if take < per_stratum:
            logger.warning("stratum %s has only %d record(s)", stratum, len(members))
        sample.extend(rng.sample(members, take))
    logger.info("drew %d record(s) across %d stratum(s)",
                len(sample), len(by_stratum))
    return sample


def metrics(labels: list[Labelled]) -> dict[str, dict[str, float]]:
    """Precision and recall overall and per stratum, with intervals."""
    buckets: dict[str, list[Labelled]] = defaultdict(list)
    for label in labels:
        buckets["ALL"].append(label)
        buckets[label.stratum].append(label)

    out: dict[str, dict[str, float]] = {}
    for name, group in sorted(buckets.items()):
        proposed = [l for l in group if l.proposed]
        correct = [l for l in proposed if l.correct]
        truths = [l for l in group if l.truth_exists]
        found = [l for l in truths if l.proposed and l.correct]

        precision = len(correct) / len(proposed) if proposed else 0.0
        recall = len(found) / len(truths) if truths else 0.0
        p_lo, p_hi = wilson(len(correct), len(proposed))
        r_lo, r_hi = wilson(len(found), len(truths))
        f1 = (2 * precision * recall / (precision + recall)
              if precision + recall else 0.0)

        out[name] = {
            "precision": precision, "precision_lo": p_lo, "precision_hi": p_hi,
            "recall": recall, "recall_lo": r_lo, "recall_hi": r_hi,
            "f1": f1, "n_proposed": len(proposed), "n_truths": len(truths),
        }
        logger.info("%-16s precision %.2f [%.2f-%.2f]  recall %.2f [%.2f-%.2f] "
                    "(n=%d/%d)", name, precision, p_lo, p_hi,
                    recall, r_lo, r_hi, len(proposed), len(truths))
    return out


def regression_check(current: dict, baseline: dict, tolerance: float = 0.03) -> bool:
    """Fail when any stratum's precision fell, not just the aggregate."""
    ok = True
    for stratum, values in current.items():
        before = baseline.get(stratum)
        if before is None:
            continue
        drop = before["precision"] - values["precision"]
        if drop > tolerance:
            logger.error("%s precision fell %.3f (%.2f -> %.2f)", stratum, drop,
                         before["precision"], values["precision"])
            ok = False
    return ok


if __name__ == "__main__":
    logger.info("label once, re-score on every matcher change")

Step-by-step walkthrough

  1. Fix the random seed. A sample that cannot be redrawn identically is a sample whose labels cannot be reused, which defeats the whole point of labelling.
  2. Sample equally per stratum, not proportionally. Proportional sampling reproduces the dataset’s imbalance and leaves the interesting strata with too few items to say anything about.
  3. Warn on thin strata. A stratum with fewer records than the target sample size will produce a wide interval, and knowing that in advance prevents over-reading its number.
  4. Record three facts per label. Whether a match was proposed, whether it was correct, and whether a true counterpart exists. Precision needs the first two; recall needs the first and third.
  5. Use a Wilson interval. At the sample sizes hand labelling affords, the naive interval extends outside the unit range and understates uncertainty near the extremes. Wilson does neither.
  6. Report the interval, not just the point. “Precision 0.94” from twenty items means something very different from the same figure from four hundred, and the interval is what makes that visible.
  7. Regression-check per stratum. An aggregate that holds steady while one stratum falls sharply is the normal shape of a quality regression, and only a per-stratum check catches it.
How the confidence interval narrows as the labelled sample grows Five sample sizes with the width of the Wilson interval around a precision of ninety percent. At twenty labelled items the interval spans roughly thirty percentage points, which is too wide to distinguish good from mediocre. At fifty it spans about eighteen. At one hundred it spans about twelve. At three hundred it spans about seven. At one thousand it spans about four. A note observes that three hundred per stratum is the usual sweet spot between labelling effort and a usable interval. Interval width around 90% precision, by sample size 20 labelled about 30 points 50 labelled about 18 points 100 labelled about 12 points 300 labelled about 7 points 1000 labelled about 4 points Below about fifty items per stratum the interval is wider than the differences you are trying to detect between matcher versions.
Three hundred per stratum is where labelling effort and interval width stop trading well against each other.
What a labelled sample is worth once it exists Four uses of one labelled sample, in increasing order of value. Scoring it once establishes where the matcher currently stands. Re-scoring on each change turns every matcher edit into a measured improvement or regression rather than an opinion. Calibrating thresholds against it replaces guessed cut-offs with ones chosen to hit a stated precision. Regression-checking every stratum turns a silent quality drift into a failed check that names the slice that moved. One afternoon of labelling, four returns score once where you stand a number, not a hope re-score per matcher change improvement or regression calibrate thresholds from evidence hit a stated precision regress-check every stratum names the slice Only the first of these needs the labelling effort; the other three are free once the labels exist and are kept.
Teams that label once and never re-score have paid the whole cost and collected a quarter of the benefit.

Verification

  • The sample redraws identically. Run the sampler twice with the same seed and confirm the two samples are the same.
  • Strata are balanced. Every stratum should have the target count, or a warning explaining why not.
  • Intervals stay inside the unit range. A reported upper bound above one means the naive interval is being used.
  • Precision and recall move in opposite directions. Tighten a threshold and confirm precision rises while recall falls; if both move together, something is wrong with the labels.
  • The regression check catches a planted drop. Degrade one stratum deliberately and confirm the check fails.

Common errors and fixes

Symptom Root cause One-line fix
Flattering precision, real complaints Uniform sample dominated by easy cases Stratify and sample equally per stratum
Labels cannot be reused Sample drawn without a fixed seed Seed the sampler and record the seed
Recall cannot be computed Only proposals labelled Label records regardless of outcome for the denominator
Confidence bound above one Naive interval used Use a Wilson score interval
Regression missed Only the aggregate compared Compare every stratum against its own baseline
Numbers drift over releases Sample scored once and never again Re-score on every matcher change
Both metrics move together Labels inconsistent between sessions Write down the labelling rule and apply it uniformly

Specification reference

The Wilson score interval for a binomial proportion is centred on a shrunken estimate and remains within the unit interval for all sample sizes, unlike the normal-approximation interval which can extend beyond it and understates uncertainty for proportions near zero or one. It is the recommended default for the small samples typical of hand-labelled evaluation sets. See any standard treatment of binomial proportion confidence intervals for the derivation and the comparison against alternatives.

Frequently Asked Questions

Why sample equally per stratum rather than proportionally?

Because a proportional sample reproduces the dataset’s imbalance, and the strata you most need to measure — dense urban areas, unnamed feature classes — are usually the small ones. Sampling equally gives each stratum enough items for a usable interval, at the cost of the overall figure no longer being a population estimate. That is the right trade, because the per-stratum numbers are the useful ones anyway.

How do I measure recall without labelling everything?

By sampling from the external records rather than from the matcher’s output, and determining for each sampled record whether a true OSM counterpart exists at all. That gives the denominator recall needs. It is several times slower per item than precision labelling, because it involves searching rather than judging, which is why the recall sample is usually smaller and more heavily stratified.

Is an F1 score useful here?

As a single summary for tracking over time, yes; as a basis for decisions, rarely. It weights precision and recall equally, and conflation almost never does — an import wants precision heavily favoured, an enrichment often the reverse. Report it if it helps somebody see a trend, but make the decision on whichever of the two numbers actually matters for the destination.

How often should the sample be re-scored?

On every change to the matcher, which with stored labels costs minutes. The value of a labelled sample is entirely in reuse: labelling once and scoring once tells you where you were, while labelling once and scoring on every change tells you whether each change helped. Add the per-stratum regression check to the same run and quality drift becomes a failed check rather than a complaint months later.

Up one level: Conflation QA & Rollback.