Scoring Conflation Candidates with Multiple Signals

Turn four separate pieces of evidence about a candidate pair into a decision a reviewer can argue with — which means keeping the evidence, not replacing it with a number.

Prerequisites

Conceptual minimum

Four signals carry most conflation evidence, and their usefulness depends on being independent and on missingness being handled honestly.

A signal is missing when the evidence simply is not there: an OSM feature with no name, an external record with no category, a pair where the two names are in different scripts. The wrong response is to score it zero, because zero means “strong evidence against” and missing means “no evidence either way”. The right response is to renormalise over the signals that are present, so a pair evidenced by distance and category alone is judged on those two rather than penalised for lacking a name.

The second idea is that the best score alone is not a decision. A pair scoring 0.85 with the next candidate at 0.30 is a different situation from one scoring 0.85 with the next at 0.83, and only the first is confident. The gap is a second number, and using both is what separates a matcher that works in dense areas from one that does not.

The four signals, what they evidence and how they fail A grid of four signals against what each one actually evidences, its typical weight, and its characteristic failure. Distance evidences plausibility, carries modest weight, and fails by being uninformative in dense areas where something is always nearby. Name similarity evidences identity, carries the largest weight where names exist, and fails when either side has no name. Category agreement evidences kind, carries moderate weight, and fails when no shared vocabulary exists. A shared identifier evidences identity almost conclusively, dominates when present, and fails only by being absent. Four signals, four different things they prove Evidences Weight Fails when Distance plausibility modest area is dense Name similarity identity largest a name is absent Category kind of thing moderate no shared vocabulary Shared identifier identity dominant simply absent Only the bottom signal is close to conclusive on its own, which is why establishing identifier links pays for itself repeatedly.
Reading the middle column as fixed weights is the mistake: they should be calibrated per source against a labelled sample.

Runnable solution

python 
from __future__ import annotations

import logging
import math
from dataclasses import dataclass, field
from enum import Enum

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.conflate.score")


class Outcome(str, Enum):
    MATCH = "match"
    REVIEW = "review"
    NO_MATCH = "no_match"


@dataclass(frozen=True)
class Signals:
    """Each field is a score in 0..1, or None when there is NO evidence.

    None is not zero: zero means 'the evidence argues against', None means
    'no evidence either way'. Conflating the two penalises features for
    lacking a tag, which is not the same as contradicting one.
    """
    distance: float | None = None
    name: float | None = None
    category: float | None = None
    identifier: float | None = None


WEIGHTS: dict[str, float] = {
    "distance": 0.20, "name": 0.45, "category": 0.20, "identifier": 0.15,
}
MATCH_FLOOR = 0.78
REVIEW_FLOOR = 0.45
MIN_GAP = 0.15
MIN_EVIDENCE_WEIGHT = 0.5      # refuse to decide on too little evidence


@dataclass(frozen=True)
class Scored:
    total: float
    present_weight: float
    signals: Signals
    components: dict[str, float] = field(default_factory=dict)


def distance_score(metres: float, radius_m: float) -> float:
    """1 at zero separation, decaying smoothly to 0 at the candidate radius."""
    if metres >= radius_m:
        return 0.0
    # A cosine taper is gentler near zero than a linear ramp, which matters
    # because positional error is concentrated near the true position.
    return 0.5 * (1.0 + math.cos(math.pi * metres / radius_m))


def combine(signals: Signals) -> Scored:
    """Weighted mean over the signals that are PRESENT, renormalised."""
    components: dict[str, float] = {}
    total = 0.0
    present = 0.0
    for name, weight in WEIGHTS.items():
        value = getattr(signals, name)
        if value is None:
            continue
        components[name] = value
        total += weight * value
        present += weight
    if present == 0.0:
        return Scored(0.0, 0.0, signals, components)
    return Scored(total / present, present, signals, components)


def classify(ranked: list[Scored]) -> tuple[Outcome, str]:
    """Decide from the best score, the runner-up gap and the evidence weight."""
    if not ranked:
        return Outcome.NO_MATCH, "no candidates"

    best = ranked[0]
    runner_up = ranked[1].total if len(ranked) > 1 else 0.0
    gap = best.total - runner_up

    if best.identifier_match():
        return Outcome.MATCH, "shared identifier"
    if best.present_weight < MIN_EVIDENCE_WEIGHT:
        return Outcome.REVIEW, f"only {best.present_weight:.2f} of evidence present"
    if best.total < REVIEW_FLOOR:
        return Outcome.NO_MATCH, f"best score {best.total:.2f} below the floor"
    if best.total >= MATCH_FLOOR and gap >= MIN_GAP:
        return Outcome.MATCH, f"score {best.total:.2f}, gap {gap:.2f}"
    if best.total >= MATCH_FLOOR:
        return Outcome.REVIEW, f"score {best.total:.2f} but gap only {gap:.2f}"
    return Outcome.REVIEW, f"score {best.total:.2f} in the middle band"


def _identifier_match(self: Scored) -> bool:
    return self.signals.identifier is not None and self.signals.identifier >= 0.99


Scored.identifier_match = _identifier_match     # small helper, kept out of the data


def score_record(candidates: list[Signals]) -> tuple[Outcome, str, list[Scored]]:
    ranked = sorted((combine(s) for s in candidates),
                    key=lambda s: (-s.total, -s.present_weight))
    outcome, reason = classify(ranked)
    logger.info("%s: %s (components %s)", outcome.value, reason,
                ranked[0].components if ranked else {})
    return outcome, reason, ranked


if __name__ == "__main__":
    near = Signals(distance=distance_score(18.0, 120.0), name=0.91,
                   category=1.0)
    far = Signals(distance=distance_score(95.0, 120.0), name=0.44,
                  category=0.0)
    score_record([near, far])

Step-by-step walkthrough

  1. Model absent evidence as None. The distinction between “no name on this feature” and “the names disagree completely” is the single most important modelling decision in the scorer, and collapsing it is why matchers under-match sparsely tagged features.
  2. Renormalise over present signals. Dividing by the weight actually available means a pair with two of four signals is judged on those two at full strength rather than scoring at most half.
  3. Track how much evidence was present. A high score computed from one weak signal is not the same as the same score from three; the evidence weight makes that visible and drives an explicit abstain.
  4. Taper distance smoothly. A cosine taper is close to flat near zero, which reflects that positional error clusters near the true position, and falls to zero exactly at the candidate radius so no candidate outside it can contribute.
  5. Short-circuit on a shared identifier. A matching reference code or Wikidata link is near-conclusive and should not be diluted by a mediocre name score.
  6. Abstain on thin evidence. Below a minimum evidence weight the scorer routes to review regardless of the number, because the number is not meaningful.
  7. Require a gap as well as a score. A high score with a close runner-up is ambiguity, not confidence, and it is exactly the dense-area case that ruins precision.
  8. Return the components and a reason. The reason string is what a reviewer reads first, and the components are what they check when the reason surprises them.
The order the classifier asks its questions in A decision node listing four checks applied in order, with the outcome each produces. A shared identifier match short-circuits immediately to a confident match. Insufficient present evidence routes to review regardless of the computed score, because the score is not meaningful. A best score below the review floor gives a no-match. A score above the match floor gives a confident match only when the gap to the runner-up is also large enough; otherwise it too goes to review. Four checks, applied in this order What does the evidence support? Order matters here Identifier wins outright Identifier match Short-circuits everything else; near-conclusive on its own Too little evidence Review regardless of score; the number is not meaningful Below the floor No match; a legitimate and common outcome High score, small gap Review; ambiguity is not confidence Reversing the first two checks would send identifier matches on sparsely tagged features to review for no reason.
Every branch here ends somewhere explicit, which is what stops uncertainty from defaulting quietly into a match.
How renormalising over present signals changes the score for partially evidenced pairs Five candidate pairs with the same underlying quality but different amounts of evidence available, scored two ways. A pair with all four signals scores the same under both approaches. A pair missing its category scores well under renormalisation and noticeably lower when the missing signal is treated as zero. A pair with only distance and name drops further under the zero treatment. A pair with only a name score collapses under the zero treatment while renormalisation still reports a high but low-confidence score. A note says the second approach is why sparsely tagged regions under-match. Same pair, two ways of handling missing evidence All four signals 0.88 either way No category: renormalised 0.86 No category: zeroed 0.69 Name only: renormalised 0.91, low evidence Name only: zeroed 0.41 The zeroed variant rejects a perfectly good name match because the feature happened to carry no category tag.
Renormalising keeps the score honest; the evidence weight, carried alongside, is what stops it becoming overconfident.

Verification

  • Missing signals do not penalise. Score a pair with a perfect name and no category; it should score close to a pair with a perfect name and a matching category, not half of it.
  • The taper reaches zero at the radius. A candidate at exactly the radius must contribute nothing.
  • Thin evidence abstains. A pair evidenced only by distance should route to review however close it is.
  • The gap check fires. Construct two candidates scoring 0.86 and 0.84; the classifier must say review, not match.
  • Calibration holds on the sample. Run the labelled sample through and confirm precision and recall match what the thresholds intend, per Measuring Conflation Precision and Recall.

Common errors and fixes

Symptom Root cause One-line fix
Unnamed features never match Missing name scored as zero Model absent evidence as none and renormalise
Scores cluster near the weights’ midpoint Missing signals dragging every total down Divide by the present weight, not the total weight
Wrong matches in dense areas Only the best score consulted Require a minimum gap to the runner-up
A single signal decides everything Weights never calibrated Fit weights against a labelled sample per source
Confident matches on one weak signal No evidence-weight floor Abstain below a minimum present weight
Identifier matches sent to review Checks applied in the wrong order Short-circuit on a shared identifier first
Reviewers cannot tell why Only the total returned Return the components and a human-readable reason

Specification reference

Weighted scoring over partially observed evidence is conventionally handled by renormalising the weights over the observed subset, so that an unobserved signal neither contributes nor penalises. This differs from imputing a neutral or zero value, which biases the result towards the imputed value. The distinction matters in OpenStreetMap conflation because tag presence varies enormously between regions and feature classes; see Tag Taxonomy & Key-Value Standards for why absence is so common.

Frequently Asked Questions

Why is a missing signal not the same as a zero score?

Because zero means the evidence argues against a match and missing means there is no evidence either way. An OSM feature with no name tag has not disagreed with your external record’s name; it has simply said nothing. Scoring that as zero penalises sparsely tagged features — which are disproportionately the ones in areas where mapping is thinner — and systematically under-matches exactly where you most need the match.

How should I choose the weights?

Fit them against a labelled sample for each source rather than adopting a set from elsewhere. The relative value of name similarity and category agreement depends entirely on whether the external dataset has good names and whether its categories map onto OSM tagging at all. A few hundred labelled pairs is enough to see which signals separate true from false pairs, and that separation is what the weights should reflect.

Why require a gap to the runner-up?

Because a high score with a near-identical alternative means the matcher is choosing between two equally plausible answers on no real basis. That situation is common in dense areas — a row of similar shops, a campus of similar buildings — and it is precisely where an absolute threshold alone produces confident wrong answers. The gap turns those cases into review items instead.

Should the scorer ever abstain?

Yes, and explicitly. When too little evidence is present the computed number is not meaningful however high it is, and pretending otherwise is how a matcher produces confident matches from a single weak signal. An explicit abstain, routed to review with a reason naming the missing evidence, is both more honest and more useful than a number nobody can interpret.

Up one level: Matching OSM Features to External Datasets.