Fuzzy Name Matching for OSM POI Conflation

Compare “St. Mary’s C of E Primary School” with “SAINT MARYS CHURCH OF ENGLAND PRIMARY” and get a useful number — without also deciding that “Bar Vega” and “Bar Vela” are the same place.

Prerequisites

Conceptual minimum

Two different kinds of similarity measure exist and they fail on opposite things.

Character-edit measures — Levenshtein and its normalised forms — count the operations needed to turn one string into another. They handle typos and spelling variants well and word reordering badly: “Vega Bar” versus “Bar Vega” is a large edit distance despite being obviously the same name.

Token-set measures split both strings into words and compare the sets. They handle reordering, extra words and missing words well, and typos badly: a single wrong letter turns a matching token into a non-matching one and the whole word is lost.

Real place names need both, and blending them into one number discards the information that distinguishes the failure modes. Keeping them separate lets a pair with a high token score and a low character score — a reordered name — be treated differently from one with the reverse profile.

The third component is the normalisation that runs before either, and its guiding rule is that it must be conservative and reversible: it produces a comparison string, never replaces the original.

Three name comparison strategies and what each one gets wrong Three panels. A character-edit measure counts insertions, deletions and substitutions, handling typos and spelling variants well but scoring reordered word sequences very low. A token-set measure compares the sets of words, handling reordering and extra or missing words well but losing a whole word to a single typo. Using both and keeping the scores separate covers each other's weaknesses and lets a reviewer see which kind of difference a pair exhibits. Two measures, two opposite blind spots Character edit Counts edits between strings Good: typos, spellings Bad: reordered words Bar Vega vs Vega Bar fails Normalised to 0 to 1 Token set Compares sets of words Good: reordering, extra words Bad: a single typo One wrong letter loses a word Ignores word order entirely Both, kept apart Compute both, store both Each covers the other Profile shows the difference High token, low char: reorder Low token, high char: typo Blending the two into one number throws away exactly the information a reviewer needs to judge an uncertain pair.
Two numbers cost nothing extra to compute and turn an opaque score into a diagnosis.

Runnable solution

python 
from __future__ import annotations

import logging
import re
import unicodedata
from dataclasses import dataclass

from rapidfuzz import fuzz

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.conflate.names")

# Generic words safe to drop: they carry no distinguishing content on their own.
# NOTE: words like "church", "school" or "hotel" are NOT here — for many places
# they are the whole distinguishing part of the name.
DROPPABLE = {
    "ltd", "limited", "plc", "inc", "llc", "gmbh", "sa", "sp", "zoo",
    "the", "and",
}
EXPANSIONS = {
    "st": "saint", "ste": "sainte", "mt": "mount", "ft": "fort",
    "rd": "road", "ave": "avenue", "sq": "square",
}
_PUNCT = re.compile(r"[^\w\s]", re.UNICODE)
_WS = re.compile(r"\s+")


@dataclass(frozen=True)
class NameScore:
    token: float          # 0..1, set-based; robust to reordering
    char: float           # 0..1, edit-based; robust to typos
    comparable: bool      # False when scripts differ or a side is empty

    @property
    def best(self) -> float:
        return max(self.token, self.char) if self.comparable else 0.0

    @property
    def profile(self) -> str:
        if not self.comparable:
            return "incomparable"
        if self.token - self.char > 0.25:
            return "reordered"
        if self.char - self.token > 0.25:
            return "typo-like"
        return "consistent"


def dominant_script(text: str) -> str:
    """Crude script detection: enough to avoid comparing across alphabets."""
    for ch in text:
        if ch.isalpha():
            name = unicodedata.name(ch, "")
            for script in ("LATIN", "CYRILLIC", "GREEK", "ARABIC", "HEBREW",
                           "HANGUL", "HIRAGANA", "KATAKANA", "CJK"):
                if script in name:
                    return script
    return "UNKNOWN"


def normalise(text: str) -> str:
    """Produce a comparison string. The caller keeps the original."""
    # NFKC folds compatibility forms; casefold is stronger than lower().
    text = unicodedata.normalize("NFKC", text).casefold()
    text = _PUNCT.sub(" ", text)
    tokens = [EXPANSIONS.get(t, t) for t in _WS.split(text) if t]
    kept = [t for t in tokens if t not in DROPPABLE]
    # If dropping emptied the name, keep the original tokens: a place genuinely
    # called "The Limited" must not normalise to nothing.
    return " ".join(kept or tokens)


def compare(left: str, right: str) -> NameScore:
    if not left or not right:
        return NameScore(0.0, 0.0, comparable=False)
    if dominant_script(left) != dominant_script(right):
        # Comparing across scripts produces meaningless numbers; say so instead.
        return NameScore(0.0, 0.0, comparable=False)

    a, b = normalise(left), normalise(right)
    if not a or not b:
        return NameScore(0.0, 0.0, comparable=False)
    return NameScore(
        token=fuzz.token_set_ratio(a, b) / 100.0,
        char=fuzz.ratio(a, b) / 100.0,
        comparable=True,
    )


def osm_names(tags: dict[str, str]) -> list[str]:
    """Every name an OSM feature carries, not just the default one."""
    keys = [k for k in tags if k == "name" or k.startswith("name:")
            or k in {"alt_name", "official_name", "short_name", "old_name"}]
    values: list[str] = []
    for key in keys:
        # Semicolon-separated multi-values are common in alt_name.
        values.extend(v.strip() for v in tags[key].split(";") if v.strip())
    return values


def best_against_osm(external: str, tags: dict[str, str]) -> NameScore:
    """Score an external name against every name the feature carries."""
    scores = [compare(external, name) for name in osm_names(tags)]
    comparable = [s for s in scores if s.comparable]
    if not comparable:
        return NameScore(0.0, 0.0, comparable=False)
    return max(comparable, key=lambda s: s.best)


if __name__ == "__main__":
    tags = {"name": "St. Mary's C of E Primary School",
            "alt_name": "Saint Marys Primary"}
    score = best_against_osm("SAINT MARYS CHURCH OF ENGLAND PRIMARY", tags)
    logger.info("token=%.2f char=%.2f profile=%s",
                score.token, score.char, score.profile)

Step-by-step walkthrough

  1. Keep the droppable list short and safe. Legal suffixes and a couple of articles only. Words like “church”, “school” and “hotel” look generic and are frequently the entire distinguishing content of a name.
  2. Expand rather than strip abbreviations. Turning “St” into “saint” makes it match “Saint”; deleting it makes both names shorter and less distinctive.
  3. Guard against emptying a name. A place called “The Limited” would normalise to nothing under a naive filter; falling back to the unfiltered tokens prevents that.
  4. Refuse cross-script comparisons. A Latin and a Cyrillic name produce a meaningless similarity number that is nevertheless a number, and it will be used. Returning an explicit incomparable result is far better than returning zero, which looks like evidence of a mismatch.
  5. Score against every name the feature carries. OSM features frequently carry alt_name, official_name, short_name and language variants, and the external dataset may use any of them. Comparing only against name throws away the easiest matches.
  6. Split semicolon multi-values. alt_name in particular often holds several names separated by semicolons, as covered in Splitting Semicolon-Separated OSM Tag Values.
  7. Return a profile, not just a number. Naming the shape of the difference — reordered, typo-like, consistent — is what makes an uncertain pair reviewable in seconds rather than minutes.
How each normalisation step affects a worked example name pair A grid showing one external name and one OSM name progressing through four normalisation steps, with the token and character similarity after each. Before normalisation the two strings share little and both scores are low. After Unicode folding and case folding the character score rises modestly. After punctuation removal the token score rises sharply because apostrophes and full stops stopped splitting words. After abbreviation expansion both scores rise again because the abbreviated saint and church of England now match their expanded forms. Where each normalisation step actually earns its place Token score Char score Raw strings 0.41 0.38 Unicode and case folded 0.44 0.52 Punctuation removed 0.71 0.58 Abbreviations expanded 0.94 0.79 Punctuation removal and abbreviation expansion do nearly all the work; suffix dropping contributes little and carries real risk.
Measuring each step on your own labelled sample is what stops normalisation from accumulating rules nobody can justify.
What the comparison profile tells a reviewer about an uncertain pair Four profiles and the judgement each suggests. A consistent profile, where both scores agree and are high, needs no review. A reordered profile, where the token score is much higher than the character score, almost always indicates the same name written in a different word order and is usually a match. A typo-like profile, where the character score is much higher, indicates a spelling variant and is usually a match. An incomparable profile means no comparison was possible and the pair must be judged on other signals entirely. Four profiles, four different review actions consistent both scores agree no review needed reordered token beats char usually a match typo-like char beats token usually a match incomparable no comparison made judge on other signals The last profile is the important one: it must not be treated as a low score, because no evidence was gathered either way.
Naming the profile is what turns a reviewer's minute of squinting at two strings into a two-second decision.

Verification

  • Known pairs score high. Take twenty labelled true pairs and confirm the best score exceeds your intended threshold on nearly all of them.
  • Known non-pairs score low. Twenty labelled false pairs, especially nearby similar businesses, should score well below it.
  • Reordered names are caught. “Bar Vega” against “Vega Bar” should score high on token similarity and carry the reordered profile.
  • Cross-script pairs are incomparable. A Latin name against a Cyrillic one must return the incomparable result, not zero.
  • Alternative names are used. A feature whose alt_name matches but whose name does not should still produce a high score.

Common errors and fixes

Symptom Root cause One-line fix
Distinct places score identically Distinguishing word in the drop list Remove generic-looking but meaningful words from it
A name normalises to nothing Every token was droppable Fall back to the unfiltered tokens
Reordered names score low Only a character measure used Add a token-set measure and keep both
Typos score low Only a token measure used Add a character measure and keep both
Cross-script pairs look like mismatches Zero returned instead of incomparable Detect the script and mark the pair incomparable
Easy matches missed Only the default name compared Compare against every name key the feature carries
Accented names never match No Unicode normalisation Apply compatibility normalisation before comparing

Specification reference

The token set ratio compares two strings by splitting them into token sets and measuring similarity over the intersection and the differences, which makes it insensitive to word order and to extra tokens on either side. The simple ratio is based on the longest matching subsequence and is sensitive to character-level differences. See the RapidFuzz documentation for the exact definitions of each scorer and their normalisation to a zero-to-one hundred range.

Frequently Asked Questions

Should I strip generic words like "church" or "school"?

No. They look generic and for many places they are the entire distinguishing content of the name — a village with a church, a school and a pub may have three features whose names differ only in that word. Restrict the drop list to legal suffixes and a couple of articles, and expand abbreviations rather than deleting them. The risk of dropping a meaningful word is much larger than the gain from a slightly shorter comparison string.

Why keep two similarity scores instead of one combined number?

Because they fail on opposite things and the difference between them is diagnostic. A high token score with a low character score means the words match but were reordered; the reverse means the order is right but the spelling differs. A reviewer can judge those two cases in seconds, while a single blended number tells them only that something was partially similar.

How should I handle names in different scripts?

Mark the pair incomparable rather than scoring it. A similarity measure over two different alphabets returns a small number, and a small number looks like evidence that the names differ when in fact no comparison was possible. If your data genuinely spans scripts, transliterate deliberately into a common form as a separate, reviewable step — and keep both the original and the transliteration.

Which OSM name tags should I compare against?

All of them. A feature commonly carries a default name, one or more language variants, an official name, a short name and one or more alternative names, and an external dataset may use any of those. Comparing only the default name discards the easiest matches available. Split semicolon-separated multi-values as well, because alternative name tags frequently hold several names in one string.

Up one level: Matching OSM Features to External Datasets.