Linking OSM Features to Wikidata Identifiers

A shared identifier turns conflation from a scoring problem into a lookup. Establishing one correctly is worth considerable care, because a wrong link is both more damaging and far harder to notice than a wrong fuzzy match.

Prerequisites

Conceptual minimum

A knowledge-base identifier is an assertion that an OSM feature and an external entity are the same thing in the world. That is a stronger claim than “these two records look similar”, and it earns its strength by being verifiable in a way a fuzzy score is not.

Two properties make verification possible. The entity carries a coordinate, so a link can be checked against the OSM feature’s position. And it carries a type — an “instance of” statement — so a link can be checked for category agreement: an OSM railway station linked to an entity that is a human is wrong regardless of how well the names match.

The third property is the one that makes the whole exercise worthwhile: links are reciprocal in practice. The knowledge base frequently records the OSM relation or way identifier for administrative entities, and OSM records the entity identifier. Where both directions exist and agree, the link is about as certain as anything in this section gets.

Three checks that verify an identifier link, and what each one catches Three panels. The coordinate check compares the entity's recorded position against the OSM feature's, catching links to a same-named place somewhere else entirely. The type check compares the entity's instance-of statement against the OSM feature's category, catching links to a person, a book or a film that shares a name with a place. The reciprocity check looks for the entity recording the OSM identifier back, which when present makes the link close to certain and when contradictory is a strong signal that one side is wrong. Three checks, three different wrong links Coordinate Entity position versus OSM Catches the same name elsewhere Tolerance by feature size A city needs kilometres Type Instance-of versus category Catches a person or a film Needs a type mapping Cheap and decisive Reciprocity Does it link back to OSM? Present and agreeing: certain Present and differing: alarm Absent: neutral, not bad Absent reciprocity is not evidence against a link; most features have no reason for the knowledge base to record them.
Each check catches a different wrong link, and the type check is both the cheapest and the most decisive.

Runnable solution

python 
from __future__ import annotations

import logging
import math
from dataclasses import dataclass

import requests

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.enrich.wikidata")

ENDPOINT = "https://query.wikidata.org/sparql"
HEADERS = {
    "User-Agent": "osm-pipeline-example/1.0 (contact@example.org)",
    "Accept": "application/sparql-results+json",
}
EARTH_RADIUS_M = 6_371_008.8

# OSM category -> acceptable entity types. Deliberately broad: the check is
# meant to catch a person or a film, not to police fine-grained taxonomy.
TYPE_EXPECTATIONS: dict[str, set[str]] = {
    "railway_station": {"Q55488", "Q55678", "Q4167836"},
    "museum": {"Q33506", "Q207694"},
    "school": {"Q3914", "Q9842"},
    "church": {"Q16970", "Q1370598"},
}


QUERY_TEMPLATE = """
SELECT ?item ?coord ?type ?osmRel WHERE {
  VALUES ?item { __VALUES__ }
  OPTIONAL { ?item wdt:P625 ?coord. }
  OPTIONAL { ?item wdt:P31 ?type. }
  OPTIONAL { ?item wdt:P402 ?osmRel. }
}
"""


@dataclass(frozen=True)
class Verdict:
    qid: str
    ok: bool
    distance_m: float | None
    type_ok: bool | None
    reciprocal: bool
    reason: str


def haversine(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
    p1, p2 = math.radians(lat1), math.radians(lat2)
    dp = p2 - p1
    dl = math.radians(lon2 - lon1)
    a = math.sin(dp / 2) ** 2 + math.cos(p1) * math.cos(p2) * math.sin(dl / 2) ** 2
    return 2 * EARTH_RADIUS_M * math.asin(math.sqrt(a))


def fetch_entities(qids: list[str]) -> dict[str, dict]:
    """One query for many entities: never one request per feature."""
    values = " ".join(f"wd:{q}" for q in qids)
    # Built by substitution rather than an f-string: a SPARQL query is full of
    # braces, and escaping every one of them makes the query unreadable.
    query = QUERY_TEMPLATE.replace("__VALUES__", values)
    response = requests.get(ENDPOINT, params={"query": query},
                            headers=HEADERS, timeout=120)
    response.raise_for_status()

    out: dict[str, dict] = {}
    for row in response.json()["results"]["bindings"]:
        qid = row["item"]["value"].rsplit("/", 1)[-1]
        entry = out.setdefault(qid, {"types": set(), "coord": None,
                                     "osm_relation": None})
        if "type" in row:
            entry["types"].add(row["type"]["value"].rsplit("/", 1)[-1])
        if "coord" in row and entry["coord"] is None:
            # Point(lon lat) — longitude first, as in every WKT.
            lon, lat = row["coord"]["value"].strip("Point()").split()
            entry["coord"] = (float(lat), float(lon))
        if "osmRel" in row:
            entry["osm_relation"] = row["osmRel"]["value"]
    return out


def verify(qid: str, entity: dict, osm_lat: float, osm_lon: float,
           osm_category: str, osm_relation_id: str | None,
           tolerance_m: float) -> Verdict:
    distance = None
    if entity.get("coord"):
        distance = haversine(osm_lat, osm_lon, *entity["coord"])
        if distance > tolerance_m:
            return Verdict(qid, False, distance, None, False,
                           f"entity is {distance/1000:.1f} km away")

    type_ok = None
    expected = TYPE_EXPECTATIONS.get(osm_category)
    if expected is not None and entity["types"]:
        type_ok = bool(entity["types"] & expected)
        if not type_ok:
            return Verdict(qid, False, distance, False, False,
                           f"entity types {sorted(entity['types'])} do not match "
                           f"{osm_category}")

    reciprocal = (entity.get("osm_relation") is not None
                  and osm_relation_id is not None
                  and str(entity["osm_relation"]) == str(osm_relation_id))
    if entity.get("osm_relation") and osm_relation_id and not reciprocal:
        return Verdict(qid, False, distance, type_ok, False,
                       "entity links to a different OSM relation")

    return Verdict(qid, True, distance, type_ok, reciprocal, "verified")


if __name__ == "__main__":
    entities = fetch_entities(["Q42", "Q90"])
    logger.info("fetched %d entity record(s)", len(entities))

Step-by-step walkthrough

  1. Batch the lookups. One query covering many identifiers rather than one request each; the endpoint is a shared resource and per-request overhead dominates otherwise.
  2. Handle the coordinate literal correctly. The point literal is longitude first, which is the opposite order from how latitude and longitude are usually spoken and the source of a familiar class of bug.
  3. Scale the tolerance to the feature. A pub’s entity coordinate should be within tens of metres; a city’s may legitimately be kilometres from any particular OSM node, because the entity describes the whole settlement.
  4. Treat a missing coordinate as neutral. Many entities have none, and their absence is not evidence against a link — only a present and distant coordinate is.
  5. Keep type expectations broad. The check is meant to catch a link to a person, a film or a book, not to enforce a fine-grained ontology. A broad allowed set keeps false alarms low while still catching the failures that matter.
  6. Treat a contradictory reciprocal link as decisive. If the entity records a different OSM object, one of the two links is wrong and the pair needs a human.
  7. Return a reason, not a boolean. “Entity is 340 km away” is actionable; False is not.
What a verified identifier link is worth to every later enrichment Four stages showing the compounding value. Establishing the link once costs a verified match, which is the expensive part. Every later enrichment from the same knowledge base becomes a join on the identifier rather than a fresh matching run. Enrichment from any other source that also carries the identifier becomes a join too, without any matching at all. And because the link survives edits that change geometry or names, it decays far more slowly than a stored fuzzy match. Pay once, join forever establish verified match the expensive part same source join, not match every later attribute other sources any that carry it no matching at all survives edits geometry and names change the link does not This is why establishing identifier links deliberately pays for itself: every subsequent enrichment stops being a conflation problem.
The last step is the quiet benefit — an identifier link outlives the geometry and naming changes that break fuzzy matches.
How coordinate tolerance should scale with the kind of feature being linked A grid of four feature scales against the coordinate tolerance each needs and the reason. A building or point of interest needs tens of metres, because the entity coordinate should name the same structure. A campus or a site needs a few hundred metres, because the entity coordinate may sit anywhere within the grounds. A settlement needs several kilometres, because the entity names the whole place and its coordinate is a chosen centre. An administrative region needs tens of kilometres for the same reason at a larger scale. One global tolerance cannot serve all four Tolerance Why Building or POI tens of metres same structure Campus or site a few hundred metres anywhere in the grounds Settlement several kilometres a chosen centre Administrative region tens of kilometres same, larger Deriving the tolerance from the feature's own bounding box is more robust than a lookup table and needs no maintenance.
A single tolerance chosen for buildings rejects every city link; one chosen for cities accepts the pub in the next town.

Verification

  • Distant links are rejected. Construct a link to an entity in another country and confirm the verdict rejects it with the distance in the reason.
  • Type mismatches are rejected. Link a station to a person entity and confirm the type check fires.
  • Missing data is neutral. An entity with no coordinate and no type should verify rather than fail.
  • Reciprocal disagreement is caught. Where the entity names a different OSM object, the verdict must be a rejection.
  • Batching actually batches. Count requests for a hundred identifiers; it should be one or two, not a hundred.

Common errors and fixes

Symptom Root cause One-line fix
Every link rejected as distant Coordinate literal parsed latitude-first The point literal is longitude first
Large features always rejected One tolerance for every feature type Scale the tolerance to the feature’s extent
Links with no coordinate rejected Missing data treated as failure Treat absent evidence as neutral
Type check rejects valid links Expectation set too narrow Broaden the allowed types; catch absurdities only
Endpoint throttles the pipeline One request per feature Batch many identifiers into one query
Wrong links go unnoticed Only a boolean returned Return a reason naming the failing check
Verified links decay quietly No re-verification schedule Re-verify periodically as with any stored match

Specification reference

Wikidata entities carry a coordinate location property, an instance-of property giving the entity’s type, and — for many administrative and geographic entities — an OpenStreetMap relation identifier property. The query service accepts SPARQL over HTTP and returns JSON, subject to a usage policy that expects an identifying user agent and discourages high request rates. See the Wikidata query service documentation and the OSM wikidata tag documentation for the tagging conventions.

Frequently Asked Questions

Why verify a link that already exists in the data?

Because wrong identifier links are both more damaging and harder to spot than wrong fuzzy matches. A link is an assertion of identity, so anything joined through it inherits that assertion without further checking — which is exactly why it is worth establishing carefully. Existing links in OSM were added by people of varying care, and a coordinate and type check over them is cheap and finds real errors.

What tolerance should the coordinate check use?

One scaled to the feature. An entity describing a pub should sit within tens of metres of the OSM node; an entity describing a city legitimately sits at whatever point somebody chose as its centre, which can be kilometres from any particular feature within it. A single global tolerance either rejects every large feature or accepts links to same-named places in the next county.

Is a missing coordinate or type a reason to reject?

No. Many entities have neither, particularly for smaller or less documented subjects, and treating absence as failure rejects perfectly good links for the crime of being about something obscure. Only present-and-contradictory evidence should reject: a coordinate far away, a type that is absurd for the feature, or a reciprocal link naming a different object.

How much is an identifier link actually worth?

A great deal, because it converts every later enrichment from a matching problem into a join. Once the link exists and is verified, any attribute from that knowledge base — and from any other dataset that also carries the identifier — attaches without scoring, thresholds or review. It also survives the geometry and naming changes that break stored fuzzy matches, so it decays far more slowly.

Up one level: Attribute Enrichment from Authoritative Sources.