Deduplicating Addresses Before an OSM Import

Before uploading an address dataset, find out what OpenStreetMap already has — which is harder than it sounds, because the same address can be mapped in three quite different ways and a naive check finds only one of them.

Prerequisites

Conceptual minimum

OpenStreetMap represents an address in at least three ways, and a deduplication check that looks for only one will report almost everything as new.

An address node is a point carrying addr:housenumber and addr:street. It is the simplest form and the one everybody checks.

An addressed building is a way or relation carrying the same tags on the building itself. There is no separate point, so a check that only examines nodes finds nothing — and then the import adds an address node inside a building that already has that address.

An interpolation way is a line between two address nodes carrying addr:interpolation, asserting that the numbers between them exist along it. The individual addresses are not mapped as objects at all; they are implied. A check that ignores interpolation will duplicate whole streets.

The three ways OSM represents an address and what each one requires of a check Three panels. An address node is a point with housenumber and street tags, found by a simple point search, and is the form everybody checks for. An addressed building is a way or relation carrying the same tags with no separate point, so a node-only check misses it entirely and the import adds a duplicate point inside the building. An interpolation way is a line carrying a range assertion where individual addresses are implied rather than mapped, so a check must evaluate the range rather than look for objects. Three representations, three different checks Address node A point with addr tags Found by a point search The form everybody checks Easy case Addressed building Tags on the way itself No separate point exists Node-only check finds nothing Import adds a duplicate Interpolation way A line with a range Addresses are implied No objects to find Must evaluate the range A deduplication check covering only the first panel reports nearly every record as new in areas mapped the other two ways.
The third form is the one that duplicates entire streets when it is overlooked.

The second complication is that housenumbers are messy. “12”, “12A”, “12-14”, “12/3” and “12 a” may all denote the same or different things depending on the country, and a string comparison over raw values is unreliable in both directions.

Runnable solution

python 
from __future__ import annotations

import logging
import re
import unicodedata
from dataclasses import dataclass

import geopandas as gpd
import pandas as pd
from shapely.geometry import Point

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.import.dedupe")

MATCH_RADIUS_M = 40.0
_WS = re.compile(r"\s+")
_NUM = re.compile(r"^(\d+)\s*([A-Za-z]?)$")


def norm_street(name: str) -> str:
    text = unicodedata.normalize("NFKC", name or "").casefold()
    return _WS.sub(" ", re.sub(r"[^\w\s]", " ", text)).strip()


def norm_number(value: str) -> str:
    """Canonical housenumber: digits plus an optional single letter suffix."""
    text = unicodedata.normalize("NFKC", value or "").casefold()
    text = _WS.sub("", text.replace("–", "-"))
    match = _NUM.match(text)
    return f"{int(match.group(1))}{match.group(2)}" if match else text


def expand_interpolation(row) -> list[str]:
    """Housenumbers a single interpolation way asserts, as canonical strings."""
    kind = (row.get("addr:interpolation") or "").lower()
    try:
        start, end = int(row["_from"]), int(row["_to"])
    except (TypeError, ValueError, KeyError):
        return []
    if start > end:
        start, end = end, start
    step = {"odd": 2, "even": 2, "all": 1}.get(kind)
    if step is None:
        # A numeric interpolation value is a literal step, e.g. "4".
        step = int(kind) if kind.isdigit() else 1
    if kind == "odd" and start % 2 == 0:
        start += 1
    if kind == "even" and start % 2 == 1:
        start += 1
    return [str(n) for n in range(start, end + 1, step)]


def existing_addresses(osm: gpd.GeoDataFrame,
                       interpolations: gpd.GeoDataFrame | None = None
                       ) -> gpd.GeoDataFrame:
    """Every address OSM already holds, in all three representations."""
    frames: list[gpd.GeoDataFrame] = []

    # 1 + 2: nodes AND ways/relations carrying address tags. Using the
    # representative point of a building is what makes the two comparable.
    addressed = osm[osm["addr:housenumber"].notna()
                    & osm["addr:street"].notna()].copy()
    addressed["geometry"] = addressed.geometry.representative_point()
    addressed["_form"] = addressed.geometry.geom_type.where(
        addressed["_osm_type"].eq("node"), "building")
    frames.append(addressed)

    # 3: interpolation ways, expanded into the numbers they assert.
    if interpolations is not None and len(interpolations):
        rows = []
        for _, way in interpolations.iterrows():
            for number in expand_interpolation(way):
                rows.append({
                    "addr:housenumber": number,
                    "addr:street": way.get("addr:street"),
                    "_form": "interpolation",
                    "geometry": way.geometry.interpolate(0.5, normalized=True),
                })
        if rows:
            frames.append(gpd.GeoDataFrame(rows, geometry="geometry",
                                           crs=interpolations.crs))

    out = pd.concat(frames, ignore_index=True)
    out["_key"] = (out["addr:street"].map(norm_street) + "|"
                   + out["addr:housenumber"].map(norm_number))
    logger.info("OSM already holds %d address(es): %s",
                len(out), out["_form"].value_counts().to_dict())
    return gpd.GeoDataFrame(out, geometry="geometry", crs=osm.crs)


def classify(candidates: gpd.GeoDataFrame, existing: gpd.GeoDataFrame,
             epsg: int) -> pd.DataFrame:
    """Split import candidates into new, duplicate and review."""
    left = candidates.to_crs(epsg=epsg).copy()
    right = existing.to_crs(epsg=epsg).copy()
    left["_key"] = (left["addr:street"].map(norm_street) + "|"
                    + left["addr:housenumber"].map(norm_number))

    probe = left.copy()
    probe["geometry"] = probe.geometry.buffer(MATCH_RADIUS_M)
    near = gpd.sjoin(probe[["_key", "geometry"]], right[["_key", "_form",
                                                         "geometry"]],
                     how="left", predicate="intersects",
                     lsuffix="new", rsuffix="osm")

    same = near["_key_new"].eq(near["_key_osm"])
    duplicate_keys = set(near.loc[same, "_key_new"])
    # A nearby address on the SAME street with a DIFFERENT number is not a
    # duplicate, but a nearby identical key beyond the radius might be.
    left["_status"] = [
        "duplicate" if k in duplicate_keys else "new" for k in left["_key"]
    ]

    far_dupes = set(left.loc[left["_status"] == "new", "_key"]) & set(right["_key"])
    left.loc[left["_key"].isin(far_dupes), "_status"] = "review"

    counts = left["_status"].value_counts().to_dict()
    logger.info("import candidates: %s", counts)
    return left[["_key", "_status", "addr:housenumber", "addr:street"]]


if __name__ == "__main__":
    logger.info("upload only rows whose status is 'new'")

Step-by-step walkthrough

  1. Collect all three representations. Address nodes, addressed buildings and expanded interpolation ways go into one frame. Omitting any of them makes the check report duplicates as new.
  2. Use a representative point for areas. A building’s representative point is guaranteed to lie inside it, unlike a centroid, which makes an addressed building directly comparable with an address node.
  3. Expand interpolation ranges honestly. Odd, even, all and numeric steps each behave differently, and the start value has to be adjusted to the right parity. A range that fails to parse contributes nothing rather than guessing.
  4. Canonicalise the key. Street name normalised, housenumber reduced to digits plus an optional letter, joined into one comparable key. "12 A" and "12a" must produce the same key.
  5. Join spatially, then compare keys. Proximity alone is not a duplicate — a different number on the same street is usually the house next door — so the spatial join finds candidates and the key comparison decides.
  6. Treat a distant identical key as review, not duplicate. The same street and number appearing far away may be a long street, a repeated name, or a positional error, and a human should look.
  7. Report the three counts. New, duplicate and review are the numbers the import plan promised, and they are the first thing a reviewer will ask for.
How the duplicate count changes as each representation is added to the check Four measurements over one municipal address dataset of fifty thousand records. Checking only address nodes finds a few thousand duplicates. Adding addressed buildings roughly triples that figure, because in this area most addresses live on the building rather than on a separate point. Adding interpolation ways adds several thousand more, concentrated on residential streets. The final duplicate count is close to half the dataset, where the node-only check had reported under a tenth. What each representation contributes to the duplicate count Address nodes only about 4,200 Plus addressed buildings about 14,800 Plus interpolation ways about 21,500 Records in the dataset 50,000 total The node-only check would have uploaded seventeen thousand duplicate addresses, which is the shape of a real import disaster.
Which representation dominates varies by region, so all three must be checked rather than the locally common one.
How one import candidate reaches its status Four steps. The canonicalise step reduces the street name and housenumber to a single comparable key on both sides. The locate step buffers the candidate and joins spatially against every existing address, collecting nearby entries regardless of their number. The compare step tests whether any nearby entry shares the canonical key, which is what distinguishes a duplicate from the house next door. The classify step marks the candidate as a duplicate when a nearby key matches, as review when the key matches only beyond the radius, and as new otherwise. Proximity finds; the key decides canonicalise street plus number one comparable key locate buffer and join nearby, any number compare does a key match? neighbours excluded classify new, duplicate, review distant match reviews Separating locate from compare is what stops the house next door being treated as the same address as its neighbour.
Both halves are necessary: proximity alone over-matches and key equality alone cannot see positional error.

Verification

  • All three forms are represented. The counts by form should be non-zero wherever the area uses them; an absent form usually means the filter missed it.
  • Interpolation expansion is plausible. A way from 2 to 20 marked even should yield ten numbers, not nineteen.
  • Key canonicalisation collapses variants. Confirm that differently formatted versions of one housenumber produce one key.
  • Proximity alone does not duplicate. A record with a different number on the same street, metres away, must be classified as new.
  • The review group is small and interesting. Sample it; the entries should be genuinely ambiguous rather than routine.

Common errors and fixes

Symptom Root cause One-line fix
Nearly everything reported as new Only address nodes checked Include addressed buildings and interpolation ways
Whole streets duplicated Interpolation ways ignored Expand interpolation ranges into implied numbers
Variants of one number treated separately Raw housenumber strings compared Canonicalise to digits plus an optional letter
Neighbouring houses marked duplicate Proximity used as the decision Compare canonical keys after the spatial join
Duplicates missed on long streets Radius smaller than the positional offset Route distant identical keys to review
Areas and points incomparable Centroid used for buildings Use a representative point, which is inside the shape
Counts do not match the plan Statuses computed but not reported Report new, duplicate and review counts every run

Specification reference

OpenStreetMap address data may be attached to a node, to a way or relation representing a building or site, or asserted over a range by an addr:interpolation way whose endpoints carry addr:housenumber values. The interpolation value may be odd, even, all, or a numeric step. See the addresses documentation on the OSM Wiki for the tagging of each form and the semantics of interpolation.

Frequently Asked Questions

Why is checking address nodes not enough?

Because in many regions most addresses are tagged on the building rather than on a separate point, and in others whole residential streets are represented by interpolation ways with no individual address objects at all. A node-only check finds none of those, reports them as new, and the import adds a duplicate address inside every already-addressed building and along every interpolated street. Which form dominates varies by region, so all three have to be checked.

How should housenumbers be compared?

On a canonical form rather than as raw strings. Reduce to the numeric part plus an optional single-letter suffix, folding case and removing whitespace, so that differently formatted versions of one number compare equal. Ranges and compound numbers need a local decision, because whether “12-14” is one address or three depends on the country’s conventions — and that decision belongs in the published import plan.

What if the same address exists far from where my record places it?

Route it to review rather than deciding automatically. A matching street and number beyond the search radius can mean a long street where both are genuine, a repeated street name in the same settlement, or a positional error on one side. All three are real, they need different responses, and none of them is safe to guess at. It is also a small enough group that human review is affordable.

Does a duplicate mean the OSM data is better?

Not necessarily, and this is exactly why what happens to a duplicate belongs in the plan rather than in the code. The existing feature may be a surveyed address a mapper walked past, or it may be a rough placement from an older import. Adding attributes it lacks is usually uncontroversial; overwriting what it has is not. Decide the policy in advance, publish it, and let the code implement one rule rather than improvise.

Up one level: Preparing an OSM Import.