Cleaning OSM Website and Contact URL Tags

A URL tag looks like the easiest field in OSM to normalise and is not, because half the values are missing a scheme, a tenth are not URLs at all, and the obvious normalisation of lower-casing the whole string breaks every case-sensitive path.

Prerequisites

Conceptual minimum

Four transformations cover almost all of it, and one common instinct is wrong.

Supply a missing scheme. A value beginning with a host name rather than a scheme is the single most common shape, and prefixing a secure scheme makes it a URL. It is a guess, and it is the one guess in this area that is safe, because the alternative is a value no consumer can follow.

Normalise the host, not the path. Host names are case-insensitive and paths are not. Lower-casing the whole string is the obvious normalisation and it breaks every path on a case-sensitive server, which is most of them.

Handle international domains. A host with non-ASCII characters has an ASCII-compatible encoding, and storing both the display form and the encoded form serves different consumers.

Reject non-URLs structurally. Email addresses, phone numbers, social media handles and free text all appear in URL tags. A value with no host, or whose host has no dot, is not a URL, and saying so is more useful than storing it.

The instinct to avoid is verifying by fetching. It is slow, it fails for reasons unrelated to the data, it makes the pipeline non-deterministic, and it means your infrastructure visits every URL in the map — which is a privacy and etiquette problem as well as a technical one.

Four transformations and the mistake each one replaces A grid of four transformations against what each one does and the naive approach it replaces. Supplying a missing scheme turns a bare host into a followable URL, replacing the alternative of discarding it. Normalising only the host preserves case-sensitive paths, replacing the instinct to lower-case the whole string. Encoding an international domain stores an ASCII-compatible form alongside the display form, replacing the alternative of storing something no resolver accepts. Structural rejection identifies values that are not URLs at all, replacing verification by fetching, which is slow and non-deterministic. Four transformations, four instincts corrected Does Replaces Supply a scheme bare host becomes a URL discarding it Normalise host only paths keep their case lower-casing everything Encode the domain stores both forms an unresolvable host Reject structurally identifies non-URLs fetching to check The last row is the important one: fetching makes a cleaning stage slow, non-deterministic and a privacy question all at once.
Every one of these is a string operation, which is what keeps the stage fast and reproducible.

Runnable solution

python 
from __future__ import annotations

import logging
import re
from dataclasses import dataclass
from urllib.parse import urlsplit, urlunsplit, quote

import idna

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.clean.url")

URL_KEYS = ("website", "contact:website", "url", "contact:url", "operator:website")
EMAIL = re.compile(r"^[^@\s]+@[^@\s]+\.[A-Za-z]{2,}$")
HANDLE = re.compile(r"^@[\w.]+$")
SPLIT = re.compile(r"\s*[;\s]\s*(?=https?://|www\.)|\s*;\s*")
DEFAULT_SCHEME = "https"


@dataclass(frozen=True)
class Url:
    normalised: str | None
    display_host: str | None
    original: str
    key: str
    status: str        # 'ok' | 'scheme_added' | 'not_a_url' | 'email' | 'handle'


def looks_like_host(text: str) -> bool:
    head = text.split("/", 1)[0]
    # A host needs a dot and no whitespace. This is deliberately structural.
    return "." in head and " " not in head and not head.endswith(".")


def normalise_one(raw: str, key: str) -> Url:
    text = raw.strip()
    if not text:
        return Url(None, None, raw, key, "not_a_url")
    if EMAIL.match(text):
        # An email in a website tag is information, in the wrong field.
        return Url(None, None, raw, key, "email")
    if HANDLE.match(text):
        return Url(None, None, raw, key, "handle")

    added = False
    if "://" not in text:
        if not looks_like_host(text):
            return Url(None, None, raw, key, "not_a_url")
        text = f"{DEFAULT_SCHEME}://{text}"
        added = True

    parts = urlsplit(text)
    if parts.scheme not in {"http", "https"} or not parts.netloc:
        return Url(None, None, raw, key, "not_a_url")

    host = parts.hostname or ""
    if not looks_like_host(host):
        return Url(None, None, raw, key, "not_a_url")

    display_host = host
    try:
        # An international domain needs its ASCII-compatible form to resolve.
        ascii_host = idna.encode(host, uts46=True).decode("ascii")
    except idna.IDNAError:
        ascii_host = host.lower()

    netloc = ascii_host
    if parts.port:
        netloc = f"{netloc}:{parts.port}"

    # Path, query and fragment keep their case: servers are case-sensitive.
    path = quote(parts.path, safe="/%:@!$&'()*+,;=~-._")
    normalised = urlunsplit((parts.scheme, netloc, path or "/",
                             parts.query, parts.fragment))
    return Url(normalised, display_host, raw, key,
               "scheme_added" if added else "ok")


def normalise_tags(tags: dict[str, str]) -> list[Url]:
    out: list[Url] = []
    for key in URL_KEYS:
        value = tags.get(key)
        if not value:
            continue
        for part in (p for p in SPLIT.split(value) if p.strip()):
            out.append(normalise_one(part, key))
    return out


def audit(results: list[Url]) -> dict[str, int]:
    counts: dict[str, int] = {}
    for url in results:
        counts[url.status] = counts.get(url.status, 0) + 1
    logger.info("url normalisation: %s", counts)
    misplaced = counts.get("email", 0) + counts.get("handle", 0)
    if misplaced:
        logger.info("%d value(s) are contact details in a website field; "
                    "route them to the right key rather than discarding",
                    misplaced)
    return counts


if __name__ == "__main__":
    tags = {"website": "www.example.org/Menu; https://Café.example/Über",
            "contact:website": "info@example.org"}
    for url in normalise_tags(tags):
        logger.info("%-18s %-14s %s", url.key, url.status,
                    url.normalised or url.original)

Step-by-step walkthrough

  1. Identify misplaced contact details first. An email address or a social handle in a website field is information in the wrong place, and naming it as such lets a pipeline move it rather than discard it.
  2. Test for a host structurally. A dot, no whitespace and no trailing dot is a cheap test that separates bare hosts from free text without pretending to validate a domain.
  3. Add the scheme only when the rest looks like a host. Prefixing a scheme onto arbitrary text produces a URL-shaped string that resolves to nothing.
  4. Restrict the accepted schemes. Anything other than the two web schemes in a website tag is either a mistake or something a consumer following links should not open.
  5. Encode international hosts, keep both forms. The encoded form is what resolves; the display form is what a human should see, and storing only one of them disappoints one audience.
  6. Leave the path alone apart from escaping. Paths are case-sensitive on most servers, and lower-casing them is the most common way a cleaning stage breaks working URLs.
  7. Report the misplaced counts. A high number of emails in website fields is a data-quality finding worth acting on upstream, not just a filtering statistic.
Three normalisations that look sensible and break working URLs Three panels. Lower-casing the whole string normalises the host correctly and destroys every case-sensitive path, which is most paths on most servers. Stripping a trailing slash changes a directory request into a file request on some servers, which redirects at best and fails at worst. Removing a query string discards parameters that identify the page entirely on sites that route through them, turning a specific link into a home page. Three tempting normalisations, all destructive Lower-case everything Host normalises correctly Path breaks Most servers are case-sensitive The most common damage Strip trailing slashes Looks tidier Directory becomes a file Redirect at best Failure at worst Drop query strings Looks like noise Often identifies the page Specific link becomes a home page Silently wrong All three produce a URL that still looks correct, which is why they survive review and are discovered by a reader clicking a broken link.
The safe rule is to normalise the host and to treat everything after it as opaque.
How URL values in a country extract actually distribute Five outcome categories with their approximate share of website and contact URL values across a country extract. Values already carrying a scheme and resolving structurally are the largest group. Values that are a bare host and gain a scheme are the next largest, close behind. Values that are email addresses in the wrong field are a small but significant share. Social media handles are smaller again. Free text that is not a URL at all is the smallest group. A note observes that the second group would be lost entirely by a pipeline that rejects anything without a scheme. Where URL values actually fall Already has a scheme about 57% Bare host, scheme added about 34% Email in the wrong field about 5% A social handle about 2.5% Not a URL at all under 2% A pipeline rejecting anything without a scheme discards the second bar, which is a third of every website value in the extract.
The size of that second group is why adding a scheme is worth the small risk the host test already removes.

Verification

  • Case-sensitive paths survive. A URL with mixed-case path segments must come through unchanged after the host.
  • Bare hosts gain a scheme. A value beginning with a host name must produce a followable URL with the added-scheme status.
  • Free text is rejected. A note or a sentence in a website field must not become a URL.
  • International domains encode. A host with non-ASCII characters should yield both an encoded and a display form.
  • Nothing is fetched. Watch the network during a run; a cleaning stage should make no requests at all.

Common errors and fixes

Symptom Root cause One-line fix
Links return not-found Whole string lower-cased Normalise the host only; leave the path as it is
Free text became a URL Scheme added without a host test Require a dot and no whitespace before prefixing
International domains fail Only the display form stored Store the encoded form alongside it
Emails silently dropped Non-URLs discarded rather than classified Detect and label them so they can be relocated
Cleaning stage is slow URLs fetched to verify Validate structurally; never fetch
Some values never seen Only the primary key read Read every key that carries a URL
Query parameters lost Query string stripped as noise Preserve the query; it often identifies the page

Specification reference

A URL’s host component is case-insensitive while its path, query and fragment are case-sensitive, and internationalised domain names have an ASCII-compatible encoding used for resolution. See RFC 3986 for the component definitions and case rules, and RFC 5891 for the internationalised domain encoding.

Frequently Asked Questions

Is adding a scheme a safe guess?

It is the one safe guess here, provided the rest of the value looks like a host. A bare host name is unambiguously intended as a web address, and prefixing a scheme makes it followable where the alternative is a value no consumer can use. What makes it safe is the host test: prefixing a scheme onto arbitrary text produces something URL-shaped that resolves to nothing, which is worse than rejecting it.

Why not lower-case the whole URL?

Because only the host is case-insensitive. Paths, query strings and fragments are case-sensitive on most servers, so lower-casing them turns working links into not-found errors. This is the most common way a URL cleaning stage does harm, and it is particularly insidious because the result still looks like a valid URL and fails only when somebody follows it.

Should I check that the URL resolves?

No, for four reasons that compound. It is slow, since every value becomes a network round trip. It is non-deterministic, since a site being down makes good data look bad. It makes the pipeline’s output depend on the moment it ran. And it means your infrastructure visits every URL in the map, which is a privacy and etiquette question as well as a technical one. Structural validation catches the values that are genuinely wrong.

What should happen to an email address in a website tag?

Label it rather than discarding it. It is real information filed under the wrong key, and a pipeline that recognises it can route it to the email field, flag it for a mapper, or simply record that it exists. Dropping it loses a contact detail somebody took the trouble to record; storing it as a website produces a link nothing can follow.

Up one level: Value Standardization & Regex Cleaning.