Best practices for OSM tag standardization across regions

Take several regional OpenStreetMap extracts that use divergent tag conventions — surface=cobblestone here, surface=sett there, name only in the local script elsewhere — and rewrite every region-specific or deprecated value to a single canonical form during streaming ETL, so a merged continental dataset queries consistently instead of fragmenting across community dialects.

Prerequisites

Confirm each item before running the code below; a skipped prerequisite is the usual reason a “normalized” extract still returns three spellings of the same surface value.

Conceptual minimum

Regional divergence is structural, not accidental: OpenStreetMap stores attributes as a sparse, free-form key-value map on every node, way, and relation, so nothing in the format prevents two communities from coining different values for the same real-world feature. Standardization is therefore a lookup-driven rewrite rather than a schema migration — you resolve each (key, value) pair against a canonical table drawn from the Tag Taxonomy & Key-Value Standards reference, leaving unrecognized pairs untouched. Because the wider OSM Data Fundamentals & Architecture model treats tags as opaque strings, the resolver must be deterministic and idempotent: running it twice on the same input must produce byte-identical output, or downstream diffing breaks.

Two refinements separate a production rewrite from a naive str.replace. First, multilingual name:* tags need a fallback hierarchy so a deduplicated record never loses its only label — if name is absent you promote name:en, then a configured local-language key. Second, audit obligations under the ODbL require that you never silently destroy source data; preserving the pre-normalization value in a was:* or source:* namespace keeps the transform reversible and traceable. Heavier value cleaning — case folding, whitespace, regex repair — belongs to Value Standardization with Regex Cleaning and runs as a sibling stage rather than inside this exact-match resolver.

Standardizing divergent regional OSM tags into one canonical merged extract Three regional PBF extracts using different surface values and name languages feed a TagStandardizer resolver that consults an alias table and a multilingual name fallback. It emits one canonical merged extract while a was:* audit branch preserves each original value for ODbL-compliant, reversible provenance. divergent regional extracts Region A · .osm.pbf surface=cobblestone name:de only Region B · .osm.pbf surface=sett name present Region C · .osm.pbf surface=bitumen name:fr only TagStandardizer resolver Alias table lookup (key, value) → canonical · O(1) Multilingual name fallback name:en → name:de → name:fr exact-match only · idempotent Canonical merged extract surface=sett (unified) name backfilled, never lost deprecated values resolved queries consistently was:* audit branch original value preserved per rewrite ODbL provenance · reversible

Runnable solution

This pyosmium handler streams an extract, rewrites region-specific and deprecated tag values to canonical equivalents, applies a multilingual name fallback, preserves every changed value under a was: prefix, and re-emits each primitive through a SimpleWriter. It targets pyosmium>=3.6.0 and Python 3.10+.

python 
import logging
import osmium

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.tag_standardizer")

# Canonical alias tables. Outer key = OSM tag key; inner map = deprecated or
# region-specific value -> canonical value. Snapshot from the OSM Wiki and pin a date.
REGIONAL_ALIASES: dict[str, dict[str, str]] = {
    "surface": {
        "cobblestone": "sett",
        "unhewn_cobblestone": "cobblestone",
        "bitumen": "asphalt",
    },
    "oneway": {
        "-1": "reversible",
    },
    "building": {
        "yes;residential": "residential",
    },
}

# Ordered fallback for a missing primary `name`. First present key wins.
NAME_FALLBACK: tuple[str, ...] = ("name:en", "name:de", "name:fr")


class TagStandardizer(osmium.SimpleHandler):
    """Stream an OSM extract, canonicalizing region-specific tag values.

    pyosmium primitives are immutable, so each element is re-emitted via
    ``element.replace(tags=...)``. Every rewritten value is preserved under a
    ``was:<key>`` tag so the transform stays auditable and reversible.
    """

    def __init__(self, writer: osmium.SimpleWriter) -> None:
        super().__init__()
        self.writer = writer
        self.stats: dict[str, int] = {"normalized": 0, "passthrough": 0, "name_filled": 0}

    def _normalize(self, tags) -> dict[str, str]:
        out: dict[str, str] = {}
        for tag in tags:
            key, value = tag.k, tag.v
            alias = REGIONAL_ALIASES.get(key)
            if alias and value in alias:
                canonical = alias[value]
                out[key] = canonical
                out[f"was:{key}"] = value  # audit trail; idempotent on re-run
                self.stats["normalized"] += 1
            else:
                out[key] = value
                self.stats["passthrough"] += 1

        # Multilingual fallback: only fill `name` when it is genuinely absent.
        if "name" not in out:
            for candidate in NAME_FALLBACK:
                if candidate in out:
                    out["name"] = out[candidate]
                    self.stats["name_filled"] += 1
                    break
        return out

    def node(self, n: osmium.osm.Node) -> None:
        self.writer.add_node(n.replace(tags=self._normalize(n.tags)))

    def way(self, w: osmium.osm.Way) -> None:
        self.writer.add_way(w.replace(tags=self._normalize(w.tags)))

    def relation(self, r: osmium.osm.Relation) -> None:
        self.writer.add_relation(r.replace(tags=self._normalize(r.tags)))


if __name__ == "__main__":
    writer = osmium.SimpleWriter("region-standardized.osm.pbf")
    handler = TagStandardizer(writer)
    try:
        handler.apply_file("region-raw.osm.pbf")
    finally:
        writer.close()  # flush the buffered output block before exit
    logger.info(
        "normalized=%(normalized)d passthrough=%(passthrough)d name_filled=%(name_filled)d",
        handler.stats,
    )

Step-by-step walkthrough

  1. Alias tables as the single source of truth — REGIONAL_ALIASES keys by tag key, then by deprecated value, so resolution is an O(1) double dictionary lookup. Keep these tables in version control and stamp the Wiki snapshot date; the code itself stays unchanged as conventions evolve.
  2. Immutable rewrite pattern — pyosmium elements cannot be mutated in place, so _normalize builds a fresh dict[str, str] and each handler calls element.replace(tags=...) to emit a modified copy through the SimpleWriter.
  3. Exact-match resolution — only (key, value) pairs present in the alias table are rewritten; everything else passes through verbatim, which keeps the transform conservative and avoids corrupting values the table does not own.
  4. Audit trail — every rewrite writes the original under was:<key>. Because the canonical value never re-matches the alias table on a second run, this stays idempotent: re-processing an already-standardized file is a no-op apart from re-emitting identical was: tags.
  5. Multilingual fallback — name is filled from the first present key in NAME_FALLBACK only when no primary name exists, so a label is never lost during a later merge or deduplication and an existing name is never overwritten.
  6. Stats counters — normalized, passthrough, and name_filled give a one-line health summary per run; a normalized count of zero on a known-dirty extract means the alias table did not load.
  7. Writer lifecycle — SimpleWriter buffers output into PBF blocks, so the finally: writer.close() is mandatory; skipping it truncates the final block and produces a file osmium fileinfo reports as corrupt.

Verification

Confirm the rewrite is correct before merging the standardized extracts:

  • Round-trip the stats line. normalized + passthrough should equal the total tag count from osmium fileinfo --extended region-raw.osm.pbf; a mismatch means tags were dropped.
  • Grep for residual aliases. osmium tags-filter region-standardized.osm.pbf nwr/surface=cobblestone must return zero matches once cobblestone maps to sett.
  • Confirm the audit trail. osmium tags-filter region-standardized.osm.pbf nwr/was:surface should return exactly the count reported as normalized.
  • Check name backfill. Spot-check a feature that had only name:en — the output must now carry both name and name:en, never an overwritten primary name.
  • Prove idempotency. Run the handler on its own output; osmium diff between the two passes must report no changes.

Common errors and fixes

Symptom Root cause One-line fix
Output PBF reported as corrupt SimpleWriter never closed Wrap apply_file in try/finally: writer.close().
normalized=0 on a dirty extract Alias table empty or wrong keys Verify REGIONAL_ALIASES keys match real OSM tag keys, not values.
Primary name overwritten Fallback runs unconditionally Gate the fallback on if "name" not in out.
Second run changes the file Audit value re-matches the table Map deprecated→canonical only; never list a canonical value as a key.
RuntimeError on add_* Mutating immutable primitives Build a new tag dict and use element.replace(tags=...).
Memory climbs on a planet file Geometry/location index loaded needlessly Pass locations=False (the default) for tag-only rewrites.
Merged dataset still has duplicates Case/whitespace variance, not aliases Hand those to a regex cleaning stage before exact-match resolution.

Specification reference

OpenStreetMap tags are free-form UTF-8 key-value pairs with no enforced enumeration; canonical values are community conventions documented per key on the OSM Wiki — see Map features and the Deprecated features list for the deprecations these alias tables encode. Any redistribution of the standardized extract remains bound by the Open Database License (ODbL), which is why the was: audit namespace preserves provenance rather than discarding the source value.

For the byte-level mechanics of how these tags are stored and re-emitted, the PBF File Structure Deep Dive covers the string-table deduplication that makes a full read-modify-write affordable, and Coordinate Reference Systems in OSM explains why any geometry filtering happens after, not during, tag resolution.

Up one level: Tag Taxonomy & Key-Value Standards.