Flagging Deprecated OSM Tags in a Pipeline Jump to heading#

Detect tags the OSM community has retired — such as highway=ford, older amenity values, and barrier=wire_fence — as an extract streams through ETL, and emit a warning with the suggested modern replacement for each one.

Prerequisites Jump to heading#

Confirm each item; a stale map or the wrong match granularity is the usual reason the report misses real deprecations or flags current tags.

• pyosmium ≥ 3.6 for the streaming path, or pandas ≥ 2.1 for the frame path (pip install "pyosmium>=3.6" / pip install "pandas>=2.1").
• A deprecation map — the key/value → replacement table below, ideally pinned to a dated snapshot of the OSM Wiki.
• Python 3.10+ for the dict[...] and dataclass typing used here.
• A tag source: a .osm.pbf extract for the streaming path, or a widened tag frame from Tag & Attribute Consistency Checks for the pandas path.
• Awareness that deprecation is informational — the data still parses; this pass suggests migrations rather than rejecting elements.

Conceptual minimum Jump to heading#

OpenStreetMap tagging is a living convention. Over the years the community retires keys and values in favour of clearer successors — a river crossing that was once highway=ford is now expressed with ford=yes on the crossing node, and various barrier sub-values were consolidated into fence with a fence_type. None of these deprecated tags is invalid: a parser reads them fine and a renderer may still draw them. But they drift out of step with the vocabulary that downstream consumers, presets, and analyses expect, so a pipeline that ingests OSM for the long term needs to surface them for migration. This is the deprecation rule class from Tag & Attribute Consistency Checks, isolated into a runnable pass.

The mechanism is a lookup, not a computation. A deprecation map keys each retired tag to its modern replacement; the checker asks, for every element, whether any of its tags matches an entry and, if so, records a finding carrying the suggested successor. The only real subtlety is match granularity: some deprecations are whole keys (any use of the key is retired), and some are a specific key-value pair (the key is fine but one value is deprecated). Conflating the two either over-reports — flagging every barrier when only one value is retired — or under-reports. The reference for which tags are current lives on the OSM Wiki, so the map should be treated as a pinned snapshot of that document rather than a hard-coded constant that silently rots.

Runnable solution Jump to heading#

This module defines a deprecation map with both match kinds, then offers two entry points that share it: a pyosmium handler for streaming a .osm.pbf under bounded memory, and a vectorized function for a pandas tag frame. Both emit the same Finding shape. It targets pyosmium>=3.6, pandas>=2.1, and Python 3.10+.

from __future__ import annotations

import csv
import logging
from dataclasses import asdict, dataclass

import osmium
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.deprecated_tags")

# Whole-key deprecations: any use of the key is retired.
KEY_DEPRECATIONS: dict[str, str] = {
    "highway=ford": "ford=yes (on the crossing node)",
}

# Key=value deprecations: the key is fine, this value is retired.
VALUE_DEPRECATIONS: dict[tuple[str, str], str] = {
    ("highway", "ford"): "ford=yes on the node",
    ("barrier", "wire_fence"): "barrier=fence + fence_type=wire",
    ("amenity", "nightclub"): "amenity=nightclub is current; verify against Wiki",
    ("shop", "organic"): "organic=only/yes on the relevant shop",
    ("highway", "unsurfaced"): "highway=* + surface=unpaved",
}


@dataclass(frozen=True)
class Finding:
    element: str
    key: str
    value: str
    replacement: str
    severity: str = "info"


def check_tags(ref: str, tags: dict[str, str]) -> list[Finding]:
    """Return a finding for every deprecated tag on one element."""
    findings: list[Finding] = []
    for key, value in tags.items():
        replacement = VALUE_DEPRECATIONS.get((key, value))
        if replacement is not None:
            findings.append(Finding(ref, key, value, replacement))
    return findings


class DeprecationHandler(osmium.SimpleHandler):
    """Stream a PBF and collect deprecated-tag findings under bounded memory."""

    def __init__(self) -> None:
        super().__init__()
        self.findings: list[Finding] = []

    def _scan(self, ref: str, obj) -> None:
        tags = {t.k: t.v for t in obj.tags}
        self.findings.extend(check_tags(ref, tags))

    def node(self, n: osmium.osm.Node) -> None:
        self._scan(f"node/{n.id}", n)

    def way(self, w: osmium.osm.Way) -> None:
        self._scan(f"way/{w.id}", w)

    def relation(self, r: osmium.osm.Relation) -> None:
        self._scan(f"relation/{r.id}", r)


def flag_stream(path: str, report: str = "deprecated.csv") -> list[Finding]:
    """Stream a PBF, flag deprecated tags, and write a CSV report."""
    handler = DeprecationHandler()
    handler.apply_file(path)
    _write_report(handler.findings, report)
    logger.info("streamed %s: %d deprecated tags found", path, len(handler.findings))
    return handler.findings


def flag_frame(df: pd.DataFrame, id_col: str = "osmid") -> pd.DataFrame:
    """Vectorized deprecation scan over a wide tag frame (one column per key)."""
    hits: list[pd.DataFrame] = []
    for (key, value), replacement in VALUE_DEPRECATIONS.items():
        if key not in df.columns:
            continue
        mask = df[key] == value
        if mask.any():
            matched = df.loc[mask, [id_col, key]].copy()
            matched["deprecated"] = f"{key}={value}"
            matched["replacement"] = replacement
            hits.append(matched.rename(columns={key: "value"}))
    result = pd.concat(hits, ignore_index=True) if hits else pd.DataFrame(
        columns=[id_col, "value", "deprecated", "replacement"])
    logger.info("frame scan: %d deprecated tags across %d rules", len(result), len(VALUE_DEPRECATIONS))
    return result


def _write_report(findings: list[Finding], path: str) -> None:
    with open(path, "w", newline="", encoding="utf-8") as fh:
        writer = csv.DictWriter(fh, fieldnames=["element", "key", "value", "replacement", "severity"])
        writer.writeheader()
        writer.writerows(asdict(f) for f in findings)


if __name__ == "__main__":
    flag_stream("extract.osm.pbf")

Step-by-step walkthrough Jump to heading#

1. Two match kinds, one lookup — VALUE_DEPRECATIONS keys on a (key, value) tuple so a specific retired value is flagged while other values of the same key pass, and check_tags does a single dictionary lookup per tag rather than iterating the whole map.
2. Shared checker — both the streaming and frame paths call the same rule data, so the definition of “deprecated” lives in exactly one place and cannot diverge between the two ingestion styles.
3. Bounded-memory streaming — DeprecationHandler subclasses osmium.SimpleHandler and receives one element at a time, so a planet file scans without ever loading into RAM; only the findings list grows, and that can be flushed periodically on a very dirty extract.
4. Type-prefixed references — findings carry node/123, way/456, or relation/789 so a reviewer knows exactly which element and type to open, since IDs are not unique across types.
5. Vectorized frame path — flag_frame tests each rule as a boolean mask over a column, collecting matched rows, which runs in C over millions of rows instead of a Python loop; it is the path to take when tags have already been widened into columns.
6. Uniform report — both paths serialize to a flat table (element, key, value, replacement) so the output feeds a migration backlog, a dashboard, or a diff without reshaping.

Verification Jump to heading#

Confirm the pass behaves before trusting the report:

• Known positive. Seed a tiny extract or frame with highway=ford and confirm exactly one finding appears with ford=yes as the replacement.
• Value granularity holds. A barrier=fence element must not be flagged while barrier=wire_fence must be, proving the map keys on the pair, not the bare key.
• Counts reconcile. The number of report rows equals the number of matching tags across the extract; a df.groupby("deprecated").size() on the frame path should match a manual grep-style count per rule.
• Streaming and frame agree. Run both paths over the same data reduced to a frame and confirm identical finding counts per rule, proving the shared rule data is applied consistently.
• Idempotent report. Re-running produces the same rows; the pass reads and reports but never mutates the source, so a second run over unchanged data is byte-identical.

Common errors and fixes Jump to heading#

Specification reference Jump to heading#

The authoritative list of current OpenStreetMap tags is the Map features page, and retired tags with their recommended successors are catalogued on the OSM Wiki Deprecated features page. Treat both as the source of truth for the deprecation map and pin your copy to a dated snapshot, because the community revises these conventions over time and a hard-coded map silently drifts out of date.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Tag & Attribute Consistency Checks — the parent reference where deprecation is one of five tag rule classes.
• Tag Taxonomy & Key-Value Standards — the vocabulary that defines which tags are current.
• Value Standardization & Regex Cleaning — the upstream pass that cleans values before a deprecation lookup.
• Authoring OSM Validation Rules — turning the deprecation map into a maintained preset.
• OSM Data Quality & Validation — the section gathering every validation discipline.

Up one level: Tag & Attribute Consistency Checks.