Authoring OSM Validation Rules Jump to heading#

Picture the on-call page nobody wants: a routing product starts sending drivers down a footpath because a contributor retagged a service road, and the defect sailed through ingestion because no rule in your pipeline knew that highway=footway should never carry car traffic. The fix is not a heroic patch; it is a validation rule you should have authored months earlier. Authoring rules is the craft at the centre of the OSM Data Quality & Validation layer — turning “we should check for that” into a durable, testable predicate that runs on every extract. The difficulty is that OSM has three different rule ecosystems, each with its own language and runtime, and teams waste enormous effort porting a check from one to another because they never separated the intent of a rule from its expression. This guide fixes that by grounding every ecosystem in one portable rule model and giving you a Python engine that runs it at pipeline scale.

Prerequisites Jump to heading#

Before authoring rules, three foundations should be in place. First, read the parent OSM Data Quality & Validation overview for the severity taxonomy and quarantine policy this guide assumes — rules produce findings, and findings only mean something inside that framework. Second, be comfortable with the Tag Taxonomy & Key-Value Standards reference, because most selectors and many predicates are expressed in terms of keys and their expected value domains. Third, your input must already be parsed and normalized: validating raw, un-normalized tags floods every rule with false positives from casing and whitespace variants, so the normalization stage runs first and the checks here run on its output.

The Unified Rule Model Jump to heading#

Every rule this guide authors, in any engine, is the same four-slot record. Naming the slots explicitly is what makes a rule portable rather than trapped in the tool it was first written for.

Two design rules apply regardless of engine. The selector must be cheap and specific — a fast tag or geometry-type test that discards the vast majority of features before any expensive predicate runs, because a loose selector that admits every node to an intersection test will dominate your runtime for no benefit. The predicate must be pure and total — it returns a verdict for every input including the awkward ones (missing tag, empty geometry, <NA> value) rather than throwing, so one malformed feature never aborts the run. A rule that raises on unexpected input is a latent pipeline outage.

The Three Engines Compared Jump to heading#

Each ecosystem occupies a different point in the data’s lifecycle, and knowing which to reach for is half of authoring well.

• JOSM validator (MapCSS). Runs inside the editor at authoring time, so its rules prevent defects before a changeset is ever uploaded. You express checks as MapCSS selectors with throwError/throwWarning actions, and validator presets constrain the allowed values for a key. This is the right home for rules that a mapper can and should fix on the spot — crossing ways, an unclosed building, a nonsensical value. The full authoring workflow is in Writing Custom JOSM Validation Presets.
• Osmose rule DSL. Runs server-side on a schedule over whole countries, so its rules surface the community’s standing backlog on a map. Analysers are configured as parameterised checks with stable item and class numbers, per-class severity levels, and localised titles. Because Osmose has been run against the planet for years, its rule catalogue is an inventory of real, recurring defect patterns worth mirroring; Authoring Osmose Rule DSL Checks covers its analyser structure and output format.
• Python custom rules. Run inside your ETL over pyosmium handlers or a geopandas frame, so they enforce the contract your specific consumers depend on. This is where domain rules live — gazetteer cross-checks, region-specific tagging policy, invariants no general tool could know. It is also the only layer you fully control end to end, which is why it is the backstop for anything the first two miss. Building Python-Based OSM Validation Rules develops production versions of the skeleton below.

The practical workflow is to prototype a rule’s intent against your data in Python, where iteration is fastest and you have a debugger, then decide whether it also belongs upstream in JOSM (to prevent the defect) or in an Osmose-style scan (to crowd-source the fix). The intent — the four slots — is identical; only the expression changes.

Step-by-Step: A Runnable Python Rule Engine Jump to heading#

The following skeleton turns the unified model into working code. A Rule bundles a selector, a predicate, a severity, and a message template; a RuleEngine dispatches every feature through the rules whose selector matches and collects ValidationIssue findings. It uses Python 3.10+ type hints and the project logger pattern, and it is deliberately engine-agnostic so a rule authored here reads the same whether its logic mirrors a JOSM test or an Osmose check.

1. Model a feature uniformly. Wrap each parsed element in a lightweight view exposing osm_type, osm_id, tags, and an optional reconstructed geometry, so selectors and predicates never touch pyosmium internals directly and the same rules run over a geopandas frame.
2. Author each rule as data. A Rule is a selector callable, a predicate callable, a severity, and a message template — no control flow, so rules are trivially unit-tested in isolation.
3. Dispatch through the engine. For each feature, run only the rules whose selector matches, evaluate the predicate, and on failure emit a finding; wrap each predicate so an unexpected exception becomes a logged internal error rather than a crashed run.
4. Collect and hand off. Return a list of ValidationIssue records for the reporting and quarantine path defined in the parent section.

from __future__ import annotations

import logging
from collections.abc import Callable, Iterable
from dataclasses import dataclass, field
from enum import Enum

logger = logging.getLogger(__name__)


class Severity(str, Enum):
    ERROR = "error"
    WARNING = "warning"
    INFO = "info"


@dataclass(slots=True)
class Feature:
    """Engine-agnostic view of one OSM element."""

    osm_type: str
    osm_id: int
    tags: dict[str, str]
    geometry: object | None = None  # a shapely geometry once reconstructed


@dataclass(slots=True)
class Issue:
    osm_type: str
    osm_id: int
    rule_id: str
    severity: Severity
    message: str


# A selector answers "does this rule apply?"; a predicate answers "is it valid?".
Selector = Callable[[Feature], bool]
Predicate = Callable[[Feature], bool]


@dataclass(slots=True)
class Rule:
    """The unified rule model: selector, predicate, severity, message."""

    rule_id: str
    selector: Selector
    predicate: Predicate           # returns True when the feature is VALID
    severity: Severity
    message: str                   # may reference {id}

    def check(self, feature: Feature) -> Issue | None:
        if not self.selector(feature):
            return None            # rule does not apply to this feature
        try:
            ok = self.predicate(feature)
        except Exception as exc:   # a bad feature must never abort the run
            logger.error("rule %s raised on %s/%s: %s",
                         self.rule_id, feature.osm_type, feature.osm_id, exc)
            return None
        if ok:
            return None
        return Issue(
            osm_type=feature.osm_type,
            osm_id=feature.osm_id,
            rule_id=self.rule_id,
            severity=self.severity,
            message=self.message.format(id=feature.osm_id),
        )


@dataclass(slots=True)
class RuleEngine:
    rules: list[Rule] = field(default_factory=list)

    def run(self, features: Iterable[Feature]) -> list[Issue]:
        issues: list[Issue] = []
        checked = 0
        for feature in features:
            checked += 1
            for rule in self.rules:
                issue = rule.check(feature)
                if issue is not None:
                    issues.append(issue)
        logger.info("checked %d features, %d issues found", checked, len(issues))
        return issues


# --- Authoring three rules against the shared model ---------------------------

VALID_ONEWAY = {"yes", "no", "-1", "reversible", "alternating"}


def _has(key: str) -> Selector:
    return lambda f: key in f.tags


engine = RuleEngine(rules=[
    Rule(
        rule_id="tag.oneway_domain",
        selector=_has("oneway"),
        predicate=lambda f: f.tags["oneway"] in VALID_ONEWAY,
        severity=Severity.WARNING,
        message="way {id}: oneway has an unrecognised value",
    ),
    Rule(
        rule_id="tag.maxspeed_numeric",
        selector=_has("maxspeed"),
        predicate=lambda f: f.tags["maxspeed"].split()[0].isdigit()
                            or f.tags["maxspeed"] in {"none", "walk"},
        severity=Severity.WARNING,
        message="way {id}: maxspeed does not parse as a speed",
    ),
    Rule(
        rule_id="tag.addr_needs_street",
        selector=_has("addr:housenumber"),
        predicate=lambda f: "addr:street" in f.tags,
        severity=Severity.INFO,
        message="feature {id}: addr:housenumber without addr:street",
    ),
])

Feeding this engine a stream of Feature objects — built from a pyosmium handler for planet-scale streaming, or from GeoDataFrame.itertuples() for a regional frame — yields a flat list of findings ready for the report and quarantine path. Because a Rule is pure data plus two callables, each one is unit-tested by constructing a single Feature and asserting the returned Issue, which is how a rule catalogue stays trustworthy as it grows to hundreds of entries.

Running Rules at Pipeline Scale Jump to heading#

A rule catalogue that runs in seconds on a city extract can take hours on a continent if it is applied naively, and the fix is structural rather than a matter of faster hardware. Three levers dominate.

First, evaluate selectors before predicates and cheapest-first within a rule. The engine above already short-circuits on the selector; go further by ordering a compound predicate so the tag test that rejects 99% of candidates runs before the geometry test. Second, index once, reuse everywhere. Rules that need spatial neighbours — near-miss junctions, features crossing a boundary — should share a single R-tree built once per batch, following the pattern in the fundamentals layer, rather than each rebuilding its own. Third, partition on PBF blocks, not rules. Because blocks are independent, you parallelise by feeding disjoint geographic partitions to separate RuleEngine workers and merging their findings, keeping memory bounded the same way the Memory-Efficient Chunk Processing stage does; rules that need global context (connected components over the whole graph) run in a dedicated non-partitioned pass.

The measurement that matters is per-rule cost. Instrument the engine to record how long each rule_id spends and how many features it touched, and the expensive outliers — almost always a rule with a loose selector or a redundant geometry rebuild — reveal themselves immediately.

Validation & Error-Handling Matrix Jump to heading#

Authoring rules has its own failure modes, distinct from the defects the rules detect. Each row is a real problem seen when building a rule catalogue.

Failure Modes & Gotchas Jump to heading#

• Predicate polarity. By convention here a predicate returns True for a valid feature, so a finding is emitted when it returns False. Mixing this with a “return True if broken” helper is the single most common authoring bug and it silently inverts a whole rule.
• Selectors on missing keys. f.tags["oneway"] raises KeyError if the selector let a feature through without the key; always gate the predicate behind a selector that guarantees the key, or use .get.
• Unit-carrying values. maxspeed, width, and maxheight may carry units (30 mph, 3.5 m); a naive isdigit() predicate flags every legitimate unit-qualified value. Parse the numeric head and validate the unit against an allow-list.
• Locale and script. Name and value predicates that assume ASCII flag perfectly valid non-Latin data. Rules over name:* must be script-aware or scoped to a specific language key.
• Severity drift. Copying a rule and forgetting to reset its severity is how an informational nudge becomes an error that quarantines valid features. Review severity whenever you clone a rule.
• Global-context rules in a partitioned run. A connectivity rule evaluated per-partition reports every partition boundary as a disconnection. Route such rules to a dedicated whole-graph pass.

False-Positive Management Jump to heading#

The fastest way to get a validation gate ignored is to let it cry wolf. A rule that fires on legitimate data trains reviewers to dismiss its class entirely, and the one real defect hidden among a hundred false alarms ships. Managing false positives is therefore a first-class authoring task, not an afterthought.

Three practices keep the signal high. Version and test every rule against a golden sample — a small, hand-verified extract where you know the correct finding count, so a change that inflates false positives is caught before deploy. Support suppression as data: some findings are known-and-accepted (a genuinely disconnected private driveway, a valid but unusual tag combination), and a rule engine needs an allow-list keyed on (type, id, rule_id) so an acknowledged finding stops re-alerting without weakening the rule for everyone else. Track the false-positive rate per rule using reviewer feedback, and retire or tighten any rule whose findings are dismissed more often than actioned. A rule that is wrong more than it is right is worse than no rule, because it consumes attention that a good rule needs.

Integration Points Jump to heading#

The rule engine’s output is a stream of Issue records, and its job ends there — routing them is the parent section’s responsibility. Error-severity findings drive quarantine; warnings and info flow to the report. The wiring is a thin adapter that partitions findings by whether they block publication:

def route_findings(issues: list[Issue]) -> tuple[list[Issue], list[Issue]]:
    """Split findings into blocking (quarantine) and advisory (report only)."""
    blocking = [i for i in issues if i.severity is Severity.ERROR]
    advisory = [i for i in issues if i.severity is not Severity.ERROR]
    logger.info("%d blocking, %d advisory findings", len(blocking), len(advisory))
    # blocking → quarantine keyed on (osm_type, osm_id); advisory → report sink
    return blocking, advisory

Upstream, the engine consumes the reconstructed, normalized features produced by the parsing and tag-normalization workflows; downstream, its findings feed the same quarantine table described in Error Handling in Large OSM Extracts. Geometry-specific rules authored here hand off to the deeper detection-and-repair logic in Geometry Validation & Repair, which the engine calls as just another predicate.

Rule-Authoring Guides in Depth Jump to heading#

Each guide below takes one engine from the model above and develops production rules in it:

• Writing Custom JOSM Validation Presets — MapCSS validator tests and value-constraining presets that catch defects in the editor before upload.
• Authoring Osmose Rule DSL Checks — the Osmose analyser rule model, item and class numbering, and the geolocated issue output it produces.
• Building Python-Based OSM Validation Rules — production rules over pyosmium and geopandas that extend the engine skeleton on this page.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• OSM Data Quality & Validation — the parent section with the severity taxonomy, quarantine policy, and reporting model these rules feed.
• Geometry Validation & Repair — the sibling stage whose detection logic a geometry rule calls as its predicate.
• Tag Taxonomy & Key-Value Standards — the key-value reference selectors and value-domain predicates are written against.
• Parsing & Tag Normalization Workflows — the upstream stage producing the reconstructed, normalized features the engine consumes.
• Memory-Efficient Chunk Processing — the block-partitioning discipline behind running rules at planet scale.
• Error Handling in Large OSM Extracts — the quarantine and remediation path error-severity findings are routed to.

Up one level: OSM Data Quality & Validation.