Node-Way-Relation Data Model Jump to heading#

The Reference-Resolution Problem Jump to heading#

OpenStreetMap stores no precomputed geometry. A way is not a line — it is an ordered list of 64-bit node identifiers, and a relation is not a polygon — it is a list of typed member references with roles. Every coordinate your pipeline emits is reconstructed by dereferencing those identifiers against the node table you built earlier in the same pass. This deferred, pointer-based model is what makes OSM editable and compact, and it is also the single largest source of silent ETL failures.

Consider a concrete failure scenario. You stream a regional .osm.pbf extract clipped to a bounding box, build a node coordinate index, then reconstruct ways. A coastal way crosses the clip boundary, so three of its node references resolve and two do not. A naive parser either raises a KeyError and aborts the whole run, or — worse — silently drops the missing nodes and emits a polygon with a sliver where the coastline was cut. The geometry passes a basic is_valid check, lands in your warehouse, and corrupts every area calculation downstream. The defect is invisible until an analyst notices that a national land-cover total is 4% short. The fix is not more validation at the end of the pipeline; it is enforcing reference closure at the moment of reconstruction, which is exactly what the node, way, and relation contract below specifies.

This reference graph is the foundation introduced in OSM Data Fundamentals & Architecture; the page you are reading is where that graph becomes runnable geometry.

Prerequisite Concepts Jump to heading#

Before reconstructing geometry you should be comfortable with three foundational ideas:

• The physical encoding the references arrive in — the PBF File Structure Deep Dive explains how dense nodes, delta-encoded IDs, and string tables deliver these primitives to your handler, and why node coordinates must be resolved with locations=True.
• The coordinate space the resolved points live in — every raw node is stored in WGS 84 (EPSG:4326), and any projection happens after assembly as covered in Coordinate Reference Systems in OSM.
• The semantic layer that turns geometry into features — keys and values follow the conventions documented in Tag Taxonomy & Key-Value Standards, which decide whether a closed way is a building, a roundabout, or a lake.

Specification & Field Reference Jump to heading#

Each primitive carries a globally unique signed 64-bit identifier, an extensible key-value tag map, and a metadata block. Identifier namespaces are independent per type: node 1, way 1, and relation 1 are three distinct objects. The table below summarizes the fields your reader must populate and the constraints it must enforce.

Two encoding facts shape every parser. First, coordinates in PBF are stored as integer nanodegrees (degrees × 10⁻⁷) and delta-encoded within a primitive group, so a node at latitude 51.5074° is carried as the integer 515074000 relative to the running accumulator. Second, a closed way is not automatically an area — the area=yes tag, or an area-implying key such as building or landuse, is what distinguishes a polygon from a closed linear loop like a roundabout. Encoding the wrong assumption here produces topologically valid but semantically wrong features.

Step-by-Step Implementation Jump to heading#

The pipeline runs in three ordered stages: index nodes, reconstruct ways, then assemble relations. Each stage depends on the working set produced by the previous one.

1. Index and validate nodes Jump to heading#

Nodes are the atomic spatial units. Stream them first, enforce strict WGS 84 bounds, reject non-finite values, and keep only what downstream stages will dereference. This streaming validator uses pyosmium and never materializes the whole file in memory.

import osmium
import numpy as np
from typing import Dict, Tuple, Optional
import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")


class NodeValidator(osmium.SimpleHandler):
    def __init__(self, max_nodes: Optional[int] = None) -> None:
        super().__init__()
        self.valid_nodes: Dict[int, Tuple[float, float]] = {}
        self.invalid_count = 0
        self.max_nodes = max_nodes

    def node(self, n: osmium.osm.Node) -> None:
        if self.max_nodes is not None and len(self.valid_nodes) >= self.max_nodes:
            return
        try:
            lat, lon = n.location.lat, n.location.lon
            # Strict WGS 84 bounds and finiteness check
            if (
                -90.0 <= lat <= 90.0
                and -180.0 <= lon <= 180.0
                and np.isfinite(lat)
                and np.isfinite(lon)
            ):
                self.valid_nodes[n.id] = (lat, lon)
            else:
                self.invalid_count += 1
                logging.debug("Invalid coordinates for node %d: (%f, %f)", n.id, lat, lon)
        except Exception as e:  # noqa: BLE001 - log and continue, never abort the stream
            logging.warning("Failed to process node %d: %s", n.id, e)
            self.invalid_count += 1

    def get_indexed_nodes(self) -> Dict[int, Tuple[float, float]]:
        return self.valid_nodes


# Usage: apply_file with locations=True so pyosmium resolves coordinates.
handler = NodeValidator()
handler.apply_file("extract.pbf", locations=True)

Untagged nodes frequently act as geometric anchors for ways and relations. Retain every valid node during reconstruction even when it carries no tags; feature-extraction stages may filter the orphans afterward to shrink storage and speed up spatial joins.

2. Reconstruct way geometry Jump to heading#

A way resolves its ordered node references into coordinates, removes degenerate segments, and becomes a LineString or, when closed and area-tagged, a Polygon. Shapely requires explicit closure (first point equal to last) for a polygon ring.

from shapely.geometry import LineString, Polygon
from shapely.validation import make_valid
from shapely.errors import TopologicalError
from typing import List, Tuple, Union

def reconstruct_way_geometry(
    node_refs: List[int],
    node_index: Dict[int, Tuple[float, float]],
    is_closed: bool,
) -> Union[LineString, Polygon, None]:
    try:
        # Reference closure: keep only refs that resolved in the node index
        coords = [node_index[nid] for nid in node_refs if nid in node_index]
        missing = len(node_refs) - len(coords)
        if missing:
            logging.warning("Way dropped %d unresolved node refs", missing)
        if len(coords) < 2:
            return None

        # Remove consecutive duplicates to prevent degenerate segments
        cleaned: List[Tuple[float, float]] = [coords[0]]
        for c in coords[1:]:
            if c != cleaned[-1]:
                cleaned.append(c)
        if len(cleaned) < 2:
            return None

        if is_closed and len(cleaned) >= 3:
            if cleaned[0] != cleaned[-1]:
                cleaned.append(cleaned[0])
            geom = Polygon(cleaned)
        else:
            geom = LineString(cleaned)

        if not geom.is_valid:
            geom = make_valid(geom)
        return geom

    except KeyError as e:
        logging.warning("Missing node reference in way reconstruction: %s", e)
        return None
    except TopologicalError as e:
        logging.error("Topological failure during geometry creation: %s", e)
        return None

Reporting missing rather than silently discarding it is the difference between the corrupt-coastline scenario above and a pipeline you can trust: a way that drops references near a clip boundary is flagged, counted, and routed for review.

3. Assemble relations Jump to heading#

Relations group nodes, ways, or other relations and assign each member a role (outer, inner, stop, forward, and so on). A type=multipolygon relation builds exterior rings from outer members and subtracts inner members as holes. Member references must resolve and roles must be consistent, or the assembled geometry inverts.

from shapely.geometry import MultiPolygon, Polygon
from shapely.ops import polygonize, unary_union
from typing import Dict, List, Tuple

def assemble_multipolygon(
    members: List[Tuple[str, int, str]],          # (type, ref, role)
    way_geoms: Dict[int, "Polygon | LineString"],  # reconstructed in stage 2
) -> "MultiPolygon | None":
    outer_lines, inner_lines = [], []
    for mtype, ref, role in members:
        if mtype != "way":
            continue  # multipolygon members are ways; skip stray node/relation refs
        geom = way_geoms.get(ref)
        if geom is None:
            logging.warning("Multipolygon references unresolved way %d", ref)
            continue
        (outer_lines if role != "inner" else inner_lines).append(geom.boundary)

    if not outer_lines:
        return None

    # polygonize stitches partial ways into closed rings, then subtract holes
    outers = list(polygonize(unary_union(outer_lines)))
    holes = unary_union(inner_lines) if inner_lines else None
    rings = [p.difference(holes) if holes else p for p in outers]
    rings = [r for r in rings if not r.is_empty]
    return MultiPolygon([r for r in rings if r.geom_type == "Polygon"]) or None

Multipolygon assembly is intricate enough to warrant its own treatment; for the full handling of overlapping inner rings, disjoint outers, and ring-direction repair, see Understanding OSM multipolygon relations for GIS.

Validation & Error-Handling Matrix Jump to heading#

Each defect class has a distinct root cause, detection method, and remediation. Quarantine genuinely defective records rather than aborting the run, but treat a malformed header as a hard stop.

Performance & Scale Considerations Jump to heading#

The cost center is the node index. A planet extract holds roughly nine billion nodes; an in-memory dict[int, tuple] of that size is impossible on commodity hardware. For continental and global runs, replace the Python dictionary with an on-disk node-location store — pyosmium’s index.create_map("sparse_file_array,nodes.cache") or a memory-mapped flat array keyed by node ID. This trades a few microseconds per lookup for a flat, predictable memory ceiling instead of unbounded heap growth.

Reconstruction is embarrassingly parallel once nodes are indexed, because each way and each relation resolves independently. Split work by primitive-group block boundaries (which the binary format already aligns for you) and fan out to a process pool; the node store is read-only at this stage, so workers can share a memory-mapped view without locking. In practice a buffered, block-aligned reader sustains hundreds of thousands of ways per second per core, and the bottleneck shifts from CPU to the random-access pattern of node lookups — which is why store locality, not parser speed, dominates wall-clock time on large extracts.

Failure Modes & Gotchas Jump to heading#

• Accumulator reset boundaries. Delta-encoded coordinates and IDs reset at every primitive-group boundary, not at every blob. Carrying an accumulator across a group boundary shifts every subsequent coordinate by a constant offset that looks like a plausible translation, so it evades range checks.
• Closed ≠ area. A closed way is a ring only when an area-implying tag is present. Treating every closed way as a polygon turns roundabouts and barrier loops into spurious filled areas.
• Ring direction. OSM does not guarantee winding order. Do not infer outer versus inner from clockwise/counter-clockwise direction; trust the member role and repair winding afterward.
• Self-referencing relations. A relation may reference another relation, and pathological data can form cycles. Bound recursion depth and detect visited IDs before assembling super-relations.
• Tag index overflow. In dense PBF blocks, tag keys and values are indices into a per-block string table. An off-by-one in the 0-terminated key/value index stream silently misattributes tags to the wrong object.

Integration Points Jump to heading#

Reconstructed geometries are the input to the transformation half of the platform. Emit each feature with its primitive type, resolved geometry, normalized tags, and provenance metadata so the next stage can normalize tags and apply schema mapping without re-touching the reference graph. The wiring below hands assembled features to the normalization stage covered in Parsing & Tag Normalization Workflows.

def emit_feature(osm_type: str, osm_id: int, geom, tags: dict, meta: dict) -> dict:
    """Hand a resolved primitive to the tag-normalization stage."""
    return {
        "osm_type": osm_type,           # node | way | relation
        "osm_id": osm_id,
        "geometry": geom.wkb,           # serialized once, projected downstream
        "tags": tags,                   # normalized in the next stage
        "version": meta["version"],
        "changeset": meta["changeset"],
        "source": "© OpenStreetMap contributors",  # ODbL attribution carried forward
    }

Records that fail reference closure or topology validation should be written to a quarantine sink instead — the same contract feeds Error Handling in Large OSM Extracts, where defective records are triaged and reprocessed.

Explore This Topic Further Jump to heading#

• Understanding OSM multipolygon relations for GIS — assembling exterior boundaries and interior holes from outer/inner way members without sliver geometries or inversions.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• PBF File Structure Deep Dive — how dense nodes and delta encoding deliver these primitives to your handler.
• OSM XML vs PBF Comparison — the format trade-offs that decide your ingestion strategy.
• Coordinate Reference Systems in OSM — projecting resolved WGS 84 geometry to a working CRS.
• Tag Taxonomy & Key-Value Standards — turning resolved geometry into typed features.
• Understanding OSM multipolygon relations for GIS — full handling of outer/inner ring assembly.
• Error Handling in Large OSM Extracts — triaging the records this stage quarantines.

This guide is part of OSM Data Fundamentals & Architecture; return to that overview to follow the data model through serialization, CRS handling, and spatial indexing.