A technical reference for engineers building and operating production OpenStreetMap data pipelines. Cover the full ETL surface — from PBF and XML parsing, through tag normalization and topology cleaning, all the way to routing graph extraction, diff-based updates, and automated validation rules.
Every page is written for mapping engineers, OSM contributors, GIS analysts, and Python ETL developers who need deterministic, reproducible workflows at continental scale. Expect deep binary-format dives, memory-aware streaming patterns, and rule-driven QA you can wire into Dask, Ray, or plain old multiprocessing.
The content spans four tracks: a data fundamentals track covering the OSM schema, serialization formats and extract clipping; a parsing and normalization track covering ingestion, routing-graph conversion and export to GeoParquet and PostGIS; a replication and diff-sync track for keeping local data current with upstream change files and monitoring how far behind you are; and a data quality and validation track covering rule-driven QA and changeset analysis.
88 in-depth pages across four sections — every one carrying worked code, hand-drawn diagrams and a light and dark reading mode.
Schema, serialization, spatial indexing, tag taxonomy, and licensing foundations for production-grade OSM pipelines.
Async PBF parsing, parser selection, batch attribute mapping, regex-driven value cleaning, error handling, and routing-graph conversion.
Keep local extracts current with .osc.gz change files, replication sequence numbers, full-history .osh.pbf, and automated minutely update pipelines.
Author JOSM, Osmose, and Python validation rules; repair geometry; check routing-graph topology and tag consistency across the pipeline.
Byte-level dissection of the PBF container, block framing, StringTable deduplication, and delta encoding for nodes, ways, and relations.
Async ingestion with Pyrosm, chunked memory-aware processing, regex-driven value standardization, and reproducible schema alignment.
OSMnx graph conversion, turn restrictions, edge contraction, one-way handling, and accessibility constraints for routing engines.
Error isolation, structured logging, quarantine workflows, automated tag taxonomy validation, and ODbL compliance checks.
Cutting regions out of a planet file without shredding the reference graph: .poly boundaries, the four osmium strategies, and one-pass multi-region splits.
Landing normalised features where consumers read them: GeoParquet schema and partitioning, osm2pgsql flex tables, and keeping the two sinks consistent.
Replication lag signals and alert thresholds that survive a normal day, plus changeset scoring that ranks suspicious edits for a human rather than reverting them.