Replacing an Overpass Query with an osmium Filter

Take a query that has outgrown the shared server and turn it into a local pipeline that answers the same question in seconds — then prove the two agree before you delete the old code.

Prerequisites

Conceptual minimum

A production Overpass query is almost always four things: a spatial bound, a tag filter, sometimes a recursion, and an output mode. Each maps onto a local equivalent.

The spatial bound becomes osmium extract, either with --bbox for a rectangle or --polygon for a boundary file. The tag filter becomes osmium tags-filter, whose expression syntax covers key existence (n/amenity), exact values (w/highway=residential) and value sets (w/highway=primary,secondary), with the leading letter selecting node, way, relation or any. The recursion becomes a flag rather than an operator: osmium tags-filter keeps referenced nodes by default so filtered ways remain drawable, which is the downward recursion most queries use. The output mode becomes a choice of output format and whether your reader resolves geometry.

Overpass clauses and their osmium equivalents A grid mapping four Overpass constructs onto local commands. A bounding box filter becomes osmium extract with a bounding box argument. An area filter becomes osmium extract with a polygon boundary file. A tag filter becomes an osmium tags-filter expression using the same key existence, exact value and value set forms. A downward recursion becomes the default reference-completing behaviour of tags-filter, while an upward recursion has no direct equivalent and needs a short script. Four constructs, four local equivalents osmium equivalent Notes Bounding box filter osmium extract --bbox exact equivalent Area filter osmium extract --polygon needs a .poly file Tag filter osmium tags-filter same match forms Downward recursion default in tags-filter references kept Upward recursion no direct command short script needed Only the last row lacks a one-command translation, and it appears in a small minority of production queries.
The mapping is this clean because osmium and Overpass are filtering the same element model with the same vocabulary.

The one real asymmetry is ordering. Overpass evaluates a query as a whole and the server decides how to execute it. A local pipeline is a chain of passes, and you choose the order — which means you can get it badly wrong. Filtering by tag before cutting spatially means reading and rewriting the whole region for features you are about to discard; cutting spatially first shrinks the input for every later pass.

Runnable solution

bash 
#!/usr/bin/env bash
# Replace:
#   [out:json][timeout:180];
#   area["name"="Kraków"]["admin_level"="8"]->.a;
#   ( node["amenity"="pharmacy"](area.a);
#     way["amenity"="pharmacy"](area.a); );
#   out center tags;
set -euo pipefail

EXTRACT="/data/poland-latest.osm.pbf"
POLY="/data/krakow.poly"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT

# 1. Spatial bound FIRST: every later pass reads a far smaller file.
osmium extract --polygon "$POLY" --strategy=complete_ways \
  --output "$WORK/area.osm.pbf" "$EXTRACT"

# 2. Tag filter. Referenced nodes are kept by default, so the matched ways
#    remain drawable — this is the downward recursion the query relied on.
osmium tags-filter --output "$WORK/pharmacies.osm.pbf" \
  "$WORK/area.osm.pbf" n/amenity=pharmacy w/amenity=pharmacy r/amenity=pharmacy

# 3. Output. GeoJSON with centroids is the local analogue of `out center`.
osmium export --output-format=geojsonseq --add-unique-id=type_id \
  --output "$WORK/pharmacies.geojsonseq" "$WORK/pharmacies.osm.pbf"

wc -l < "$WORK/pharmacies.geojsonseq"
cp "$WORK/pharmacies.geojsonseq" ./pharmacies.geojsonseq

Reconciliation is the step that makes the switch safe:

python 
from __future__ import annotations

import json
import logging
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.migrate.reconcile")


def keys_from_overpass(path: Path) -> set[str]:
    payload = json.loads(path.read_text(encoding="utf-8"))
    return {f"{el['type']}/{el['id']}" for el in payload["elements"]
            if el.get("tags")}


def keys_from_geojsonseq(path: Path) -> set[str]:
    keys: set[str] = set()
    for line in path.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        feature = json.loads(line)
        # --add-unique-id=type_id emits ids like "n123" / "w456".
        raw = str(feature.get("id", ""))
        prefix = {"n": "node", "w": "way", "r": "relation"}.get(raw[:1])
        if prefix:
            keys.add(f"{prefix}/{raw[1:]}")
    return keys


def reconcile(overpass: Path, local: Path) -> bool:
    a, b = keys_from_overpass(overpass), keys_from_geojsonseq(local)
    only_remote, only_local = a - b, b - a
    logger.info("overpass %d, local %d, shared %d", len(a), len(b), len(a & b))
    for label, missing in (("only in Overpass", only_remote),
                           ("only in local", only_local)):
        if missing:
            logger.warning("%d %s, e.g. %s", len(missing), label,
                           sorted(missing)[:5])
    # A handful of differences is expected: the two sources are different dates.
    drift = len(only_remote | only_local) / max(len(a | b), 1)
    logger.info("symmetric difference %.2f%%", drift * 100)
    return drift < 0.02


if __name__ == "__main__":
    ok = reconcile(Path("overpass_archive.json"), Path("pharmacies.geojsonseq"))
    logger.info("reconciliation %s", "PASSED" if ok else "FAILED")

Step-by-step walkthrough

  1. Cut spatially first. The extract pass runs over the country file once; every subsequent pass reads a city-sized file. Reversing this order costs minutes per run forever.
  2. Choose the cut strategy deliberately. complete_ways keeps ways that cross the boundary intact along with their outside nodes, matching what an Overpass area filter effectively gives you for drawable geometry.
  3. Name all three element types. The original query unioned nodes and ways; the filter lists node, way and relation prefixes explicitly, which is both clearer and cheaper than a catch-all.
  4. Rely on reference completion, not a separate pass. tags-filter keeps the nodes a matched way references, so the output is drawable without a second step. That is the > recursion, done by default.
  5. Pick an output analogous to the query’s mode. osmium export with centroids corresponds to out center; exporting full geometry corresponds to out geom. Choosing the wrong one changes the file size by an order of magnitude, exactly as it does server-side.
  6. Emit stable ids. --add-unique-id=type_id gives each feature a type-prefixed identifier, which is what makes the reconciliation below possible and what downstream joins need anyway.
  7. Reconcile against an archived response. The two sources are different dates, so a small symmetric difference is expected and healthy. A large one means a clause was translated wrongly, and the sample identifiers in the log point straight at which.
  8. Set the drift threshold explicitly. Two percent is a starting point for a daily extract against a live query; tighten it if your extract is fresh and loosen it if it is a week old.
The order of passes in a local replacement and why it is not negotiable Four passes in order. The spatial cut runs once over the full regional file and produces a city-sized file, which every later pass reads instead. The tag filter runs over that smaller file and keeps referenced nodes so matched ways stay drawable. The export converts the filtered file to the output format with stable type-prefixed identifiers. The reconcile step compares the identifier set against an archived response from the query being replaced. A note warns that swapping the first two passes makes every run read the whole region. Cut, filter, export, reconcile — in that order cut spatial bound first once over the region filter tags on a small file references kept export format plus stable ids mode matches out reconcile against an archive threshold, not zero Filtering before cutting reads and rewrites the entire region for features that are about to be discarded anyway.
The server chose this order for you in Overpass; locally it is your decision, and it is the one that dominates runtime.
Relative wall-clock cost of the two possible pass orderings on a country extract Four measurements comparing two orderings on a country-sized input. Cutting spatially and then filtering by tag spends most of its time in the single cut pass and almost none in the filter, because the filter reads a city-sized file. Filtering by tag and then cutting spends a long time in the filter, which reads and rewrites the entire country, and then a short time in the cut. The total for the second ordering is several times the first. Same two passes, several times the runtime Cut first: the cut reads the country Cut first: the filter reads a city Filter first: the filter rewrites it all Filter first: the cut reads the rewrite The second ordering pays the country-sized read twice and writes a country-sized intermediate file nobody ever looks at.
Nothing about the result differs between the two orderings — only the time, and the disk the intermediate file consumes.

Verification

  • The symmetric difference is small. A few percent between a daily extract and a live query is normal; twenty percent means a clause is wrong.
  • Differences are dated, not structural. Sample identifiers that appear only in the live response should be recently created objects, not a whole feature class.
  • The spatial cut runs once. Time both orderings on a country extract; cutting first should be several times faster overall.
  • Matched ways are drawable. Open the filtered output and confirm ways have coordinates rather than dangling references.
  • Identifiers are type-prefixed. A numeric id alone collides across element types and will silently merge unrelated features in a later join.

Common errors and fixes

Symptom Root cause One-line fix
Pipeline slower than the query Tag filter runs before the spatial cut Cut spatially first, always
Ways have no geometry Reference completion disabled Let tags-filter keep referenced nodes by default
Whole feature class missing One element prefix omitted from the filter List node, way and relation prefixes explicitly
Roads end at the boundary Simple cut strategy used Cut with complete_ways when geometry crosses the edge
Reconciliation shows huge drift Area filter translated to a loose bounding box Use a .poly boundary matching the original area
Duplicate features after a join Numeric ids without a type prefix Export with type-prefixed unique identifiers
Reconciliation never passes Threshold set to exact equality Expect drift; the two sources are different dates

Specification reference

osmium tags-filter selects objects by expressions of the form [nwr]/key=value, where the leading characters restrict the object type and the value part may be omitted to match any value or given as a comma-separated list. By default the command also outputs the nodes referenced by matched ways and the members of matched relations, so that geometry can be reconstructed. See the osmium-tool documentation for the full expression grammar and the reference-completion options.

Frequently Asked Questions

Why does the order of passes matter so much?

Because each pass reads its whole input. Cutting spatially first means the expensive tag pass runs over a city-sized file; doing it the other way round means the tag pass reads and rewrites an entire country before the spatial cut discards most of it. Overpass hides this decision behind a query planner, so people migrating a query often do not realise they have inherited responsibility for it.

What replaces an upward recursion?

There is no single command for it, because finding the relations that reference a set of ways requires a pass over relations that you then intersect with your set. In practice it is a short script: read the relations, keep those with a member in your identifier set, then re-run reference completion so their members are present. It appears in a small minority of production queries, which is why the rest of the translation is as mechanical as it is.

How much difference between the two results is acceptable?

Enough to account for the age gap and no more. A daily extract compared against a live query will differ by whatever was edited in between, which for a city-sized area of interest is typically well under a couple of percent. What matters is the shape of the difference: objects that exist only in the live response should be recent creations scattered across the area, not an entire feature class, which would indicate a mistranslated clause.

Do I still need a boundary polygon, or is a bounding box enough?

If the original query used an area filter, use a polygon. A bounding box around a city includes neighbouring territory, and the surplus features it admits look exactly like a translation error when you reconcile. Extract the boundary once from the same administrative relation the query named, keep it in version control alongside the pipeline, and the two definitions stay aligned.

Up one level: Choosing Between Overpass and a Local Extract.