Choosing Between Overpass and a Local Extract

Almost every OSM pipeline eventually has the same argument, usually after something has already broken: should this question be answered by a live query or by a file on disk? The argument is winnable, because the decision is governed by four measurable properties rather than by taste.

The four properties that decide where an OSM question should be answered A grid of four properties against what each one implies. Frequency favours a local extract as soon as a question repeats on a schedule. Breadth favours a local extract once the answer covers a whole region rather than a bounded neighbourhood. Freshness favours a live query only when minutes matter, since extracts are daily. Reproducibility favours a local extract absolutely, because a file can be archived and a live query cannot be replayed. Four properties, and three of them point the same way Favours Overpass Favours a file Which dominates Frequency one-off scheduled frequency Breadth a neighbourhood a region breadth Freshness minutes matter daily is fine freshness Reproducibility never always reproducibility Only freshness ever argues for a live query in a scheduled job, and the replication stream answers that argument without one.
Two of these can be traded off; reproducibility cannot, which is why an auditable pipeline ends up on files regardless.

The Problem This Topic Solves

You have a question about OSM data. Overpass will answer it, and so will a filtered extract, and both look reasonable in a prototype. The decision matters because the two paths diverge sharply as the work grows: one of them degrades into rate limits and non-reproducible results, and the other degrades into storage and a parsing stack.

The concrete failure scenario is familiar by now. A prototype queries Overpass, works beautifully, and gets scheduled. Six weeks later the query is slower, then intermittently failing, then blocked; a colleague cannot reproduce last month’s numbers because the query returns different data now; and the fix — moving to a local extract — is a rewrite rather than a configuration change, because the code was shaped around a query engine’s response format. Deciding correctly at the start costs nothing; deciding late costs a rewrite.

Prerequisites

Know what Overpass actually does, from Overpass API Query Language. Know what a local extract costs to obtain and keep current, from OSM Extract Providers & Automated Downloads. And know the filtering tools available on the file side, from Choosing an OSM Parser: pyosmium, pyrosm or osmium-tool.

The Four Properties in Detail

Frequency. A question asked once is cheap anywhere. A question asked on a schedule accumulates: cost to the shared server, exposure to its availability, and — the part people forget — exposure to its changes. A file is fetched once and reused; a query is re-run and can return something different every time.

Breadth. Overpass is at its best with a tight spatial bound. As the bound widens the candidate set the server must consider grows, and past a region-sized area the query is doing work a local osmium tags-filter pass does in seconds. There is no sharp threshold, but the practical rule is that if the answer covers more than a city, a file is probably cheaper for everybody.

Freshness. This is the only property that genuinely argues for a live query. Overpass tracks the map within minutes; a published extract is daily. If your consumers need an edit made twenty minutes ago, a daily file cannot serve them. But note the alternative: a local extract kept current with replication diffs is also minutes-fresh, which removes this argument in exchange for running an update pipeline.

Reproducibility. A file is an artefact you can archive, checksum and cite. A query is a request whose answer depends on when you asked. For anything whose output will be defended later — a published figure, a regulatory report, a model’s training data — this property alone decides the question.

A decision procedure for placing one OSM question A decision node asking whether the question will run more than once, with three outcomes. A genuinely one-off question bounded to a small area belongs in Overpass, where it costs a single request and no setup. A recurring question, or one covering a region, belongs in a local extract filtered with osmium, which is reproducible and costs a shared server nothing. A recurring question that also needs minute-level freshness belongs in a local extract kept current by replication diffs, which gives both properties at the cost of an update pipeline. Ask once: will this run more than once? Once, or on a schedule? Frequency first, then freshness Breadth usually follows frequency Overpass Genuinely one-off, small area, no archive needed Local extract Recurring or region-wide; daily freshness is enough Extract plus diffs Recurring and minute-fresh; run the update pipeline The third branch is the one teams reach for last and should reach for second, because it removes the only real argument for querying.
Notice that two of the three branches end at a file — the live query survives only the genuinely one-off case.

Translating a Query into a Filter

The migration from a query to a file is more mechanical than it looks, because most production Overpass queries are one spatial bound plus one tag filter plus an output mode, and all three have direct equivalents on the file side.

A spatial bound becomes either the choice of extract (for a region-sized bound) or an osmium extract with a boundary polygon. A tag filter becomes osmium tags-filter, whose expression language covers the same key-existence, exact-value and value-set cases that dominate real queries. An output mode becomes a choice of output format and whether to resolve geometry. The one genuinely awkward case is a recursion that follows relation membership upward, which has no single-command equivalent and needs a small script.

Replacing an Overpass Query with an osmium Filter walks a real query through that translation, clause by clause.

How an Overpass query translates clause by clause into a file pipeline Four translations shown in order. A spatial bound becomes either the choice of published extract or an osmium extract against a boundary polygon. A tag filter becomes an osmium tags-filter expression covering the same key existence, exact value and value set cases. A recursion becomes either an osmium reference-completing pass or, for upward relation traversal, a short script. An output mode becomes a choice of output format and whether geometry is resolved during parsing. Four clauses, four direct equivalents spatial bound pick the extract or osmium extract tag filter osmium tags-filter same match forms recursion reference completion upward needs a script output mode format plus geometry decided at parse time Only the third clause lacks a single-command equivalent, and only in the upward direction, which most production queries never use.
The translation is this mechanical because production queries are far simpler than the language permits.

Estimating Before Committing

Both sides can be sized before you build anything, and doing so converts the argument into arithmetic.

On the query side, out count returns the element count for a query in one cheap request. Multiply by a rough bytes-per-element figure for your chosen output mode and you have the response size; compare against the query’s declared timeout and the server’s ceilings and you know whether it will finish. Estimating the Cost of an Overpass Query develops this into a repeatable procedure.

On the file side, the numbers are the extract’s download size, the storage for it and its derived outputs, and the wall-clock time of a filtering pass. All three are measurable on a small region and scale predictably.

Validation and Error Handling

Condition Root cause Detection Remediation
Scheduled query starts failing Volume grew past the shared quota Intermittent rate limits, then blocks Migrate the question to a local extract
Last month’s numbers cannot be reproduced Live query re-run against a changed map Counts differ with no code change Archive the extract and its digest, not the query
Query times out only at month end Data volume grows over time Failures correlate with the calendar Size with out count and re-check periodically
File-based pipeline is always a day behind Daily extract without replication Consumers see stale edits Attach the diff stream to the local copy
Migration turns into a rewrite Code shaped around the query response Response parsing spread through the codebase Isolate the source behind one function from day one
Local pipeline slower than the query Filtering the wrong way round Full parse before tag filtering Filter with osmium before anything reads the file

Performance and Scale

The crossover is not a single number, but its shape is consistent. Below a few queries a day over a bounded area, the live query wins on effort and costs nobody anything measurable. Above a scheduled daily run over a region, the file wins decisively — usually by more than an order of magnitude in wall-clock time, and by removing an external dependency from the critical path.

The middle ground is where judgement is needed, and the tie-breaker is almost always reproducibility. If somebody will ever ask “why did this number change”, the file is the right answer even when the query is technically adequate, because a file can be archived and re-run and a query cannot.

One structural recommendation makes the whole decision reversible: isolate the data source behind a single function that returns normalized features, and let the rest of the pipeline consume that. When the crossover arrives, you replace one implementation rather than unpicking response-shaped assumptions from across a codebase.

Failure Modes and Gotchas

  • Prototypes lie about scale. A query tested on a neighbourhood tells you nothing about the same query on a country; the cost is in the candidate set, not the result.
  • Rate limits arrive as success, then failure. Throttling degrades gradually, so a job can be “working” for weeks while quietly slowing.
  • A live query is not archivable. Storing the query text is not storing the data; the same text returns different data later.
  • Daily is not the only file cadence. Replication diffs make a local copy minute-fresh, which removes the one property that favours querying.
  • Filtering order matters on the file side too. Running osmium tags-filter before parsing is the difference between seconds and minutes.
  • Two sources means two answers. Running some questions against Overpass and others against a file of a different date produces inconsistencies nobody can explain later.

Integration Points

Whichever side you land on, the output should be the same shape: normalized features entering the workflows in Parsing & Tag Normalization Workflows. On the query side that conversion is Converting Overpass JSON to a GeoDataFrame; on the file side it is whichever parser Choosing an OSM Parser: pyosmium, pyrosm or osmium-tool recommends for your access pattern. Keeping those two converters behind one interface is what makes the migration a swap rather than a project.

Guides in This Topic

Frequently Asked Questions

Is there a hard threshold where I should switch to a file?

Not a single number, but there is a reliable trigger: the moment a question moves from being asked by a person to being asked by a scheduler. A one-off query costs a shared server nothing and needs no setup; a scheduled query accumulates cost, exposes your pipeline to an external dependency, and returns different data every run. If the question is on a timer and covers more than a neighbourhood, the file is almost always right.

What if I need data fresher than a daily extract?

Then attach the replication diff stream to your local copy rather than reaching for a live query. Replication gives a local file the same minute-level freshness Overpass has, while keeping the reproducibility and the independence from a shared server. It costs an update pipeline to operate, which is real work, but it removes the only property that genuinely favours querying on a schedule.

How do I keep the choice reversible?

Put exactly one function between your pipeline and the data source, returning normalized features rather than a service’s response shape. Every consumer talks to that function. When the crossover arrives you replace its implementation and nothing else changes. Pipelines that become expensive to migrate are the ones where response parsing leaked into a dozen call sites, and preventing that costs one interface on day one.

Can I use both, for different questions?

You can, but be careful about consistency. Answering one question from a live query and another from a file cut yesterday produces results that disagree in ways nobody can reconstruct later, especially when the two are joined. If both sources are genuinely needed, record which source and which version answered each question, and avoid joining outputs derived from different vintages of the map.

Does a private Overpass instance change the calculus?

It removes the quota argument but not the reproducibility one. A private instance answers unlimited queries against data you control, which is genuinely useful when you need the query language’s expressiveness at volume. It still returns whatever it holds at the moment you ask, so archiving a result still means archiving the data rather than the query, and you have added a database to operate. It is the right answer when the query language itself is what you need, and overkill when you only wanted the data.

Up one level: Querying OSM: Overpass, Nominatim & APIs.