Estimating the Cost of an Overpass Query

Find out how big a query’s answer is before you ask for it, using one request that costs almost nothing — and refuse to send anything the pipeline cannot handle.

Prerequisites

Conceptual minimum

Overpass offers a counting output mode. Replacing the final output statement with out count; returns a single element whose tags carry the number of nodes, ways, relations and the total — and it does so without serialising any of them. The query still has to be evaluated, so a badly shaped query can still time out while counting, but the response is a few bytes and the server does no serialisation work.

That count is the input to a simple size model. Each element costs a roughly predictable number of bytes in the response, and the multiplier depends almost entirely on the output mode:

  • out ids is a handful of bytes per element.
  • out tags adds the tag payload, which for typical OSM features averages a couple of hundred bytes.
  • out center adds one coordinate pair per way and relation.
  • out geom inlines every coordinate of every way, which for a road network is the dominant term by a wide margin — a way with forty nodes carries forty coordinate pairs, and a shared junction node appears once per way that uses it.
Approximate response bytes per element under each Overpass output mode Five output modes ranked by approximate bytes per returned element. Identifier-only output is the smallest at a few tens of bytes. Tags-only output is a few hundred bytes for a typical feature. Centre output adds a coordinate pair to that. Metadata output adds version, timestamp, changeset and user fields. Full geometry output is by far the largest because it inlines every coordinate of every way, with shared nodes repeated once per way that references them. The output mode decides the payload, not the filter out ids tens of bytes out tags a few hundred out center plus a coordinate out meta plus provenance out geom every coordinate On a dense street network the last row can be twenty times the second, which is why the mode is checked before the filter is tuned.
Changing one word in the output statement is the cheapest order-of-magnitude saving available in Overpass.

The model does not need to be accurate. It needs to distinguish “a few megabytes” from “several gigabytes”, and a rough per-element figure does that reliably.

Runnable solution

python 
from __future__ import annotations

import logging
import re
from dataclasses import dataclass

import requests

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.overpass.estimate")

ENDPOINT = "https://overpass-api.de/api/interpreter"
HEADERS = {"User-Agent": "osm-pipeline-example/1.0 (contact@example.org)"}

# Rough bytes per element, by output mode. Deliberately generous.
PER_ELEMENT = {
    "ids": 30,
    "tags": 260,
    "center": 300,
    "meta": 400,
    "geom": 1400,          # dominated by inlined way coordinates
}

_OUT_RE = re.compile(r"^\s*\.?\w*\s*out\b[^;]*;", re.M)


class QueryTooLarge(RuntimeError):
    """The estimated response exceeds what this pipeline will accept."""


@dataclass(frozen=True)
class Estimate:
    nodes: int
    ways: int
    relations: int
    total: int
    mode: str

    @property
    def bytes(self) -> int:
        # Geometry cost falls on ways and relations; nodes are cheap everywhere.
        if self.mode == "geom":
            return (self.nodes * PER_ELEMENT["tags"]
                    + (self.ways + self.relations) * PER_ELEMENT["geom"])
        return self.total * PER_ELEMENT[self.mode]


def to_count_query(query: str) -> str:
    """Replace the final output statement with a counting one."""
    if not _OUT_RE.search(query):
        raise ValueError("query has no out statement to replace")
    # Replace only the LAST out statement; earlier ones may be intermediate probes.
    matches = list(_OUT_RE.finditer(query))
    last = matches[-1]
    return query[:last.start()] + "\nout count;" + query[last.end():]


def count(query: str) -> dict[str, int]:
    response = requests.post(ENDPOINT, data={"data": to_count_query(query)},
                             headers=HEADERS, timeout=180)
    response.raise_for_status()
    elements = response.json().get("elements", [])
    if not elements:
        return {"nodes": 0, "ways": 0, "relations": 0, "total": 0}
    tags = elements[0].get("tags", {})
    return {k: int(tags.get(k, 0))
            for k in ("nodes", "ways", "relations", "total")}


def estimate(query: str, mode: str) -> Estimate:
    counts = count(query)
    result = Estimate(counts["nodes"], counts["ways"], counts["relations"],
                      counts["total"], mode)
    logger.info("%d element(s) (%dn/%dw/%dr), est. %.1f MiB in %s mode",
                result.total, result.nodes, result.ways, result.relations,
                result.bytes / (1 << 20), mode)
    return result


def run_if_affordable(query: str, mode: str, ceiling_bytes: int) -> dict:
    """Refuse to send a query whose answer we already know we cannot handle."""
    projected = estimate(query, mode)
    if projected.bytes > ceiling_bytes:
        raise QueryTooLarge(
            f"estimated {projected.bytes / (1 << 20):.0f} MiB exceeds the "
            f"{ceiling_bytes / (1 << 20):.0f} MiB ceiling — narrow the query")
    response = requests.post(ENDPOINT, data={"data": query},
                             headers=HEADERS, timeout=600)
    response.raise_for_status()
    return response.json()


if __name__ == "__main__":
    q = ('[out:json][timeout:120];'
         'way["highway"](50.0,19.8,50.2,20.2);'
         'out geom;')
    try:
        run_if_affordable(q, mode="geom", ceiling_bytes=64 << 20)
    except QueryTooLarge as exc:
        logger.error("refused: %s", exc)

Step-by-step walkthrough

  1. Rewrite the output statement, not the query. to_count_query replaces only the last out, leaving filters, set bindings and intermediate probes intact. Estimating a different query than the one you will run is worse than not estimating.
  2. Replace the last statement, not the first. A query with intermediate out calls for debugging would otherwise be truncated at the wrong point.
  3. Read the counts per type. Nodes, ways and relations are separated because geometry cost falls almost entirely on ways and relations, and a query dominated by untagged nodes has a very different profile from one dominated by roads.
  4. Model geometry separately. In geom mode the estimate applies the large per-element figure only to ways and relations, which is what makes the number useful rather than uniformly pessimistic.
  5. Be generous with the constants. The figures err high deliberately. An estimate that occasionally refuses a query you could have run is a minor annoyance; one that lets through a response that exhausts memory is an incident.
  6. Refuse, do not warn. run_if_affordable raises rather than logging, because the whole value of the estimate is that it prevents the request.
  7. Name the remedy in the message. The exception says “narrow the query”, because the person reading it at three in the morning needs the next action, not just the number.
Four steps between writing a query and sending it Four steps. The rewrite step swaps the final output statement for a counting one, leaving every filter and set binding untouched. The probe step sends the counting query, which is one small request that does no serialisation on the server. The model step multiplies the per-type counts by generous per-element byte figures chosen for the output mode that will actually be used. The decide step compares the projection against an explicit ceiling and refuses to send anything above it. Probe, model, decide — then send rewrite swap the last out filters untouched probe one small request no serialisation model counts times bytes per output mode decide compare to a ceiling refuse, do not warn The probe still evaluates the query, so a query too expensive to count is already a query too expensive to run.
A counting probe that times out has told you the answer just as clearly as one that returns a number.
What a counting probe tells you in each of its three possible outcomes Three panels covering the outcomes of a counting probe. A small count means the query is well shaped and the full request is safe to send in any output mode. A large count means the filters are correct but the answer is bigger than the pipeline can hold, so either the output mode must be reduced or the spatial bound tightened. A probe that times out means the query is expensive to evaluate rather than merely to serialise, so no output mode will rescue it and the work belongs in a local extract. Three outcomes, three different next actions Small count Filters are well shaped Any output mode is affordable Send the real query No further action needed Large count Filters are correct Answer exceeds the ceiling Try a cheaper output mode Or tighten the spatial bound Probe times out Expensive to evaluate Not merely to serialise No output mode rescues it Move the work to a file The third outcome is the most informative: it identifies a query that would have failed however patiently the client waited.
Each outcome names a different fix, which is why the probe is worth running even when you expect the first one.

Verification

  • The counting query returns the same filters. Diff the rewritten query against the original; only the final output statement should differ.
  • The estimate brackets reality. Run a query for real and compare the actual response size against the projection; the estimate should be within a factor of two and should err high.
  • Geometry mode is visibly more expensive. Estimate the same query in center and geom modes; the ratio should be large on a way-heavy query and small on a node-heavy one.
  • The ceiling actually refuses. Set a deliberately low ceiling and confirm no request for the full query is made.
  • A counting probe that times out is handled. Point the estimator at a continent-wide query; it should surface the timeout as a refusal rather than an unhandled exception.

Common errors and fixes

Symptom Root cause One-line fix
Estimate for the wrong query Whole query rebuilt instead of the output swapped Replace only the final output statement
Counting probe also times out Query is expensive to evaluate, not just to serialise Narrow spatially; the probe has already answered you
Estimate wildly low geom modelled with a flat per-element figure Apply the geometry figure to ways and relations only
Estimate ignored Projection logged rather than enforced Raise on exceeding the ceiling
Probe returns zero A set was overwritten before the output statement Bind sets explicitly and count the intended one
Ceiling never triggers Ceiling set larger than available memory Derive the ceiling from the smallest worker’s memory

Specification reference

The out count; statement returns a single result element whose tags carry the number of nodes, ways, relations and areas in the set being printed, without serialising the elements themselves. The query is still executed, so its declared timeout and memory ceiling still apply. See the Overpass QL output statement documentation for the counting mode and the other output modifiers.

Frequently Asked Questions

Does a counting query cost the server nothing?

It costs the evaluation but not the serialisation, which is usually the smaller half for a large result and the larger half for an expensive filter. That means a counting probe is cheap for the common case of a well-shaped query returning a lot of data, and no cheaper than the real thing for a badly shaped query that scans too much. Usefully, that second case is itself the answer: a probe that times out tells you the query needs narrowing.

How accurate do the per-element byte figures need to be?

Not very. The purpose is to distinguish a few megabytes from a few gigabytes, and any figure within a factor of two does that. Choose values that err high, so the estimator occasionally refuses something you could have run rather than occasionally admitting something that exhausts memory. Measure a handful of real responses against the projection once, adjust the constants, and leave them alone.

Why separate nodes from ways in the estimate?

Because geometry cost is not uniform across element types. In full-geometry mode a way carries one coordinate pair per node it references, so a road with forty nodes is roughly forty times the size of a tagged point. A query returning a million untagged nodes and a query returning a million ways have wildly different payloads, and a single per-element figure cannot express that.

Should the estimate refuse, or just warn?

Refuse. A warning is written to a log that nobody reads on a run that succeeded, and the failure it was warning about — a response that exhausts the worker’s memory — arrives minutes later as something much harder to diagnose. Raising an exception that names the projected size and tells the reader to narrow the query converts a mysterious out-of-memory kill into a clear, immediate message.

Up one level: Choosing Between Overpass and a Local Extract.