Handling Overpass Timeouts and Rate Limits

Turn a client that dies on the first HTTP 429 into one that rides a shared server’s throttle, caches what it already fetched, and refuses to materialise a response too large to hold.

Prerequisites

Conceptual minimum

Overpass public instances allocate a small number of concurrent execution slots per client address, plus a rolling budget of execution time. When you exceed either, the server responds with HTTP 429 and, on most deployments, a body describing when a slot frees up. Separately, every query declares a timeout in its settings block; exceed it and the request ends as a gateway timeout, typically HTTP 504.

These two failures mean opposite things and need opposite responses. A 429 is about you — you are asking too often — and the correct response is to slow down and keep the work you already have. A 504 is about the query — it is too expensive to finish — and the correct response is to change the query, because retrying an identical query that just failed to complete will fail again in exactly the same way. A client that treats both as “retry later” will retry its way into a block on one and into an infinite loop on the other.

How a client should respond to each Overpass failure status Three panels naming the right response to each status. A 429 means the client is asking too often, so the fix is to wait the interval the server named, halve concurrency and keep every result already fetched. A 504 means the query itself could not finish in its declared timeout, so retrying unchanged is pointless and the query must be narrowed spatially or split. A runtime error about memory means the candidate set exceeded the declared maxsize, so the filters must select fewer elements rather than the ceiling being raised. Three statuses, three different fixes HTTP 429 Meaning: you are asking too often Cause: slots or time budget spent Wait the interval the server named Halve concurrency for the run Keep everything already fetched Retrying now escalates to a block HTTP 504 Meaning: the query cannot finish Cause: too many candidate elements Retrying unchanged fails identically Narrow the spatial filter first Split by area or by tag Raising timeout only delays it Out of memory Meaning: candidate set too large Cause: filters select too broadly Reported as a runtime error Reduce elements, not the ceiling Add a tag filter before the area maxsize is a guard, not a budget Only the first of these three is worth retrying automatically; the other two are bugs in the query that a retry loop will hide.
Blanket retry logic turns two query bugs into a silent loop, which is why the status has to steer the handler.

Runnable solution

python 
from __future__ import annotations

import hashlib
import json
import logging
import random
import threading
import time
from pathlib import Path

import requests

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.overpass.client")

ENDPOINT = "https://overpass-api.de/api/interpreter"
USER_AGENT = "osm-pipeline-example/1.0 (contact@example.org)"
MAX_BYTES = 64 * 1024 * 1024          # refuse to materialise more than this
MAX_ATTEMPTS = 5


class QueryTooExpensive(RuntimeError):
    """The query itself cannot finish — retrying it unchanged will not help."""


class ResponseTooLarge(RuntimeError):
    """The response exceeded the client's hard size ceiling."""


class OverpassClient:
    def __init__(self, cache_dir: Path, max_concurrency: int = 1) -> None:
        self.cache_dir = cache_dir
        self.cache_dir.mkdir(parents=True, exist_ok=True)
        # ONE limiter for the whole process: adding workers must not multiply
        # the request rate against a shared public server.
        self._slots = threading.Semaphore(max_concurrency)
        self._session = requests.Session()
        self._session.headers["User-Agent"] = USER_AGENT

    def _cache_path(self, query: str) -> Path:
        digest = hashlib.sha256(query.encode("utf-8")).hexdigest()[:32]
        return self.cache_dir / f"{digest}.json"

    def _read_cache(self, query: str) -> dict | None:
        path = self._cache_path(query)
        if not path.exists():
            return None
        logger.info("cache hit for %s", path.name)
        return json.loads(path.read_text(encoding="utf-8"))

    def _write_cache(self, query: str, payload: dict) -> None:
        tmp = self._cache_path(query).with_suffix(".tmp")
        tmp.write_text(json.dumps(payload), encoding="utf-8")
        tmp.replace(self._cache_path(query))   # atomic: never a half-written cache

    def _stream_json(self, response: requests.Response) -> dict:
        """Read the body with a hard ceiling so a mis-scoped query fails fast."""
        chunks: list[bytes] = []
        total = 0
        for chunk in response.iter_content(chunk_size=1 << 16):
            total += len(chunk)
            if total > MAX_BYTES:
                response.close()
                raise ResponseTooLarge(f"response exceeded {MAX_BYTES} bytes")
            chunks.append(chunk)
        return json.loads(b"".join(chunks))

    def run(self, query: str, *, use_cache: bool = True) -> dict:
        if use_cache:
            cached = self._read_cache(query)
            if cached is not None:
                return cached

        delay = 2.0
        for attempt in range(1, MAX_ATTEMPTS + 1):
            with self._slots:
                started = time.monotonic()
                response = self._session.post(
                    ENDPOINT, data={"data": query}, timeout=300, stream=True)

            if response.status_code == 200:
                payload = self._stream_json(response)
                elapsed = time.monotonic() - started
                logger.info("ok in %.1fs, %d element(s)",
                            elapsed, len(payload.get("elements", [])))
                if use_cache:
                    self._write_cache(query, payload)
                return payload

            if response.status_code in (429, 503):
                wait = float(response.headers.get("Retry-After", delay))
                wait += random.uniform(0, wait * 0.25)   # jitter: never in lockstep
                logger.warning("throttled (%s), sleeping %.1fs (attempt %d/%d)",
                               response.status_code, wait, attempt, MAX_ATTEMPTS)
                time.sleep(wait)
                delay = min(delay * 2, 120)
                continue

            if response.status_code in (504, 400):
                # 504: the query ran out of time. 400 usually carries a runtime
                # error such as "Query run out of memory". Neither is retryable.
                raise QueryTooExpensive(
                    f"HTTP {response.status_code}: narrow the query, do not retry it")

            response.raise_for_status()

        raise RuntimeError(f"gave up after {MAX_ATTEMPTS} throttled attempts")


if __name__ == "__main__":
    client = OverpassClient(Path(".overpass-cache"))
    result = client.run(
        "[out:json][timeout:60];"
        'node["amenity"="drinking_water"](50.02,19.87,50.10,20.05);'
        "out center tags;"
    )
    logger.info("%d element(s)", len(result["elements"]))

Step-by-step walkthrough

  1. Cache first, ask second. The cache key is a hash of the exact query text, so an identical query never reaches the network twice. In development this removes the large majority of requests, because reruns dominate.
  2. Write the cache atomically. The payload lands in a temporary file and is renamed into place, so an interrupted run leaves either the old cache entry or the new one, never a truncated file a later run would parse as valid.
  3. One semaphore for the process. The limiter is an instance attribute shared by every caller, so adding threads increases parallel work without increasing the request rate against the shared endpoint.
  4. Stream with a ceiling. The body is read in chunks and abandoned the moment it crosses the byte limit, so a query that accidentally selects a country fails in seconds rather than exhausting memory minutes later.
  5. Honour Retry-After. When the server states how long to wait, that value is used verbatim; the exponential fallback only applies when the header is absent.
  6. Add jitter. A quarter of the wait is randomised so a fleet of workers throttled at the same moment does not retry in a synchronised wave, which is what turns a throttle into a block.
  7. Refuse to retry a query bug. A 504 or a memory runtime error raises immediately with a message that names the fix, because the same query will fail the same way however long you wait.
  8. Bound the attempts. Five throttled attempts with doubling delay is roughly two minutes of patience; past that, surfacing the failure is more useful than looping.
How the backoff delay grows across five throttled attempts A timeline of five attempts. The first request is throttled and the client waits about two seconds plus jitter. The second waits about four seconds. The third waits about eight. The fourth waits about sixteen, and by then concurrency has been halved for the rest of the run. After the fifth the client gives up and raises, because a throttle that has not cleared in two minutes is a capacity problem rather than a transient one. Doubling delay, capped patience attempt 1 throttled wait about 2s attempt 2 still throttled wait about 4s attempt 3 still throttled wait about 8s give up raise, do not loop about 2 minutes total Unbounded retry is indistinguishable from an attack from the server's side, and it hides a capacity problem from yours.
Bounded patience makes the failure visible to whoever scheduled the job, which is the person who can actually fix it.

Verification

  • A second identical run makes no network call. Watch the log: the second invocation should report a cache hit and finish in milliseconds.
  • An interrupted run leaves no partial cache. Kill the process mid-download and rerun; the query should be re-fetched cleanly, not parsed from a truncated file.
  • The size guard trips. Point the client at a deliberately unbounded query and confirm it raises ResponseTooLarge within seconds rather than consuming memory.
  • A 504 does not retry. Submit a query with an absurdly low declared timeout; the client should raise QueryTooExpensive immediately rather than sleeping.
  • Concurrency is actually capped. Start ten threads against the client and count concurrent requests at the endpoint; the count must never exceed the configured limit.
The four layers of a well-behaved Overpass client and what each one removes Four stacked layers. The cache layer removes repeat traffic entirely and is the single largest reduction in most projects. The limiter layer caps concurrent requests at one shared value so worker count and request rate stay independent. The backoff layer converts a throttle into a delay instead of an escalation. The guard layer refuses oversized responses and unretryable statuses so a bad query fails in seconds rather than minutes. Each layer removes a different kind of waste Cache Serve an identical query from disk removes repeat traffic Limiter One semaphore for the whole process decouples workers from rate Backoff Honour Retry-After, add jitter turns a throttle into a delay Guards Size ceiling and unretryable statuses fails a bad query in seconds Built in this order, each layer makes the one below it matter less — which is why the cache is worth writing before the backoff.
Most teams write the backoff first and the cache last, and then wonder why the backoff is doing so much work.

Common errors and fixes

Symptom Root cause One-line fix
Repeated 429 despite backoff A limiter per worker instead of per process Share one semaphore instance across all callers
Client hangs forever No request timeout set Always pass an explicit timeout= to the HTTP call
Memory exhausted on a big result Body read in one call Stream in chunks and enforce a byte ceiling
Cache never hits Query text differs by whitespace between runs Normalise the query string before hashing
Corrupt cache entry after a crash Cache written in place Write to a temporary file and rename atomically
Retry loop never ends 504 treated as retryable Raise on 504; narrow the query instead
Blocked despite polite backoff No identifying User-Agent Set an agent naming the project and a contact

Specification reference

Public Overpass instances enforce a per-client limit on concurrent query slots and on cumulative execution time, and respond with HTTP 429 when either is exceeded; the [timeout:n] and [maxsize:n] settings declared in a query are upper bounds enforced by the server, not resource reservations. The Overpass API usage policy and commonly used limits describe the slot model and the expectation that clients identify themselves and back off rather than retry immediately.

Frequently Asked Questions

Should I retry a 504 the way I retry a 429?

No. A 429 means you are asking too often and the same query will succeed once a slot frees up, so waiting is the correct response. A 504 means the query could not finish within its declared timeout, and an identical query submitted later will fail in exactly the same way. Treat a 504 as a signal to narrow the spatial filter, split the work by area or tag, or move to a local extract — never as something to sleep through.

Does raising maxsize fix an out-of-memory runtime error?

Only by letting a query that selects far too many elements run for longer before failing, usually on a server that other people are also using. The setting is a guard against a mis-scoped query, not a budget to spend. The real fix is to make the candidate set smaller: apply the spatial filter before the tag filters, replace a key-existence test with exact values, and check the element count with a cheap counting query before asking for any geometry.

How much should I cache, and for how long?

Cache every successful response keyed on the exact query text, and expire on a schedule that matches how fresh your consumers actually need the data. For development and test runs the cache can be effectively permanent, because reproducibility matters more than freshness. For scheduled production jobs an expiry matched to the job’s cadence is usually right, and a cache that is never invalidated at all is a stale-data incident waiting to happen.

Why add jitter if the server tells me exactly how long to wait?

Because every client it throttled at the same moment was told the same interval. Without jitter a fleet of workers wakes up simultaneously and delivers a synchronised burst, which is more likely to be read as abuse than the original traffic was. A random fraction of the interval spreads the retries out at no cost to any single client’s latency.

Up one level: Overpass API Query Language.