Running a Local Overpass Instance for Bulk Queries

Stand up your own Overpass endpoint from a regional extract so a bulk workload runs at whatever rate your hardware allows, without taking slots from the shared public servers.

Prerequisites

Conceptual minimum

An Overpass instance is a purpose-built database, not a wrapper over a PBF file. The import pass reads the extract and writes a set of index structures — by element id, by tag, and by geographic cell — into a database directory. Queries are then answered from those indexes. Two consequences follow.

The first is that the import is the expensive part and it is a one-off. It is write-heavy, largely single-threaded in its later stages, and dominated by disk. Running it on a network volume or a small burst-credit disk turns hours into days. The second is that an instance is only as fresh as its last update. A freshly imported instance reflects the extract’s timestamp, and without an update loop it silently ages exactly the way a stale extract does — a failure mode covered in OSM Replication & Diff Sync.

Area filters deserve special mention. Areas are not stored in the source data; the Overpass software derives them in a separate pass over boundary relations and closed ways. That pass is optional and it is not cheap, so a minimal instance answers element queries perfectly and returns nothing for every area query. If your workload uses (area.x) filters — the ones built in Writing Overpass QL Area and Bounding Box Queries — you must enable and schedule area generation, and you must wait for its first run before those queries work.

The four stages of standing up a private Overpass instance Four stacked stages. Planning sizes the disk and memory from the extract size and decides whether updates are needed. The import reads the extract and writes the element and tag indexes, which is the longest and most disk-bound stage. Area generation is a separate optional pass that must run before any area filter works. The update loop attaches replication diffs so the instance does not age, and is what distinguishes a living instance from a snapshot. Four stages, and two of them are optional until they are not Plan Size disk and memory from the extract six to ten times the file Import Build element and tag indexes the long, disk-bound stage Areas Derive areas from boundary relations skip it and area filters return nothing Update Attach replication diffs without it the instance ages silently Teams routinely discover the third stage exists only when every area query on their new instance quietly returns zero elements.
The area pass is the stage most often skipped and the one whose absence produces no error message at all.

Runnable solution

The container images maintained by the Overpass community handle the import and the update loop together; the work is in configuring them deliberately rather than accepting defaults.

bash 
#!/usr/bin/env bash
# Stand up a private Overpass instance from a regional extract.
set -euo pipefail

EXTRACT_URL="https://download.geofabrik.de/europe/poland-latest.osm.pbf"
# The replication directory MUST match the region the extract covers.
UPDATE_URL="https://download.geofabrik.de/europe/poland-updates/"
DB_DIR="/srv/overpass/db"
META="yes"          # keep version/timestamp/user so `out meta` works
AREAS="yes"         # run the area-generation pass; needed for (area.x) filters

mkdir -p "$DB_DIR"

docker run -d --name overpass \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL="$EXTRACT_URL" \
  -e OVERPASS_DIFF_URL="$UPDATE_URL" \
  -e OVERPASS_META="$META" \
  -e OVERPASS_RULES_LOAD=10 \
  -e OVERPASS_UPDATE_SLEEP=60 \
  -e OVERPASS_ALLOW_DUPLICATE_QUERIES=yes \
  -v "$DB_DIR:/db" \
  -p 12345:80 \
  wiktorn/overpass-api

# The import runs inside the container and can take hours. Watch it:
docker logs -f overpass

Once the import finishes, verify the instance answers a query you already know the answer to:

python 
from __future__ import annotations

import logging

import requests

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.overpass.local")

LOCAL = "http://localhost:12345/api/interpreter"
PUBLIC = "https://overpass-api.de/api/interpreter"
HEADERS = {"User-Agent": "osm-pipeline-example/1.0 (contact@example.org)"}

COUNT_QUERY = (
    "[out:json][timeout:60];"
    'node["amenity"="pharmacy"](50.02,19.87,50.10,20.05);'
    "out count;"
)
AREA_QUERY = (
    "[out:json][timeout:60];"
    'area["name"="Kraków"]["admin_level"="8"]->.a;'
    'node["amenity"="pharmacy"](area.a);'
    "out count;"
)


def count(endpoint: str, query: str) -> int:
    response = requests.post(endpoint, data={"data": query},
                             headers=HEADERS, timeout=180)
    response.raise_for_status()
    tags = response.json()["elements"][0]["tags"]
    return int(tags["total"])


def compare() -> None:
    local_bbox = count(LOCAL, COUNT_QUERY)
    public_bbox = count(PUBLIC, COUNT_QUERY)
    logger.info("bbox query: local=%d public=%d", local_bbox, public_bbox)
    if abs(local_bbox - public_bbox) > max(2, public_bbox * 0.02):
        logger.warning("counts diverge by more than 2%% — check import freshness")

    local_area = count(LOCAL, AREA_QUERY)
    if local_area == 0 and public_bbox > 0:
        logger.error("area query returned nothing: area generation has not run")
    else:
        logger.info("area query: local=%d", local_area)


if __name__ == "__main__":
    compare()

Step-by-step walkthrough

  1. Match the replication directory to the extract. The update URL must serve diffs for exactly the region the extract covers. A country extract updated from a continent’s diff directory applies changes for territory the database does not contain, and the mismatch is not detected for you.
  2. Decide about metadata before importing, not after. Keeping version, timestamp and user roughly doubles the index size but is the only way out meta works. Changing your mind means a full re-import.
  3. Enable area generation explicitly. It is a separate pass with its own schedule. Without it, every (area.x) filter returns an empty set, and — as the parent topic warns — an empty area produces no error.
  4. Give the import a real disk. The import is dominated by random writes. Local NVMe turns a multi-hour import into a manageable one; a network volume can turn it into an overnight job.
  5. Set the update interval deliberately. A sixty-second sleep gives near-minutely freshness at the cost of continuous background work. If your consumers are happy with hourly data, a longer interval leaves far more capacity for queries.
  6. Compare against the public endpoint once. The verification script runs the same bounding-box query against both and warns when the counts diverge by more than a couple of percent, which is the signal that the import is stale or incomplete.
  7. Test an area query separately. A zero result from the area query while the bounding-box query returns hundreds is the unambiguous signature of area generation not having run.
Roughly how much disk each import option adds, relative to the extract size Five bars showing storage multiples relative to the source extract size. The element and tag indexes alone are about four times the extract. Adding metadata — version, timestamp and user — takes it to about seven times. Adding generated areas takes it to about eight. Reserving headroom for the update loop's working files takes the practical requirement to about ten times. A note warns that these are shapes rather than benchmarks and vary with how densely the region is mapped. Plan for roughly ten times the extract, not four Source extract 1x baseline Element and tag indexes about 4x Plus metadata about 7x Plus generated areas about 8x Plus update headroom plan for 10x Densely mapped regions land at the top of this range and sparsely mapped ones below it, so measure once on your own region.
Running out of disk part way through an import means starting the import again, which is why the headroom row is not optional.
Choosing between the public endpoint, a private instance and a local extract A decision node about sustained query volume with three outcomes. Occasional interactive queries belong on the public endpoint, where a polite client is all that is needed. Sustained bulk querying that still needs the Overpass query language belongs on a private instance, which removes the quota at the cost of operating a database. Repeated whole-region extraction that does not need ad hoc queries belongs in a local extract filtered with osmium, which needs no server at all. How much do you query, and do you need the language? Volume and query shape? Both questions, not just volume The second one decides the tool Public endpoint Occasional, interactive, a polite client is enough Private instance Sustained volume that still needs Overpass QL Local extract plus osmium Repeated whole-region filtering, no ad hoc queries Most workloads that outgrow the public endpoint turn out to belong in the third branch, not the second.
Self-hosting is the right answer only when you genuinely need the query language; otherwise it is a database to operate for nothing.

Verification

  • The bounding-box count matches the public endpoint. Within a couple of percent; a larger gap means the extract predates recent edits or the import did not complete.
  • An area query returns a non-zero count. If it returns zero while the same features are found by bounding box, area generation has not run.
  • out meta returns version fields. Query one known element with out meta and confirm version and timestamp are present; if they are not, the import discarded metadata.
  • The update loop advances. Check the instance’s replication state after an hour; the sequence number must have increased.
  • A representative workload query completes. Run the slowest query from your real workload and record its duration, so you have a baseline to compare against after the next import.

Common errors and fixes

Symptom Root cause One-line fix
Every area query returns zero Area generation never ran Enable the area pass and wait for its first completion
out meta returns no version Imported without metadata Re-import with metadata enabled; it cannot be added later
Import fails near the end Disk exhausted Provision around ten times the extract size before starting
Counts drift from the public endpoint Update loop stopped or wrong diff URL Match the replication directory to the extract’s region
Queries slow after weeks of updates Index fragmentation from continuous diffs Schedule a periodic re-import rather than updating forever
Import takes more than a day Network-attached or burst-credit disk Import on local NVMe, then move the database directory
Instance answers, results look truncated Extract covers less than the query area Query only inside the region the extract covers

Specification reference

The Overpass database is built by an import pass that writes element, tag and geographic indexes into a database directory, and areas are produced by a separate osm3s area-generation rule pass that must be scheduled independently of the main import. Replication diffs are applied by an update loop that tracks its own sequence state. See the Overpass API installation documentation for the stages, their ordering, and the metadata options fixed at import time.

Frequently Asked Questions

How much disk does a private Overpass instance actually need?

Plan for roughly ten times the source extract size once metadata, generated areas and update headroom are included. The raw element and tag indexes alone are around four times the extract, metadata adds most of the rest, and the update loop needs working space. Densely mapped regions sit at the top of that range. Running out of space part way through an import means restarting it from the beginning, so headroom is cheaper than the retry.

Why do my area filters return nothing on a fresh instance?

Because areas are derived by a separate generation pass that does not run as part of the main import. Until that pass has completed at least once, no area objects exist, and an area filter over an empty set returns an empty result with no error. Enable area generation explicitly, schedule it, and confirm a known area query returns a non-zero count before you rely on any query that uses an area filter.

Do I need minutely updates, or is a periodic re-import enough?

It depends entirely on what your consumers need. An update loop keeps the instance within minutes of the live map but runs continuously and slowly fragments the indexes, so long-lived instances get gradually slower. A weekly or monthly re-import from a fresh extract is simpler to operate, gives a predictable performance profile, and is entirely adequate when the questions being asked do not depend on very recent edits.

Can I import a country extract and query outside it?

No, and the failure is quiet. A query whose bounding box extends past the extract’s coverage returns whatever exists inside the imported region and nothing for the rest, with no indication that part of the answer is missing. Either import a region that covers every query you will make, or add an explicit guard that rejects queries whose bounds fall outside the imported area.

Up one level: Overpass API Query Language.