Converting Overpass JSON to a GeoDataFrame

Take the raw JSON an Overpass query returns and produce a GeoDataFrame whose geometry column is correct for every element type, without silently dropping the ways that arrived as node references.

Prerequisites

Conceptual minimum

An Overpass JSON response is a flat elements array, not a nested feature collection. Every entry has a type (node, way or relation), an id, and usually a tags object. What it carries for geometry depends entirely on the output mode the query asked for, and that is the only thing you need to branch on.

With out center, ways and relations carry a center object with lat and lon; nodes carry lat and lon directly. Every row becomes a Point, and the conversion is trivial. With out geom, ways carry a geometry array of coordinate objects, and relations carry a members array where each member has its own geometry. With out body, ways carry only a nodes array of node ids, so geometry has to be assembled from the node elements in the same response — which only works if the query included a recursion to fetch them.

What each Overpass output mode carries for each element type A grid showing what geometry information arrives for nodes, ways and relations under three output modes. Under out center, nodes carry latitude and longitude directly while ways and relations carry a single centre point. Under out geom, nodes carry a coordinate, ways carry a full coordinate array, and relations carry per-member coordinate arrays. Under out body, nodes carry a coordinate, ways carry only node identifiers, and relations carry only member references, so geometry must be assembled from other elements in the same response. Branch on the output mode, not on the element type out center out geom out body node lat and lon lat and lon lat and lon way one centre point coordinate array node ids only relation one centre point per-member arrays member refs only Assembly needed none none yes, from nodes A converter that guesses the mode from the payload will get it wrong on a mixed response; pass the mode in explicitly.
The bottom row is the one that decides your code path: only out body requires you to hold node coordinates in memory.

One more model detail matters: the id is not unique on its own. A node and a way can share the numeric id 12345, so the key for any table built from OSM elements is the pair (type, id). Losing that distinction is the classic source of phantom duplicates after a join, and it is the same identity problem worked through in OSM Feature Identity & ID Stability.

Runnable solution

python 
from __future__ import annotations

import logging
from typing import Any, Literal

import geopandas as gpd
import pandas as pd
from shapely.geometry import LineString, Point, Polygon

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.overpass.geodataframe")

Mode = Literal["center", "geom", "body"]
WGS84 = "EPSG:4326"


def _coords(seq: list[dict[str, float]]) -> list[tuple[float, float]]:
    """Overpass emits {'lat': .., 'lon': ..}; shapely wants (x, y) = (lon, lat)."""
    return [(p["lon"], p["lat"]) for p in seq]


def _is_area(tags: dict[str, str], ring: list[tuple[float, float]]) -> bool:
    """A closed way is an area only if its tags say so — a roundabout is not."""
    if len(ring) < 4 or ring[0] != ring[-1]:
        return False
    if tags.get("area") == "no":
        return False
    return bool(tags.keys() & {"building", "landuse", "natural", "leisure",
                               "amenity", "area", "water", "place"})


def _geometry(el: dict[str, Any], mode: Mode,
              nodes: dict[int, tuple[float, float]]) -> Any | None:
    tags = el.get("tags", {})
    if el["type"] == "node":
        return Point(el["lon"], el["lat"])

    if mode == "center":
        centre = el.get("center")
        return Point(centre["lon"], centre["lat"]) if centre else None

    if mode == "geom":
        if el["type"] == "way":
            ring = _coords(el.get("geometry", []))
            if len(ring) < 2:
                return None
            return Polygon(ring) if _is_area(tags, ring) else LineString(ring)
        # A relation under out geom carries per-member geometry; join the outer
        # members end to end only when the caller has already validated them.
        parts = [_coords(m["geometry"]) for m in el.get("members", [])
                 if m.get("geometry")]
        flat = [pt for part in parts for pt in part]
        return LineString(flat) if len(flat) >= 2 else None

    # mode == "body": ways carry node ids and nothing else.
    if el["type"] == "way":
        ring = [nodes[n] for n in el.get("nodes", []) if n in nodes]
        if len(ring) < 2:
            logger.warning("way %s: %d/%d nodes present — did the query recurse?",
                           el["id"], len(ring), len(el.get("nodes", [])))
            return None
        return Polygon(ring) if _is_area(tags, ring) else LineString(ring)
    return None


def to_geodataframe(payload: dict[str, Any], mode: Mode) -> gpd.GeoDataFrame:
    """Convert one Overpass JSON response into a GeoDataFrame."""
    elements = payload.get("elements", [])
    # Node coordinates are needed only for body mode, but building the map is cheap.
    nodes = {el["id"]: (el["lon"], el["lat"])
             for el in elements if el["type"] == "node" and "lon" in el}

    rows: list[dict[str, Any]] = []
    geoms: list[Any] = []
    skipped = 0
    for el in elements:
        # Untagged nodes exist only to give ways their shape; they are not features.
        if el["type"] == "node" and not el.get("tags"):
            continue
        geom = _geometry(el, mode, nodes)
        if geom is None or geom.is_empty:
            skipped += 1
            continue
        row: dict[str, Any] = {
            "osm_type": el["type"],
            "osm_id": el["id"],
            "osm_key": f"{el['type']}/{el['id']}",   # the only unique key
        }
        row.update(el.get("tags", {}))
        rows.append(row)
        geoms.append(geom)

    frame = gpd.GeoDataFrame(pd.DataFrame(rows), geometry=geoms, crs=WGS84)
    logger.info("built %d feature(s), skipped %d without usable geometry",
                len(frame), skipped)
    return frame


if __name__ == "__main__":
    sample = {"elements": [
        {"type": "node", "id": 1, "lat": 50.06, "lon": 19.94,
         "tags": {"amenity": "pharmacy", "name": "Apteka"}},
        {"type": "way", "id": 2, "center": {"lat": 50.07, "lon": 19.95},
         "tags": {"amenity": "pharmacy"}},
    ]}
    gdf = to_geodataframe(sample, mode="center")
    logger.info("columns: %s", list(gdf.columns))

Step-by-step walkthrough

  1. Take the mode as an argument. The converter never guesses. A mixed response — some elements with center, some with geometry — is a symptom of two queries merged, and guessing hides it.
  2. Build the node map once. In body mode the node coordinates are the only source of way geometry, so they are collected in a single pass before any way is touched.
  3. Skip untagged nodes. A query with a recursion returns thousands of geometry-carrier nodes with no tags. They are not features and including them turns a table of two hundred pharmacies into one of forty thousand rows.
  4. Swap the coordinate order. Overpass names its fields lat and lon; Shapely takes (x, y), which is (lon, lat). This inversion is the single most common bug in OSM conversion code and it produces plausible-looking points in the wrong hemisphere.
  5. Decide area-ness from tags, not from closure. A closed way is a ring, but a roundabout is a closed line. The tag test is what distinguishes a building outline from a circular road, and getting it wrong silently converts roads into polygons.
  6. Warn on incomplete ways. In body mode a way whose nodes are missing means the query had no recursion. The warning names that cause, because the symptom — empty geometry — does not.
  7. Keep a composite key. osm_key is type/id, which is the only unique identifier. Every downstream join should use it rather than the bare numeric id.
  8. Flatten tags into columns. Spreading the tag dictionary across columns gives a usable analytic frame; in a wide, sparse result you may prefer to keep the dictionary in one column instead.
The four transformations between an Overpass response and a usable frame Four steps. First index the nodes in the response by identifier so way geometry can be assembled. Second drop the untagged nodes, which exist only to carry coordinates and are not features. Third build geometry per element according to the declared output mode, deciding area versus line from the tags. Fourth assemble the frame with a composite type and identifier key, flattened tags, and an explicit WGS 84 coordinate reference system. Four steps from response to frame index nodes id to coordinate needed for body mode drop carriers untagged nodes out they are not features build geometry branch on the mode tags decide area or line assemble composite key, flat tags set the CRS explicitly Step two is the one that turns a table of two hundred features into one of forty thousand rows when it is skipped.
Only the third step is interesting; the other three are where the bugs actually live.

Verification

  • Row count matches the tagged element count. Count elements in the raw JSON that carry a tags object; the frame should have that many rows minus any with unusable geometry.
  • No geometry in the wrong hemisphere. Assert every point’s latitude is within your query’s bounding box. A swapped coordinate order shows up immediately as a point off the coast of Africa.
  • Closed roads are lines. Find a roundabout in the result and confirm its geometry type is LineString, not Polygon.
  • The key is unique. frame["osm_key"].is_unique must be True; if it is not, the response contained the same element twice, which happens when two union branches overlap.
  • The CRS is set. frame.crs must be WGS 84; an unset CRS silently breaks every later reprojection, as covered in Coordinate Reference Systems in OSM.
Three conversion bugs that produce plausible-looking but wrong output Three panels naming bugs whose symptom is not an error. Swapped coordinates produce valid points in the wrong part of the world and are caught by asserting latitude against the query bounding box. Included carrier nodes inflate the row count by orders of magnitude and are caught by comparing the frame length against the count of tagged elements. Closure used as the area test converts roundabouts into polygons and is caught by checking a known circular road's geometry type. Three bugs that never raise an exception Swapped coordinates Symptom: points far from the query Cause: lat/lon order to Point() Numbers stay valid, plots fine Catch: assert lat in the bbox Carrier nodes kept Symptom: 40k rows, not 200 Cause: untagged nodes included Every way node became a feature Catch: compare to tagged count Closure as area test Symptom: roads became polygons Cause: ring closure used alone Area totals silently inflated Catch: check a known roundabout None of these three raises anything: each produces a full frame with the right column names and the wrong contents.
Every one of these is caught by a single assertion, and none of them is caught by reading the code.

Common errors and fixes

Symptom Root cause One-line fix
Points in the Gulf of Guinea Latitude and longitude swapped Shapely takes (lon, lat), Overpass gives lat, lon
Tens of thousands of empty rows Untagged carrier nodes included Skip nodes with no tags key
Ways have no geometry out body without a recursion Add >; before out, or switch to out geom
Roundabouts became polygons Closure used as the area test Decide area-ness from tags, not from a closed ring
Duplicate features after a join Joined on osm_id alone Join on the type/id composite key
ValueError on empty frame No elements matched the query Return an empty frame with the right columns and CRS
Reprojection silently wrong crs never set on construction Pass crs="EPSG:4326" when building the frame

Specification reference

The Overpass JSON output format returns a top-level elements array in which each element carries type and id, with lat and lon on nodes, an optional center when the query used the center output mode, and an optional geometry array of coordinate objects when it used geom. Relation members appear under members with their own role and, under geom, their own geometry. See the Overpass API output formats documentation for the field-level definition of each mode.

Frequently Asked Questions

Why are my points in the wrong place?

Almost certainly because latitude and longitude were passed to Shapely in the order Overpass names them. Overpass returns objects with lat and lon fields, but Shapely’s Point constructor takes x then y, which is longitude then latitude. Swapping them produces coordinates that are still valid numbers and still plot, which is why the bug survives review — the points land in a plausible-looking but completely wrong part of the world.

How do I tell a closed way that is an area from one that is a line?

By its tags, never by whether the ring closes. A roundabout, a racetrack and a circular hiking route are all closed ways that are genuinely lines. The convention is that certain keys imply an area — building, landuse, natural, leisure and others — and that an explicit area equals no overrides them. Encode that test once and apply it everywhere, because deciding from closure alone converts roads into polygons and quietly corrupts any area calculation downstream.

Should tags become columns or stay in one dictionary column?

It depends on the shape of the result. For a focused query where most features share a handful of keys, flattening into columns gives a frame you can filter and group naturally. For a broad query across many feature types, flattening produces a very wide and very sparse table, and keeping the tags as a single dictionary or JSON column is both smaller and easier to work with. Decide per query rather than adopting one rule.

Why does my way have no geometry under out body?

Because out body returns ways as lists of node identifiers and nothing else, and those nodes are only present in the response if the query asked for them with a recursion operator. Without the recursion the converter has ids that resolve to nothing. Either add the recursion before the output statement so the nodes arrive alongside the ways, or switch to out geom and let the server inline the coordinates for you.

Up one level: Overpass API Query Language.