Exporting an OSMnx Graph to GeoPackage

A graph and a GIS file disagree about what a thing is. The graph has typed Python objects on edges and an implicit topology; the file has fixed columns and no notion of a node being connected to anything. The export is where that disagreement has to be settled deliberately.

Prerequisites

Conceptual minimum

A graph exports as two layers: nodes and edges. The nodes layer is straightforward — a point per node with its identifier. The edges layer is where the problems are.

Edges need their endpoints as columns. A GIS file has no concept of connectivity, so the only way a graph can be rebuilt is if each edge row carries its source and target node identifiers explicitly.

Parallel edges need a key. Two roads can connect the same pair of junctions, and without a third component in the key the two collapse into one on reload.

List-valued attributes have no column type. An OSM way merged from several source ways carries a list of identifiers; a GeoPackage column holds a scalar. Dropping the attribute loses provenance and writing the Python representation of a list produces a string nothing can parse reliably. Joining on a separator, consistently, is the workable answer.

Nulls and mixed types break the write. A column that is an integer on most rows and a string on a few has no GeoPackage type, and the write either fails or silently coerces.

What each graph concept becomes in the file, and what breaks if it is omitted A grid of four graph concepts against their file representation and the consequence of omitting them. Node identity becomes an identifier column on the nodes layer, without which nothing can be joined. Edge endpoints become source and target columns on the edges layer, without which the topology is unrecoverable. Parallel edges need a key column distinguishing them, without which two roads between the same junctions collapse into one. List attributes need a consistent joined string, without which provenance is either lost or written in a form nothing can parse. Four concepts, four columns, four failures In the file If omitted Node identity an id column nothing joins Edge endpoints source and target topology lost Parallel edges a key column two roads become one List attributes joined on a separator provenance lost Only the first of these is produced automatically by a naive export; the other three have to be arranged deliberately.
A file that opens correctly in a GIS viewer can still be missing everything needed to rebuild the graph.

Runnable solution

python 
from __future__ import annotations

import logging
from pathlib import Path

import geopandas as gpd
import networkx as nx
import osmnx as ox
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.graph.export")

LIST_SEPARATOR = "|"          # not a comma: OSM values contain commas


def flatten_lists(frame: gpd.GeoDataFrame) -> gpd.GeoDataFrame:
    """Join list-valued cells into strings. A GeoPackage column holds scalars."""
    out = frame.copy()
    for column in out.columns:
        if column == "geometry":
            continue
        has_lists = out[column].map(lambda v: isinstance(v, (list, tuple))).any()
        if not has_lists:
            continue
        out[column] = out[column].map(
            lambda v: LIST_SEPARATOR.join(str(x) for x in v)
            if isinstance(v, (list, tuple)) else v)
        logger.info("flattened list values in column %r", column)
    return out


def coerce_types(frame: gpd.GeoDataFrame) -> gpd.GeoDataFrame:
    """Give every column one type. Mixed columns have no GeoPackage type."""
    out = frame.copy()
    for column in out.columns:
        if column == "geometry":
            continue
        series = out[column]
        if series.dtype == object:
            kinds = {type(v).__name__ for v in series.dropna()}
            if len(kinds) > 1:
                # Mixed types: stringify the whole column rather than let the
                # driver coerce unpredictably or drop rows.
                logger.warning("column %r has mixed types %s; storing as text",
                               column, sorted(kinds))
                out[column] = series.map(lambda v: None if v is None else str(v))
    return out


def export(graph: nx.MultiDiGraph, path: Path) -> Path:
    nodes, edges = ox.graph_to_gdfs(graph, nodes=True, edges=True,
                                    fill_edge_geometry=True)

    # Endpoints and the parallel-edge key live in the index; a file has no
    # index, so they must become real columns or the topology is unrecoverable.
    edges = edges.reset_index()
    for required in ("u", "v", "key"):
        if required not in edges.columns:
            raise ValueError(f"edges are missing {required!r}; the graph cannot "
                             f"be rebuilt from this export")
    nodes = nodes.reset_index().rename(columns={"osmid": "node_id"})

    edges = coerce_types(flatten_lists(edges))
    nodes = coerce_types(flatten_lists(nodes))

    if nodes.crs is None or edges.crs is None:
        raise ValueError("graph has no CRS; set one before exporting")

    path.unlink(missing_ok=True)
    nodes.to_file(path, layer="nodes", driver="GPKG")
    edges.to_file(path, layer="edges", driver="GPKG")
    logger.info("wrote %d node(s) and %d edge(s) to %s",
                len(nodes), len(edges), path.name)
    return path


def rebuild(path: Path) -> nx.MultiDiGraph:
    """Read the file back into a graph. This is the only real verification."""
    nodes = gpd.read_file(path, layer="nodes").set_index("node_id")
    edges = gpd.read_file(path, layer="edges").set_index(["u", "v", "key"])
    graph = ox.graph_from_gdfs(nodes, edges)
    logger.info("rebuilt graph with %d node(s) and %d edge(s)",
                graph.number_of_nodes(), graph.number_of_edges())
    return graph


def compare(original: nx.MultiDiGraph, restored: nx.MultiDiGraph) -> bool:
    ok = True
    for name, a, b in (("node", original.number_of_nodes(),
                        restored.number_of_nodes()),
                       ("edge", original.number_of_edges(),
                        restored.number_of_edges())):
        if a != b:
            logger.error("%s count differs: %d before, %d after", name, a, b)
            ok = False
    before = sum(d.get("length", 0.0) for *_, d in original.edges(data=True))
    after = sum(d.get("length", 0.0) for *_, d in restored.edges(data=True))
    if abs(before - after) > max(1.0, before * 1e-9):
        logger.error("total length differs: %.1f before, %.1f after", before, after)
        ok = False
    return ok


if __name__ == "__main__":
    logger.info("export, rebuild, compare — an export that cannot be rebuilt "
                "is a picture of a graph rather than a graph")

Step-by-step walkthrough

  1. Promote the index to columns. The endpoint identifiers and the parallel-edge key live in the frame’s index, and a file has no index. Failing to reset it produces a file that draws correctly and cannot be rebuilt.
  2. Assert the three key columns exist. Checking before writing turns a silent loss of topology into a clear error at the right moment.
  3. Choose a separator that does not appear in values. A comma is the obvious choice and the wrong one, because OSM names and references contain commas. A pipe is safe in practice.
  4. Flatten lists rather than dropping them. The merged-way identifier list is provenance, and losing it means a feature in the file cannot be traced to the OSM ways it came from.
  5. Coerce mixed-type columns to text. A column that is numeric on most rows and a string on a few has no file type, and letting the driver decide produces either a failure or a silent coercion.
  6. Require a CRS. GeoPackage stores it, so a wrong or missing value travels with the file and misplaces everything downstream.
  7. Rebuild and compare. Node count, edge count and total length together catch collapsed parallel edges, lost rows and mangled geometry. An export that has not been rebuilt has not been verified.
The export and the verification that makes it meaningful Four steps. The convert step turns the graph into node and edge frames with geometry filled in for every edge. The promote step moves the endpoint identifiers and parallel-edge key out of the index and into real columns, since a file has no index. The clean step flattens list-valued attributes onto a safe separator and gives mixed-type columns a single type. The verify step reads the file back into a graph and compares node count, edge count and total length against the original. Convert, promote, clean, verify convert nodes and edges fill edge geometry promote index to columns or topology is lost clean flatten and coerce one type per column verify rebuild and compare the only real check An export that opens in a viewer proves the geometry is fine and says nothing about whether the graph survived.
The fourth step is the only one that would notice parallel edges collapsing, which a viewer cannot show.
Three things a GIS viewer cannot tell you about an exported graph Three panels. Collapsed parallel edges look identical in a viewer, because two overlapping roads between the same junctions draw as one line either way, and only a count comparison reveals the loss. An unparseable list attribute looks like an ordinary text column, and only attempting to split it reveals that the values were written in a form nothing can read back. Missing endpoint columns are invisible entirely, because the geometry draws perfectly and the topology simply is not there. Three losses a viewer renders perfectly Collapsed parallel edges Two roads draw as one Either way, identically Only counts reveal it Routing silently changes Unparseable lists Looks like a text column Until you split it Provenance unrecoverable Discovered much later Missing endpoints Geometry draws perfectly Topology is absent Invisible in a viewer Found by the router Each of these is caught immediately by rebuilding the graph, and by nothing else that anybody would think to do.
The export's purpose is the graph, so the check has to be on the graph rather than on the picture.

Verification

  • The graph rebuilds. Reading the file back must produce a graph, not an error about missing columns.
  • Counts match exactly. Node and edge counts before and after must be identical; a lower edge count means parallel edges collapsed.
  • Total length is preserved. Summing edge lengths should agree to floating-point tolerance.
  • List attributes survive. Find an edge merged from several ways and confirm its identifier list is present and splittable.
  • The CRS is correct. Open the file in a GIS and confirm it lands where expected.

Common errors and fixes

Symptom Root cause One-line fix
Graph cannot be rebuilt Endpoint identifiers left in the index Reset the index so they become columns
Edge count falls on reload Parallel-edge key not exported Include the key column in the edges layer
List attributes become unparseable Python list representation written Join on a separator absent from the values
Write fails on a column Mixed types in one column Coerce the whole column to text
Features land in the wrong place CRS unset or wrong Require a CRS before writing
Separator splits real values A comma used as the separator Use a character that does not occur in OSM values
Edges have no geometry Geometry not filled for straight edges Fill edge geometry during conversion

Specification reference

GeoPackage stores vector features in SQLite tables with a declared geometry column and coordinate reference system, one table per layer, with fixed column types. It has no representation for list-valued attributes or for graph connectivity, so both must be encoded explicitly by the writer. See the GeoPackage specification for the table and geometry model.

Frequently Asked Questions

Why must the endpoint identifiers become columns?

Because a GIS file has no notion of an index or of connectivity. In memory the edge frame is indexed by source, target and key, and that index is what encodes the topology; writing the file discards it. Without those three as real columns, the file contains a set of lines that draw correctly and cannot be reassembled into a graph, which is usually discovered by whoever tries to route on it.

Why not use a comma to join list values?

Because OSM values contain commas routinely — names, references, opening hours and address fields all do. Splitting on a comma then breaks a single value into fragments, and the damage is silent because the fragments look like plausible list members. A character that does not occur in the data, such as a pipe, avoids the problem entirely and costs nothing.

What should happen to a column with mixed types?

Coerce the whole column to text and say so. A GeoPackage column has one type, so a column that is numeric on most rows and a string on a few must be resolved somehow; letting the driver choose produces either a failed write or a silent coercion that loses the non-conforming values. Stringifying preserves everything at the cost of the consumer parsing it back.

Is opening the file in a GIS a sufficient check?

No. A viewer shows that the geometry is present and correctly located, which is the part least likely to be wrong. It cannot show that parallel edges collapsed, that a list attribute became unparseable, or that the endpoint columns are missing. Rebuilding the graph from the file and comparing counts and total length is the only check that exercises what the export was for.

Up one level: OSMnx Graph Conversion Techniques.