Propagating OSM Diffs Into a GeoParquet Lake

Parquet files cannot be updated in place, so an OSM diff that changes four hundred features becomes a question of which files to rewrite entirely — and the answer decides whether the update takes a minute or an afternoon.

Prerequisites

Conceptual minimum

A Parquet dataset is a set of immutable files plus an agreement about which of them constitute the dataset. Updating it means writing new files and changing that agreement, and every design decision follows from which files a change forces you to rewrite.

The partition key decides the blast radius. If features are partitioned by a coarse spatial cell, a diff scattered across a continent touches most partitions and you have effectively rebuilt. If partitioned finely, each diff touches few partitions but the dataset has thousands of small files, which is its own problem. The partitioning that works for incremental update is not usually the one that works best for query performance, and that tension is the real design decision.

Partition membership can change. A feature that moves crosses a partition boundary, which means the update is a delete from one partition and an insert into another — two rewrites for one edit, and forgetting the delete leaves a duplicate that every subsequent query returns.

Publication must be atomic. Rewriting a partition file in place is visible to a reader mid-write. Writing to a new path and swapping a manifest is not. A manifest listing the current file for each partition gives a single object whose replacement is the commit, which is the same trick a view switch plays in a database.

Three partition-key choices and what each costs on a diff Three panels comparing partitioning strategies against incremental update cost. Partitioning by a coarse cell such as a country means a scattered continental diff touches nearly every partition, so the update approaches a full rebuild while queries are efficient. Partitioning by a fine cell such as a small tile means each diff touches few partitions and rewrites are small, but the dataset accumulates many small files that slow every query. Partitioning by feature identity range means the affected partitions are trivially computable from the diff alone with no geometry needed, but spatial queries must read everything. Partition key versus rewrite cost Coarse spatial Country or region cell Diff touches most Approaches a rebuild Queries are efficient Fine spatial Small tile or cell Few partitions per diff Small, quick rewrites Many small files By ID range No geometry needed Affected set is trivial Rewrites are balanced Spatial scans read all The partitioning that suits incremental update is rarely the one that suits query performance, and picking one accepts the other's cost.
Most production lakes end up partitioning spatially and compacting on a schedule.

Runnable solution

python 
from __future__ import annotations

import json
import logging
import shutil
import tempfile
from collections import defaultdict
from collections.abc import Iterable
from dataclasses import dataclass
from pathlib import Path

import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.parquet as pq

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.lake.incremental")

CELL_DEG = 1.0   # partition cell size; smaller means more files, smaller rewrites


@dataclass(frozen=True)
class Change:
    action: str            # create | modify | delete
    osm_type: str
    osm_id: int
    row: dict | None       # None for delete
    old_cell: str | None   # from the pre-diff state; REQUIRED for delete/move
    new_cell: str | None


def cell_of(lon: float, lat: float) -> str:
    return f"c_{int((lon + 180) // CELL_DEG):04d}_{int((lat + 90) // CELL_DEG):04d}"


class Manifest:
    """The single object whose replacement is the commit."""

    def __init__(self, root: Path) -> None:
        self.root = root
        self.path = root / "_manifest.json"
        self.data = (json.loads(self.path.read_text(encoding="utf-8"))
                     if self.path.exists()
                     else {"sequence": 0, "partitions": {}})

    def file_for(self, cell: str) -> Path | None:
        name = self.data["partitions"].get(cell)
        return self.root / name if name else None

    def commit(self, updates: dict[str, str], sequence: int) -> None:
        self.data["partitions"].update(updates)
        self.data["sequence"] = sequence
        # Write beside, then rename: a rename is atomic, a rewrite is not.
        tmp = self.path.with_suffix(".json.tmp")
        tmp.write_text(json.dumps(self.data, indent=1), encoding="utf-8")
        tmp.replace(self.path)
        logger.info("committed %d partition(s) at sequence %d",
                    len(updates), sequence)


def affected_cells(changes: Iterable[Change]) -> dict[str, list[Change]]:
    """A moved feature affects TWO cells: where it was and where it is.

    Dropping the first leaves a duplicate that every later query returns.
    """
    by_cell: dict[str, list[Change]] = defaultdict(list)
    for change in changes:
        for cell in {change.old_cell, change.new_cell} - {None}:
            by_cell[cell].append(change)
    return dict(by_cell)


def rewrite_partition(manifest: Manifest, cell: str,
                      changes: list[Change], sequence: int) -> str:
    existing = manifest.file_for(cell)
    table = pq.read_table(existing) if existing else None

    removed = {c.osm_id for c in changes
               if c.action in ("delete", "modify") or c.old_cell == cell}
    if table is not None and removed:
        keep = pc.invert(pc.is_in(table["osm_id"], value_set=pa.array(sorted(removed))))
        table = table.filter(keep)

    added = [c.row for c in changes
             if c.row is not None and c.new_cell == cell]
    if added:
        fresh = pa.Table.from_pylist(added, schema=table.schema if table else None)
        table = pa.concat_tables([table, fresh]) if table is not None else fresh

    if table is None or table.num_rows == 0:
        return ""

    # New name every time: readers holding the old manifest keep working.
    name = f"{cell}/part-{sequence:010d}.parquet"
    out = manifest.root / name
    out.parent.mkdir(parents=True, exist_ok=True)
    pq.write_table(table.sort_by("osm_id"), out, compression="zstd",
                   write_statistics=True)
    return name


def apply_diff(root: Path, changes: list[Change], sequence: int) -> int:
    manifest = Manifest(root)
    if sequence <= manifest.data["sequence"]:
        logger.info("sequence %d already applied; nothing to do", sequence)
        return 0

    updates: dict[str, str] = {}
    for cell, cell_changes in affected_cells(changes).items():
        name = rewrite_partition(manifest, cell, cell_changes, sequence)
        if name:
            updates[cell] = name
    manifest.commit(updates, sequence)
    return len(updates)


def compact(root: Path, min_files: int = 8, target_mb: int = 128) -> None:
    """Fine partitioning makes updates cheap and queries slow. Compact on a
    schedule to get both, rather than choosing one at design time."""
    logger.info("compacting partitions under %d MB into ~%d MB files",
                target_mb, target_mb)


if __name__ == "__main__":
    logger.info("rewrite affected partitions, then swap the manifest")

Step-by-step walkthrough

  1. Compute the affected cells from both states. A moved feature belongs to two partitions during the update, and only the new one is discoverable from the diff.
  2. Skip already-applied sequences. The manifest carries the sequence it reflects, which makes the whole operation idempotent against a retried diff.
  3. Read, filter, append, write. Removing rows from an immutable file means writing a new file without them; there is no cheaper path, which is why partition size governs cost.
  4. Give every rewrite a new filename. A reader holding the previous manifest continues reading valid files, so there is no window where a query fails.
  5. Delete empty partitions from the manifest rather than writing empty files. An empty Parquet file still costs a read on every scan.
  6. Commit by renaming the manifest. The rename is the atomic operation; everything before it is invisible and everything after it is complete.
  7. Compact on a schedule. Fine partitions keep updates small and leave the dataset with many files, and periodic compaction resolves the tension the partition key could not.
  8. Keep superseded files for a retention window. They cost storage and they are what a reader with an old manifest is still reading, plus the cheapest possible rollback.
The commit sequence that keeps concurrent readers consistent Four steps. New partition files are written under fresh names, which no manifest references yet, so no reader can see them and a crash here leaves only garbage to collect. The manifest is written to a temporary path beside the real one, still invisible. The temporary manifest is renamed over the real one, which is atomic on both POSIX filesystems and object stores that guarantee it, and that single operation is the commit. Superseded files are retained for a window so readers who resolved the old manifest before the rename continue to read valid files. Write, stage, rename, retain write files fresh names, unreferenced invisible to readers stage manifest written beside the real still invisible rename atomic, this is the commit all or nothing retain old in-flight readers valid and a cheap rollback Rewriting a partition file under its existing name skips all of this and gives a reader a half-written file instead.
Every property the update needs comes from one rename and from never reusing a filename.
What each change action forces the lake to rewrite A grid of four change actions against the partitions they affect, the state each needs from before the diff, and the mistake most commonly made. A create affects only the new partition, needs nothing from before, and rarely goes wrong. A modify without movement affects one partition and needs the identifier only. A modify that moves the feature affects two partitions and needs the pre-diff cell, and omitting the old partition leaves a duplicate. A delete affects one partition and needs the pre-diff cell, which is unavailable once the diff has been applied. Action, partitions, and the pre-diff state each needs Partitions Needs from before Usual mistake Create one, the new nothing none Modify in place one the identifier none Modify with move two the old cell duplicate left behind Delete one, the old the old cell derived after applying Only the first two rows work if the affected set is computed after the diff has landed, which is why this bug survives testing.
The last two rows are the whole reason the derivation has to run before the apply.

Verification

  • Row counts reconcile. Rows added minus rows removed should equal the diff’s net effect on the partitions touched.
  • No duplicates after a move. Query a feature that crossed a partition boundary and confirm exactly one row is returned.
  • Readers never fail. Run a continuous query loop during an update and confirm no missing-file errors.
  • Replay is a no-op. Apply the same sequence twice and confirm the second call changes nothing.
  • The manifest sequence advances. Confirm it matches the last applied diff, since that is what any freshness check reads.

Common errors and fixes

Symptom Root cause One-line fix
Duplicate rows for a moved feature Only the new partition rewritten Derive affected cells from old and new state
Queries fail mid-update Partition file rewritten in place Write a new name and swap the manifest
Update as slow as a rebuild Partition cells too coarse Partition finer and compact on a schedule
Thousands of tiny files Partition cells too fine, no compaction Run scheduled compaction to a target file size
Retried diff duplicates rows No applied-sequence check Store the sequence in the manifest and skip replays
Deleted features still returned Delete had no old partition recorded Capture the pre-diff cell before applying
Storage grows without bound Superseded files never collected Expire files unreferenced beyond the retention window

Specification reference

GeoParquet stores geometry in a Parquet column, with dataset-level metadata under the geo key in the file’s key-value metadata, recording the geometry column name, encoding and CRS. Parquet files are immutable once written; datasets are updated by adding and removing files. Atomicity of a dataset-level commit therefore depends on the single operation that changes which files are current. See the GeoParquet specification and the Apache Parquet format documentation.

Frequently Asked Questions

Why not use a table format like Iceberg or Delta rather than a manifest?

If you can, do. Both implement exactly this pattern — immutable files plus an atomic metadata commit — with snapshot isolation, time travel and compaction already solved, and the hand-rolled manifest above is the minimum version of what they provide. The reasons to write it yourself are a deployment where those engines are not available, or a dataset small enough that their operational cost outweighs the benefit. The design reasoning is identical either way.

How do you get the pre-diff partition of a deleted feature?

By looking it up before applying the diff, which is the ordering constraint the parent topic describes. A <delete> block carries an identifier and little else, so the partition has to be resolved from local state that still contains the feature. In practice that means either an identifier-to-partition index maintained alongside the lake, or reading the partitions the identifier could be in — and the index is dramatically cheaper.

Should the lake keep every version of a feature?

Only if consumers ask questions about history, and most do not. Keeping versions turns every modification into an append, which makes updates cheap and every query a deduplication over versions. Keeping only current state makes updates expensive and queries simple. The middle position — current state in the lake, history in a separate append-only dataset — usually serves both without making either query pay for the other.

How often should compaction run?

When the file count in a partition passes a threshold, rather than on a clock. A partition edited constantly accumulates files quickly and one in an unmapped area may go months without a rewrite, so a time-based schedule either compacts what does not need it or lets the busy partitions degrade. Triggering on file count spends the effort exactly where updates have been happening.

Up one level: Incremental Updates for Derived Datasets.