Updating a Search Index from OSM Diffs

A search index is the easiest derived dataset to update incrementally and the easiest to get quietly wrong, because documents embed context from features other than the one they describe.

Prerequisites

Conceptual minimum

Identity makes the base case trivial. A document identifier of n123456 or w987654 means a create is an upsert, a modify is an upsert, and a delete is a delete by identifier. Ninety-something percent of an OSM diff maps onto that directly, and a pipeline that does only this is right most of the time.

The remaining percentage is where the work is, and it has three sources.

Denormalised context. A street document typically carries the name of its city, its region and its country, because searching for “Main Street, Springfield” has to match something. Those names come from features the diff may never mention. When the city is renamed, every document that embedded the old name is stale, and nothing in the change file points at them.

Filter transitions. A feature that stops being searchable — a place=town retagged to something else, a name removed — does not appear in the diff as a delete. It appears as a modify, and a pipeline that upserts modifications and deletes deletions leaves it in the index forever. Every modify must be evaluated against the searchability filter, and a feature that fails it must be deleted rather than skipped.

Re-parenting. A feature whose containing boundary changed — because the boundary moved, not because the feature did — needs its context recomputed without having changed at all.

The practical resolution is that context changes are a separate, slower path. Identity-keyed updates run per diff in seconds. Context updates run when a context-bearing feature changes, are potentially enormous, and belong in a background reindex of the affected area rather than in the minutely loop.

Three ways an index goes stale without any wrong document being written Three panels. Denormalised context goes stale when a city is renamed, because every street document embedded the old city name and the change file names only the city. A filter transition leaves a document behind when a feature stops qualifying for the index, since that appears as a modification rather than a deletion and a pipeline that upserts modifications keeps it searchable forever. Re-parenting changes a document's administrative context when a boundary moves, even though the feature itself was never edited and appears in no diff. Three stale-index paths Stale context City renamed once Streets embedded old name Diff names only the city Documents never touched Filter transition Feature stops qualifying Appears as a modify Upsert keeps it indexed Searchable forever Re-parenting Boundary moved Feature never edited Context now wrong Appears in no diff All three produce documents that are individually well formed, which is why no schema validation or write-path check will ever notice them.
The identity-keyed path handles none of these, and it is the only path most pipelines implement.

Runnable solution

python 
from __future__ import annotations

import logging
from collections.abc import Iterable, Iterator
from dataclasses import dataclass, field

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.search.incremental")

# Features worth indexing at all. Everything else is filtered before any work.
SEARCHABLE_KEYS = ("place", "amenity", "shop", "tourism", "highway", "boundary")
# Features whose own attributes are embedded in OTHER documents.
CONTEXT_KEYS = ("boundary", "place")


@dataclass(frozen=True)
class Change:
    action: str                 # create | modify | delete
    osm_type: str
    osm_id: int
    tags: dict[str, str] = field(default_factory=dict)
    old_tags: dict[str, str] = field(default_factory=dict)


def doc_id(osm_type: str, osm_id: int) -> str:
    return f"{osm_type[0]}{osm_id}"


def is_searchable(tags: dict[str, str]) -> bool:
    """A feature with no name is not findable by name, whatever else it has."""
    if not tags.get("name"):
        return False
    return any(key in tags for key in SEARCHABLE_KEYS)


def is_context_bearing(tags: dict[str, str]) -> bool:
    return any(key in tags for key in CONTEXT_KEYS)


@dataclass
class Plan:
    upserts: list[str] = field(default_factory=list)
    deletes: list[str] = field(default_factory=list)
    reindex_areas: list[tuple[str, int]] = field(default_factory=list)


def plan_from_diff(changes: Iterable[Change]) -> Plan:
    plan = Plan()
    for change in changes:
        identifier = doc_id(change.osm_type, change.osm_id)

        if change.action == "delete":
            plan.deletes.append(identifier)
        elif is_searchable(change.tags):
            plan.upserts.append(identifier)
        elif is_searchable(change.old_tags):
            # It WAS indexed and no longer qualifies. A modify, not a delete —
            # skipping it here is how features stay searchable after they stop
            # being findable in reality.
            plan.deletes.append(identifier)

        # A renamed city or a moved boundary invalidates documents that
        # embedded its name. Those documents are not in this diff at all.
        context_changed = (
            is_context_bearing(change.tags) or is_context_bearing(change.old_tags)
        ) and change.tags.get("name") != change.old_tags.get("name")
        if context_changed or (change.action != "modify"
                               and is_context_bearing(change.tags)):
            plan.reindex_areas.append((change.osm_type, change.osm_id))

    return plan


def bulk_operations(plan: Plan, index: str) -> Iterator[dict]:
    """Emit operations in delete-then-upsert order.

    A feature that moved out of one document shape and into another must lose
    the old document before gaining the new one, or both exist briefly.
    """
    for identifier in plan.deletes:
        yield {"delete": {"_index": index, "_id": identifier}}
    for identifier in plan.upserts:
        yield {"index": {"_index": index, "_id": identifier}}


def apply(plan: Plan, client, index: str, sequence: int) -> None:
    if plan.deletes or plan.upserts:
        client.bulk(list(bulk_operations(plan, index)), refresh=False)
        logger.info("seq %d: %d upsert(s), %d delete(s)", sequence,
                    len(plan.upserts), len(plan.deletes))

    # The slow path. Queued, not run inline: one boundary rename can touch
    # hundreds of thousands of documents and must not stall the minutely loop.
    for osm_type, osm_id in plan.reindex_areas:
        client.enqueue_area_reindex(osm_type, osm_id)
        logger.info("seq %d: queued context reindex for %s%d",
                    sequence, osm_type, osm_id)

    client.set_marker(index, sequence)


if __name__ == "__main__":
    logger.info("identity path inline, context path queued")

Step-by-step walkthrough

  1. Filter before doing anything. Most changes touch features the index does not contain, and discarding them first is the difference between a diff costing milliseconds and costing seconds.
  2. Evaluate every modify against the filter twice. Once against the new tags and once against the old, because the transition out of searchability is a modify that must become a delete.
  3. Treat a missing name as unsearchable. A feature nobody can search for by name is a document that only adds noise, and removing the name is a common edit.
  4. Detect context-bearing changes separately. A boundary or place whose name changed invalidates documents that embedded it, none of which appear in the diff.
  5. Queue the context reindex; never run it inline. One country rename can touch millions of documents, and a minutely loop that attempts it stops being minutely.
  6. Order deletes before upserts in the bulk body. A document whose identifier changed shape must lose its old form before gaining the new one.
  7. Do not refresh per diff. Forcing a refresh on every minutely update multiplies segment churn for a freshness guarantee search users cannot perceive.
  8. Advance the index’s own marker. The index’s sequence is not the database’s, and conflating them is how a stalled index reports itself current.
Routing a change to the fast path or the slow path A decision with three branches based on what a change touches. A change to a feature that is not searchable, before or after, is discarded immediately and costs nothing, which covers most of a typical diff. A change to a searchable non-context feature goes to the inline identity path, becoming an upsert or a delete in the current bulk request. A change to a context-bearing feature such as a boundary or a place name queues an area reindex on the background path, because it can invalidate an unbounded number of documents that the diff never names. Which path a change takes What does this change touch? Evaluated against old and new tags Most diffs stop at the first branch Nothing searchable: discard Costs nothing, and covers most of a typical change file A searchable feature: inline Upsert or delete by identifier in this diff's bulk request Context bearing: queue a reindex Unbounded fan-out; never run inside the minutely loop Running the third branch inline is how a minutely index update occasionally takes forty minutes and nobody can say why.
The branches differ by orders of magnitude in cost, which is the reason to separate them.
How a single city rename reaches documents the diff never names Four steps. A change file contains one modification: a place node whose name tag changed. The identity path upserts exactly one document, which is correct and insufficient. The context detector notices the changed feature is context bearing and its name differs from the previous name, so it queues an area reindex keyed on that feature. The background worker resolves the feature geometry, queries the primary database for every searchable feature inside it, and bulk upserts those documents with the corrected context, which may be hundreds of thousands of documents from one edit. One edit, hundreds of thousands of documents one modify a place name changed a single diff entry identity path upserts one document correct, insufficient queue reindex context bearing, renamed keyed on that feature background pass everything inside it context corrected The fan-out between the second and fourth step is unbounded, which is the whole reason the two paths are separated.
Nothing in the change file points at the documents the fourth step rewrites.

Verification

  • A renamed feature becomes findable by the new name. Search for it after the next diff.
  • An unsearchable transition removes the document. Strip a feature’s name and confirm it disappears from results.
  • A deleted feature is gone. Delete and search; the absence should be immediate after refresh.
  • Context reindex is queued, not run. Rename a boundary and confirm the diff completes quickly with a queued job.
  • The index marker advances. Compare it against the database’s sequence and confirm the gap is bounded.

Common errors and fixes

Symptom Root cause One-line fix
Removed features still searchable Modify out of searchability treated as a skip Evaluate old tags too and emit a delete
Stale city names in street documents Denormalised context never invalidated Queue an area reindex on context changes
Minutely loop occasionally takes an hour Context reindex run inline Move it to a background queue
Documents duplicated after an edit Upserts ordered before deletes Emit deletes first in the bulk body
Index reported current while stale Sharing the database’s sequence marker Give the index its own marker
Search latency degrades over time Forced refresh on every diff Refresh on an interval, not per update
Diffs cost seconds for no changes No searchability filter before work Filter first; most changes are irrelevant

Specification reference

A bulk request applies its actions in the order given. Index actions create or replace a document with the given identifier; delete actions remove it. A refresh makes recent operations visible to search and is expensive relative to indexing, so it should not be requested per operation in a streaming workload. See the Elasticsearch bulk API and refresh documentation, and Incremental Updates for Derived Datasets for the affected-set framing.

Frequently Asked Questions

How do you bound the cost of a context reindex?

By scoping it to the geometry of the changed context feature. A renamed city invalidates documents inside that city, not everywhere, so the reindex is a spatial query against the primary database plus a bulk upsert of what it returns. The cost is proportional to the feature count inside the boundary, which for a country is genuinely large — and that case is the reason it runs in the background with a rate limit rather than inline.

Should the index store OSM tags or a flattened document?

Flattened, almost always. A search index is queried by people typing names, not by pipelines asking about tags, and storing raw tags pushes the interpretation work into query time where it is repeated on every request. Flatten once during indexing into the handful of fields search actually uses, and keep the raw tags in the primary database where the next reindex can read them.

What happens when the index falls a long way behind?

The same calculation as any derived dataset: at some gap, replaying diffs costs more than reindexing from current state. For a search index the crossover is usually favourable to reindexing, because a full reindex is a single scan with bulk writes and no per-diff overhead. Keeping the index’s own marker is what makes the gap visible enough to make that call.

Do deletes need to be immediate?

More than upserts do. A missing result is a mild disappointment; a result that leads somewhere that no longer exists is a user following directions to a closed business. If anything is worth running promptly in an otherwise batched update, it is the delete path, and it is also the cheapest part since a delete carries no document body.

Up one level: Incremental Updates for Derived Datasets.