Handling Deleted and Redacted OSM Objects

A deleted object is one somebody removed from the map. A redacted version is one that was removed from the record, usually because it should never have been there. The two look similar from a distance and need entirely different handling.

Prerequisites

Conceptual minimum

A deletion is an ordinary edit. The object gains a final version marked not visible, its identifier is retired, and the whole history remains readable. A stored reference resolves to a not-found response, and the history explains what happened and who did it.

A redaction removes one or more versions from the public history, typically because they contained data that could not be licensed or that should not have been published. The object may still exist with later versions intact; what vanishes is the record of those particular versions. The visible effect is a gap in the version sequence — an object whose history jumps from version 3 to version 6.

The distinction matters for three reasons. A deletion is reversible by an ordinary revert and a redaction is not. A deletion leaves the history intact for reconstruction and a redaction deliberately does not. And data your pipeline captured from a since-redacted version is data you may not be entitled to keep, which is a question about your own storage rather than about the map.

Deletion and redaction compared on what disappears and what remains Three panels. A deletion adds a final version marked not visible, retires the identifier, leaves the entire history readable and is reversible by an ordinary revert. A redaction removes specific versions from the public history, leaves a gap in the version numbering, may leave the object otherwise intact, and is not reversible by any editing operation. The third panel covers what a pipeline should do differently: a deletion means the feature left the map and can be recorded as such, while a redaction may mean data you hold should not be retained. Two disappearances, two different questions Deletion Final version, not visible Identifier retired History stays readable Revertible normally The feature left the map Redaction Versions removed entirely Gap in the numbering Object may still exist Not revertible The record was corrected Your pipeline Deletion: record it left Redaction: check what you hold Content may be unlicensable Retention is your question Not the map's problem Only the third panel requires a decision from you; the first two describe what upstream did and why.
A redaction is a statement about what should never have been published, which makes it your storage's problem too.

Runnable solution

python 
from __future__ import annotations

import logging
import xml.etree.ElementTree as ET
from dataclasses import dataclass
from enum import Enum

import requests

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger("osm.identity.removal")

API = "https://api.openstreetmap.org/api/0.6"
HEADERS = {"User-Agent": "osm-pipeline-example/1.0 (contact@example.org)"}


class State(str, Enum):
    PRESENT = "present"
    DELETED = "deleted"
    REDACTED_VERSIONS = "redacted_versions"
    UNKNOWN = "unknown"


@dataclass(frozen=True)
class Status:
    osm_type: str
    osm_id: int
    state: State
    current_version: int | None
    missing_versions: tuple[int, ...]
    detail: str


def fetch_history(osm_type: str, osm_id: int) -> list[ET.Element]:
    response = requests.get(f"{API}/{osm_type}/{osm_id}/history",
                            headers=HEADERS, timeout=60)
    if response.status_code == 404:
        return []
    response.raise_for_status()
    return ET.fromstring(response.content).findall(osm_type)


def assess(osm_type: str, osm_id: int) -> Status:
    elements = fetch_history(osm_type, osm_id)
    if not elements:
        return Status(osm_type, osm_id, State.UNKNOWN, None, (),
                      "no history returned; the object may never have existed")

    versions = sorted(int(e.get("version")) for e in elements)
    highest = versions[-1]
    # A redaction removes versions, leaving holes in an otherwise dense run.
    missing = tuple(v for v in range(1, highest + 1) if v not in set(versions))

    last = max(elements, key=lambda e: int(e.get("version")))
    deleted = last.get("visible", "true") == "false"

    if missing and deleted:
        detail = (f"deleted, and versions {missing} are absent from the history")
        state = State.DELETED
    elif missing:
        detail = f"present, but versions {missing} are absent from the history"
        state = State.REDACTED_VERSIONS
    elif deleted:
        detail = "deleted by an ordinary edit; full history remains"
        state = State.DELETED
    else:
        detail = "present with a complete version sequence"
        state = State.PRESENT

    logger.info("%s/%d: %s (%s)", osm_type, osm_id, state.value, detail)
    return Status(osm_type, osm_id, state, highest, missing, detail)


def handle_stored_reference(status: Status, stored_version: int) -> str:
    """What a pipeline should do about a reference it holds."""
    if status.state is State.PRESENT:
        return "resolve normally"
    if status.state is State.DELETED:
        # The feature left the map. Record that; do not silently drop the row.
        return ("mark the reference as no longer present and keep the record "
                "of when it was; the history explains what happened")
    if status.state is State.REDACTED_VERSIONS:
        if stored_version in status.missing_versions:
            # We captured data from a version that has since been removed.
            return ("REVIEW: the stored version was redacted; re-derive from a "
                    "current version and consider whether the captured content "
                    "may still be retained")
        return "re-resolve against the current version; the gap does not affect us"
    return "investigate: the object cannot be assessed from its history"


if __name__ == "__main__":
    status = assess("way", 4305800)
    logger.info("%s", handle_stored_reference(status, stored_version=3))

Step-by-step walkthrough

  1. Detect a redaction by the gap, not by an error. Nothing in the API announces a redaction; a missing version number in an otherwise dense sequence is the only signal available.
  2. Check both conditions independently. An object can be both deleted and have redacted versions, and reporting only the first loses the more consequential fact.
  3. Treat an empty history as unknown, not as deleted. A not-found response means the identifier was never used or the request was wrong, which is a different situation from an object that existed and was removed.
  4. Decide per stored version. A redaction only matters to you if the version you captured is one of the removed ones. Otherwise the gap is upstream housekeeping.
  5. Record a deletion rather than dropping the row. A reference to something that left the map is information; deleting the row loses the fact that you once knew about it and when.
  6. Escalate a redacted stored version to review. Whether you may keep content captured from a version that has since been removed is a question for somebody, and the pipeline’s job is to surface it rather than to decide it.
  7. Use a history file at volume. Assessing thousands of references one API call at a time is neither fast nor polite.
What a pipeline should do about a reference whose object disappeared A decision node taking the assessed state and the stored version, with three outcomes. An ordinary deletion means the feature left the map, so the reference is marked as no longer present while the record of having held it is retained. A redaction not covering the stored version means the gap is upstream housekeeping and the reference is simply re-resolved against the current version. A redaction covering the stored version means content was captured from a version since removed, which needs a human decision about retention rather than an automated one. Disappeared — but in which way? Deleted, or redacted? The stored version decides Only one needs a human Deleted Mark as gone, keep the record of having held it Redacted elsewhere Re-resolve; the gap does not touch your version Your version redacted Review: retention of the captured content is a decision Only the third branch needs a human, and conflating it with the first is how a pipeline quietly keeps content that was withdrawn.
The distinction costs one comparison and is the difference between housekeeping and a retention question.
What an object's version sequence looks like in each situation Four version sequences shown side by side. A healthy object runs one, two, three, four with no gaps and a visible final version. A deleted object runs one, two, three, four with the final version marked not visible. A redacted object runs one, two, five, six, with versions three and four absent from the sequence entirely and the final version visible. An object that is both shows gaps and a final version marked not visible. A note adds that only the sequence itself distinguishes the third case. Read the sequence, not just the last version healthy 1 2 3 4 no gaps final version visible nothing to do deleted 1 2 3 4* no gaps final not visible record it left redacted 1 2 _ _ 5 6 gaps present final still visible check your version both 1 _ 3 4* gaps and not visible two facts at once report both A pipeline that reads only the final version sees the first, second and fourth cases and is blind to the third entirely.
The asterisk marks a version flagged not visible; the underscores mark versions that are simply absent.

Verification

  • A known deleted object reports deleted. Pick one and confirm the final version is marked not visible and the history is complete.
  • A gap is detected. Construct or find an object with a missing version and confirm the assessment reports it.
  • An unused identifier reports unknown. Query a very high identifier and confirm it is not reported as deleted.
  • Stored-version logic branches correctly. Assess the same object with a stored version inside and outside the gap and confirm the two recommendations differ.
  • Deleted references are retained, not dropped. Check that the pipeline marks rather than deletes.

Common errors and fixes

Symptom Root cause One-line fix
Redactions never noticed Only current state checked Look for gaps in the version sequence
Unused identifiers reported as deleted Empty history treated as deletion Return an unknown state for a not-found history
Retention question missed Redaction handled the same as deletion Compare the stored version against the missing ones
History of a deletion lost Row dropped when the object disappeared Mark the reference and keep the record
Deleted and redacted conflated One condition checked, not both Test visibility and gaps independently
Assessment too slow at volume One API call per reference Assess in bulk from a history extract
Gap reported on a new object Version numbering assumed to start at one Compare against the observed range, not an assumption

Specification reference

Deleting an OpenStreetMap object creates a new version marked as not visible; the object’s identifier is not reused and its full history remains available. Redaction is a separate administrative action that removes specific object versions from public access, leaving those versions unavailable through the history and data APIs. See the OSM API v0.6 documentation for the visibility flag and the redaction documentation for what a redaction removes and why.

Frequently Asked Questions

How do I detect that a version was redacted?

By looking for a gap in the version sequence. Nothing in the API announces a redaction explicitly; a redacted version simply is not returned, so an object whose history runs 1, 2, 3, 6, 7 has had versions 4 and 5 removed. That structural check is the only signal available, which is why a pipeline that only looks at the current state will never notice a redaction at all.

Is a redaction the same as a revert?

No. A revert is an ordinary edit that creates a new version restoring earlier content, and everything involved stays in the history. A redaction removes versions from public access entirely and cannot be undone by editing, because it is an administrative action about what may be published rather than about what the map should say. The two are frequently confused and need completely different handling.

Should I delete data captured from a redacted version?

That is a decision for somebody rather than for the pipeline, and the pipeline’s job is to surface it. A version is usually redacted because its content could not be licensed or should not have been published, which means data you captured from it may be data you are not entitled to keep. Flagging the affected records for review, rather than either silently keeping or silently deleting them, is the honest handling.

What should happen to a reference whose object was deleted?

Mark it as no longer present and keep the row. Dropping it loses the fact that you once held a reference to that feature and when, which is exactly the information somebody will want when they ask why a count fell. A deletion is also fully explained by the remaining history, so the record can carry who removed it and in which changeset, turning a disappearance into an answerable question.

Up one level: OSM Feature Identity & ID Stability.