Changeset Analysis & Vandalism Detection

The validation work covered elsewhere in OSM Data Quality & Validation asks whether an object is well-formed. This topic asks a different question: whether an edit looks like one a careful mapper would make. The two are independent. A changeset that deletes four thousand buildings across three countries produces perfectly valid geometry and perfectly consistent tags, and every rule engine on this site will pass it without comment.

Detecting that class of problem means looking at the diff stream rather than at the data, and treating the changeset — not the object — as the unit of analysis. It also means being clear-eyed about what the pipeline is for. Most edits that trip these signals are not vandalism; they are imports, mechanical edits, licence-driven removals and bulk retagging, all legitimate and all indistinguishable from malice by any arithmetic. The output of this pipeline is a ranked queue for a person to look at, and building it any other way produces a system that reverts good work.

From the diff stream to a ranked review queue A four-stage chain. The diff stream delivers edits as .osc.gz files each minute. A feature stage computes per-changeset size, geographic spread and account age using arithmetic only. A scoring stage combines them into a weighted, explainable score rather than a single opaque number. The result is a triage queue ranked for a human, with nothing automatically reverted. Signals are cheap; the judgement is not diff stream the edits, as they land osc.gz per minute per-changeset features size · spread · account age arithmetic only score weighted, explainable never a single opaque number triage queue ranked for a human nothing auto-reverted The output of this pipeline is a ranked queue, not a verdict. Automated reverting of OSM edits is a community decision, not a pipeline one.
Every stage before the last is mechanical. The last is deliberately not: the pipeline ranks, and a person decides.

Prerequisite concepts

This topic sits on the replication machinery. The edits arrive as OsmChange documents from the stream described in OSM Replication & Diff Sync, and the per-object metadata each operation carries — changeset identifier, user, version, timestamp — is what makes changeset-level grouping possible at all. The fields the diff does not carry, notably the changeset comment and the editor string, come from the changeset API, a split covered in Extracting Changeset Metadata from History Files.

The signals worth computing

Six changeset signals and the bands that separate routine from suspicious A grid of six signals with three bands each. Objects touched: one to two hundred is routine, two hundred to five thousand is worth a look, above five thousand is high suspicion. Deletions as a share of the changeset: under five percent routine, five to forty percent worth a look, above forty percent suspicious. Geographic spread: one town routine, one country worth a look, multiple continents suspicious. Account age at the time of the edit: over a year routine, one to thirty days worth a look, under a day suspicious. Editor used: JOSM or iD routine, an unknown string or none declared worth a look. Tag churn on a single object: one to two versions routine, three to ten worth a look, more than ten in an hour suspicious. The same signal means different things at different scales likely routine worth a look high suspicion objects touched 1–200 200–5 000 over 5 000 deletions as a share under 5% 5–40% over 40% geographic spread one town one country multiple continents account age at edit over 1 year 1–30 days under 1 day editor used JOSM, iD an unknown string none declared tag churn on one object 1–2 versions 3–10 over 10 in an hour No single row is evidence. A large, wide, deletion-heavy changeset from a day-old account is — and that conjunction is what a score exists to express.
Large imports trip four of these six rows and are entirely legitimate. That is exactly why the output is a queue rather than a revert.

Six signals do most of the work, and all six are arithmetic over a grouped diff.

Size — the count of objects touched. Cheap, and by itself almost meaningless: routine imports are enormous and a malicious edit can be three objects.

Deletion share — the proportion of operations that are deletions. This is the strongest single signal, because ordinary mapping is overwhelmingly additive and corrective. A changeset that is eighty percent deletions is doing something unusual whether or not it is malicious.

Geographic spread — the diagonal of the changeset bounding box, or better, the number of distinct H3 cells touched at a coarse resolution, using the cell scheme from Spatial Index Selection. A human editing in one session works in one place; an edit spanning continents is either a bot or a script.

Account age at edit time — the interval between account creation and the changeset. New accounts making large edits are worth attention, and this signal is also the one most likely to be unfair: new mappers exist and are welcome. Weight it low.

Editor string — created_by on the changeset. Absent or unrecognised values correlate with scripted edits. This is a weak signal and a noisy one; treat it as a tiebreaker.

Tag churn — the number of versions an object accumulates within a short window. Repeated rewriting of the same object is characteristic of an edit war rather than of vandalism, and it is worth surfacing separately because the response is different.

Computing them from a diff

python 
import logging
from collections import defaultdict
from dataclasses import dataclass, field
from datetime import datetime

import h3
import osmium

logger = logging.getLogger(__name__)


@dataclass
class ChangesetStats:
    """Everything the score needs, accumulated in one pass over the diff."""
    changeset: int
    user: str = ""
    created: int = 0
    modified: int = 0
    deleted: int = 0
    cells: set[str] = field(default_factory=set)
    first_seen: datetime | None = None
    last_seen: datetime | None = None

    @property
    def touched(self) -> int:
        return self.created + self.modified + self.deleted

    @property
    def deletion_share(self) -> float:
        return self.deleted / self.touched if self.touched else 0.0


class ChangesetCollector(osmium.SimpleHandler):
    """Group an OsmChange stream by changeset, accumulating scoring features."""

    def __init__(self, cell_resolution: int = 3) -> None:
        super().__init__()
        self.resolution = cell_resolution
        self.stats: dict[int, ChangesetStats] = defaultdict(
            lambda: ChangesetStats(changeset=0)
        )

    def _record(self, obj, lat: float | None = None, lon: float | None = None) -> None:
        st = self.stats[obj.changeset]
        st.changeset = obj.changeset
        st.user = obj.user or st.user
        if not obj.visible:
            st.deleted += 1
        elif obj.version == 1:
            st.created += 1
        else:
            st.modified += 1
        if lat is not None and lon is not None:
            st.cells.add(h3.latlng_to_cell(lat, lon, self.resolution))
        ts = obj.timestamp
        st.first_seen = min(st.first_seen or ts, ts)
        st.last_seen = max(st.last_seen or ts, ts)

    def node(self, n) -> None:
        loc = n.location if n.visible and n.location.valid() else None
        self._record(n, loc.lat if loc else None, loc.lon if loc else None)

    def way(self, w) -> None:
        self._record(w)

    def relation(self, r) -> None:
        self._record(r)

Two details are load-bearing. Deletions are identified by visible being false rather than by the operation block, because a <delete> block carries only the stub — the semantics set out in Applying .osc Change Files with osmium. And spread is measured in coarse H3 cells rather than as a bounding-box diagonal, because a bounding box around two edits on opposite sides of a country reports a huge area for two objects, whereas a cell count reports two.

Scoring, and why it must be explainable

python 
WEIGHTS = {
    "large": 2.0,          # over 5 000 objects
    "deletion_heavy": 3.0,  # over 40% deletions
    "wide": 2.0,           # more than 12 coarse cells
    "very_new_account": 1.5,
    "unknown_editor": 0.5,
}

def score(st: ChangesetStats, account_age_days: float, editor: str | None) -> tuple[float, list[str]]:
    """Return a score and the list of reasons that produced it."""
    reasons: list[str] = []
    total = 0.0
    if st.touched > 5_000:
        reasons.append(f"large: {st.touched} objects"); total += WEIGHTS["large"]
    if st.deletion_share > 0.40:
        reasons.append(f"deletion-heavy: {st.deletion_share:.0%}"); total += WEIGHTS["deletion_heavy"]
    if len(st.cells) > 12:
        reasons.append(f"wide: {len(st.cells)} cells"); total += WEIGHTS["wide"]
    if account_age_days < 1:
        reasons.append("account under a day old"); total += WEIGHTS["very_new_account"]
    if not editor:
        reasons.append("no editor declared"); total += WEIGHTS["unknown_editor"]
    return total, reasons

Returning the reasons alongside the number is not a nicety. A reviewer opening a queue item needs to know in one second why it is there, and a score with no explanation gets treated as noise. It is also the only way to tune: when the queue fills with false positives, the reason list tells you which weight to move.

How a week of changesets distributes across score bands A bar chart of 41 200 changesets from one week of a country diff stream. 39 100 score zero and are never surfaced. 1 840 score one or two, tripping a single weak signal, and are counted but not queued. 218 score three to five and are worth a look, roughly 31 a day. 34 score six or more and are reviewed the same day. Of those, six are confirmed problematic after human review. What a scored queue actually contains one week of a country diff stream, 41 200 changesets scored 0 — no signal tripped 39 100 · never surfaced scored 1–2 — one weak signal 1 840 · counted, not queued scored 3–5 — worth a look 218 · ~31/day for a human scored 6+ — high suspicion 34 · reviewed same day confirmed problematic 6 · after human review The ratio that matters is the last two rows: thirty-four flagged, six real. A review queue with that hit rate gets worked; one at one in a hundred does not.
Tune the thresholds against the last row. A queue whose confirmed rate falls below roughly one in ten stops being read, and an unread queue detects nothing.

Validation and error-handling matrix

Condition Root cause Detection Action
Queue full of legitimate imports Size weighted too heavily Reason lists dominated by “large” Lower the size weight; add an import allowlist
Deletion share always zero visible not checked; block type used instead No changeset ever flags on deletions Read visible, not the operation block
Spread always 1 cell Ways and relations have no location Only node-only changesets score on spread Resolve way geometry, or accept node-only spread
Every changeset scores 0 Account age or editor unavailable, treated as safe Two reasons never appear Treat unknown as unknown, not as clean
Same changeset queued repeatedly Changesets span multiple diffs Duplicate queue entries Key the queue on changeset id, upsert
Queue ignored by reviewers Confirmed rate too low Items ageing without action Raise thresholds until the hit rate recovers

The fifth row is a structural property of the stream rather than a bug: a large changeset is applied across many minutely diffs, so its statistics arrive in pieces. Accumulate into a store keyed by changeset identifier and score on a delay of an hour or so, rather than scoring each diff independently.

Performance and scale considerations

The grouping pass is cheap — one pass over the diff, a dictionary keyed by changeset, and a small set of H3 cells per group. On a minutely diff it is microseconds of work against an HTTP fetch. The cost lives in two other places.

The first is changeset metadata. Account age and the editor string are not in the diff, so they require an API call per changeset. At a few hundred distinct changesets a minute that is a lot of calls, and the fix is a cache: account creation dates never change, and changeset metadata is immutable once the changeset closes. A local table keyed by user identifier removes almost all of the traffic.

The second is the accumulation store if it is unbounded. Changesets stay open for up to a day, so an in-memory dictionary that never evicts grows all day. Flush and score groups whose last-seen timestamp is more than an hour old, the same bounded-accumulator pattern as in Bounded LRU Node Cache for OSM Streaming.

Failure modes and gotchas

The failure with real-world consequences is treating the score as a verdict. Automated reverting is a socially governed action in OpenStreetMap, it has an established process, and a pipeline that reverts on a threshold will eventually revert an import a local community spent months coordinating. Rank, explain, and hand to a person.

A quieter failure is scoring on a stream you have already filtered. If your diff-sync applies a geographic clip before scoring, every changeset looks narrow, because you deleted the evidence of spread. Score the unfiltered stream and filter afterwards.

Third, be careful with account age as a signal in isolation. It is the signal most likely to systematically flag new contributors, whose edits are usually small, local and correct. Weighted low and combined with size and deletion share it adds information; used alone it produces a queue of newcomers.

Calibrating the weights

Weights chosen by intuition produce a queue that is either empty or unreadable, and the only way to find out which is to run them against edits whose outcome is already known. A full-history file makes that possible: replay a year of changesets, score each one with a candidate weighting, and compare the top of the resulting queue against the changesets that were in fact reverted. Reverts are recorded in OSM as ordinary changesets whose comments follow recognisable conventions, so the ground truth is available without any manual labelling.

The metric to optimise is not accuracy, which is meaningless when the positive class is a fraction of a percent of the population. It is the confirmed rate within the number of items a reviewer will actually open in a day. If a reviewer works thirty items, the question is how many of the top thirty by score were genuinely problematic, and a weighting that scores twelve of them correctly is far more useful than one with better overall separation and three.

That framing has a consequence for thresholds. The score threshold should be set from reviewer capacity rather than from the score distribution: measure how many items a day get worked, set the cut-off so the queue holds about that many, and let the weights determine which ones they are. A queue that grows faster than it is worked is a queue that gets abandoned, and every signal in it stops mattering.

Recalibrate on a schedule rather than on incident. Tagging conventions shift, import campaigns start and finish, and a weighting tuned on last year’s edits drifts. A quarterly replay against the previous quarter’s history, comparing the queue that would have been produced against what actually needed attention, is enough to catch the drift while it is still small.

One further note on fairness. Every signal here is a proxy, and proxies encode assumptions about how mapping is done. Account age assumes established contributors are more trustworthy, which is usually true in aggregate and routinely wrong in the individual case — a new account may belong to an experienced mapper starting fresh, or to a local expert invited by a community group. Editor strings assume familiar tools mean familiar practice, which under-weights regional editors popular outside Europe and North America. Geographic spread assumes a session happens in one place, which is false for anyone doing armchair mapping from imagery.

None of that makes the signals useless; it makes them signals rather than judgements, and it is another reason the output is a queue. A reviewer looking at a flagged changeset can see immediately that a new account made a large local edit with an unfamiliar editor and recognise a mapping party rather than an attack. A threshold cannot.

In this section

Frequently Asked Questions

Should the pipeline revert anything automatically?

No. Reverting in OpenStreetMap is a community process with established conventions, and the signals here cannot distinguish a malicious mass deletion from a coordinated licence removal or a planned import cleanup. Produce a ranked, explained queue. If a reviewer decides a revert is warranted, they perform it through the normal channels.

What is the single most useful signal?

Deletion share. Ordinary mapping is overwhelmingly additive, so a changeset dominated by deletions is unusual by construction, and unlike raw size it does not flag every import. Combined with geographic spread it identifies the shape of edit that most warrants a look.

How do I avoid flagging legitimate imports?

Maintain an allowlist of accounts and editor strings known to perform coordinated imports in your area of interest, and subtract from the score rather than skipping the changeset entirely. Subtracting keeps the changeset visible if it also trips unrelated signals, which is what you want when an import account behaves unusually.

Can this run on a full-history file instead of the diff stream?

Yes, and it is the right way to backfill or to tune weights, because a history file lets you replay months of changesets and measure how a weighting would have performed. The streaming path is for detection; the history path is for calibration. The reduction is the one described in Reconstructing OSM Features at a Past Date.

Up one level: OSM Data Quality & Validation.