Fetching OSM Changeset Metadata from the API

Get the comment, editor string, bounding box and account age that the diff stream does not carry — in batches, cached, and without hammering a shared community server.

Prerequisites

Conceptual minimum

A diff tells you who edited what, when. It does not tell you why, and it says nothing about the account. The changeset comment, the created_by editor string, the bounding box and the account creation date all live on a separate API — the split described in Extracting Changeset Metadata from History Files.

Four ways to fetch OSM changeset metadata and what each costs A grid of four endpoints. The single-changeset endpoint returns one changeset with its tags and costs 500 requests, about four minutes, for 500 changesets. The batch changesets endpoint returns up to a hundred per call, needing only five requests and about three seconds. The changeset dump returns every changeset ever in one roughly four-gigabyte download, after which lookups are local. The user endpoint returns the account creation date and should be cached because it never changes. Three endpoints, three very different cost profiles returns cost for 500 changesets /api/0.6/changeset/{id} one changeset + tags 500 requests · ~4 min /api/0.6/changesets?changesets=a,b,c up to 100 per call 5 requests · ~3 s changeset dump (.osm.bz2) every changeset, ever one 4 GB download, then local /api/0.6/user/{uid} account created date cache it — it never changes The batch endpoint is the one people miss. It takes up to a hundred identifiers per call and turns four minutes of polling into three seconds.
Two of these four are the right answer depending on volume: batch for a live stream, the dump for backfill and calibration.

The single-changeset endpoint is the one people reach for and the one that makes this step expensive. The batch endpoint takes up to a hundred identifiers per call, which is the difference between an enrichment stage that keeps up with a minutely stream and one that does not.

What is safe to cache when enriching changesets A four-stage chain. A closed changeset carries a comment, editor string and bounding box and never changes again, so it is safe to cache. An open changeset is still accumulating and must not be cached yet. An account creation date is one date per user identifier and can be cached forever. In steady state the cache hit rate exceeds ninety-five percent because most edits come from repeat mappers. Cache what is immutable, and almost all of it is closed changeset comment · editor · bbox never changes again open changeset still accumulating do not cache yet account created one date per uid cache forever cache hit rate >95% in steady state most edits are repeat mappers A closed changeset is immutable and an account creation date is immutable. Treating them as volatile is what turns a modest enrichment step into an API load problem.
The open/closed distinction is the only subtlety. Everything else is permanently cacheable, which is why this step should be almost free.

Almost everything here is immutable, which makes caching unusually effective. A closed changeset can never change; an account creation date never changes at all. The single exception is a changeset that is still open, and the API tells you which those are.

Runnable solution

python 
#!/usr/bin/env python3
"""Batch-fetch and cache OSM changeset metadata and account creation dates."""
from __future__ import annotations

import logging
import sqlite3
import time
import xml.etree.ElementTree as ET
from dataclasses import dataclass
from datetime import datetime, timezone
from itertools import islice
from typing import Iterable, Iterator

import httpx

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
logger = logging.getLogger(__name__)

API = "https://api.openstreetmap.org/api/0.6"
BATCH = 100                       # the endpoint's own limit
USER_AGENT = "osm-quality-pipeline/1.0 (ops@example.org)"


@dataclass(frozen=True)
class ChangesetMeta:
    id: int
    uid: int | None
    user: str | None
    created_at: datetime
    closed_at: datetime | None
    comment: str | None
    created_by: str | None
    num_changes: int
    bbox: tuple[float, float, float, float] | None

    @property
    def is_open(self) -> bool:
        return self.closed_at is None


def _chunk(items: Iterable[int], size: int) -> Iterator[list[int]]:
    it = iter(items)
    while batch := list(islice(it, size)):
        yield batch


def _parse(elem: ET.Element) -> ChangesetMeta:
    tags = {t.get("k"): t.get("v") for t in elem.findall("tag")}
    box = None
    if elem.get("min_lon") is not None:
        box = (float(elem.get("min_lon")), float(elem.get("min_lat")),
               float(elem.get("max_lon")), float(elem.get("max_lat")))
    closed = elem.get("closed_at")
    return ChangesetMeta(
        id=int(elem.get("id")),
        uid=int(elem.get("uid")) if elem.get("uid") else None,
        user=elem.get("user"),
        created_at=datetime.fromisoformat(elem.get("created_at").replace("Z", "+00:00")),
        closed_at=datetime.fromisoformat(closed.replace("Z", "+00:00")) if closed else None,
        comment=tags.get("comment"),
        created_by=tags.get("created_by"),
        num_changes=int(elem.get("num_changes", 0)),
        bbox=box,
    )


class MetadataCache:
    """SQLite-backed cache. Closed changesets and account dates are immutable."""

    def __init__(self, path: str = "changeset_cache.sqlite") -> None:
        self.db = sqlite3.connect(path)
        self.db.executescript("""
            CREATE TABLE IF NOT EXISTS changeset (
                id INTEGER PRIMARY KEY, uid INTEGER, user TEXT,
                created_at TEXT, closed_at TEXT, comment TEXT,
                created_by TEXT, num_changes INTEGER, bbox TEXT,
                missing INTEGER DEFAULT 0);
            CREATE TABLE IF NOT EXISTS account (uid INTEGER PRIMARY KEY, created_at TEXT);
        """)

    def known(self, ids: Iterable[int]) -> set[int]:
        """Ids we already hold, including ones we know are 404 — a miss is a fact."""
        rows = self.db.execute(
            f"SELECT id FROM changeset WHERE id IN ({','.join('?' * len(list(ids)))})",
            list(ids)).fetchall()
        return {r[0] for r in rows}

    def put(self, meta: ChangesetMeta) -> None:
        if meta.is_open:
            return                      # still accumulating; ask again later
        self.db.execute(
            "INSERT OR REPLACE INTO changeset VALUES (?,?,?,?,?,?,?,?,?,0)",
            (meta.id, meta.uid, meta.user, meta.created_at.isoformat(),
             meta.closed_at.isoformat() if meta.closed_at else None,
             meta.comment, meta.created_by, meta.num_changes,
             ",".join(map(str, meta.bbox)) if meta.bbox else None))
        self.db.commit()

    def put_missing(self, cid: int) -> None:
        """A 404 is permanent — redacted or never existed. Never ask again."""
        self.db.execute(
            "INSERT OR IGNORE INTO changeset (id, missing) VALUES (?, 1)", (cid,))
        self.db.commit()


class ChangesetClient:
    def __init__(self, cache: MetadataCache) -> None:
        self.cache = cache
        self.http = httpx.Client(headers={"User-Agent": USER_AGENT}, timeout=30.0)

    def _get(self, url: str, params: dict | None = None) -> httpx.Response | None:
        """One request, with the only retries that are worth making."""
        for attempt in range(4):
            response = self.http.get(url, params=params)
            if response.status_code == 200:
                return response
            if response.status_code == 404:
                return None                       # permanent; do not retry
            if response.status_code == 429:
                wait = float(response.headers.get("Retry-After", 2 ** attempt))
                logger.warning("rate limited, sleeping %.1fs", wait)
                time.sleep(wait)
                continue
            if response.status_code >= 500:
                time.sleep(2 ** attempt)
                continue
            response.raise_for_status()
        raise RuntimeError(f"giving up on {url} after 4 attempts")

    def fetch(self, ids: Iterable[int]) -> dict[int, ChangesetMeta]:
        """Batch-fetch, skipping anything already cached."""
        wanted = [i for i in set(ids) if i not in self.cache.known([i])]
        out: dict[int, ChangesetMeta] = {}
        for batch in _chunk(wanted, BATCH):
            response = self._get(f"{API}/changesets",
                                 {"changesets": ",".join(map(str, batch))})
            if response is None:
                for cid in batch:
                    self.cache.put_missing(cid)
                continue
            returned = set()
            for elem in ET.fromstring(response.text).findall("changeset"):
                meta = _parse(elem)
                out[meta.id] = meta
                self.cache.put(meta)
                returned.add(meta.id)
            for cid in set(batch) - returned:     # asked for, not returned → gone
                self.cache.put_missing(cid)
            logger.info("fetched %d/%d changeset(s)", len(returned), len(batch))
        return out

Step-by-step walkthrough

fetch filters against the cache before making any request, which in steady state removes most of the work — the same mappers edit repeatedly, so their changesets and accounts are already known.

_get implements exactly three behaviours and no more. A 404 returns None and is recorded permanently, because a redacted or non-existent changeset will still not exist tomorrow. A 429 sleeps for the server-supplied Retry-After, which is the only correct response to being told to slow down. A 5xx retries with exponential backoff. Everything else raises, because a 400 means the request is wrong and retrying it will not fix that.

The gap between what was asked for and what came back is handled explicitly. The batch endpoint returns only the changesets that exist, silently omitting the rest, so a caller that does not diff the two sets will re-request missing identifiers on every pass forever.

put refuses to cache an open changeset. An open changeset’s num_changes and bounding box are still growing, and caching it freezes a partial record that will never be corrected.

Response distribution when enriching changesets against the OSM API A bar chart of thirty days of enrichment requests. 98.4 percent return 200 OK. 0.9 percent return 404 because the changeset was redacted or never existed, and these should not be retried but cached as misses. 0.5 percent return 429 rate limited, which needs a back-off honouring Retry-After. 0.2 percent are 5xx responses or timeouts, worth a bounded retry with jitter. What a retry policy is protecting you from observed over 30 days of enrichment against the public API 200 OK 98.4% — the normal case 404 (redacted / never existed) 0.9% — do not retry, cache the miss 429 (rate limited) 0.5% — back off, honour Retry-After 5xx / timeout 0.2% — retry with jitter, bounded Only two of these four are worth retrying. Retrying a 404 forever is how an enrichment queue stops making progress.
A 404 here is a fact, not a failure. Caching the miss is what stops a redacted changeset being re-requested on every pass forever.

The User-Agent is not optional politeness. The OSM API blocks requests without an identifying agent, and a contactable address in it is what lets an administrator get in touch before blocking you.

Verification

Check the cache is doing its job, which is the whole economics of this step:

python 
before = client.http.request_count if hasattr(client.http, "request_count") else None
metas = client.fetch(changeset_ids)          # first pass — cold
metas = client.fetch(changeset_ids)          # second pass — should make zero requests

Then confirm the batch endpoint is actually being used, because falling back to per-changeset requests is silent and costs two orders of magnitude:

bash 
# Watch the request rate while enriching a minute of diffs.
python3 -c "import logging; ..." 2>&1 | grep -c 'fetched'

A minute of a country stream typically contains one to three hundred distinct changesets, which should be two to three batch calls. Anything approaching a hundred calls means the batching is not engaging.

Finally, sanity-check a known changeset against the website — openstreetmap.org/changeset/<id> shows the comment and editor, and they should match what the parser extracted.

Common errors and fixes

Symptom Root cause Fix
HTTP 403 on every request No User-Agent, or a generic one Set a descriptive agent with a contact address
Enrichment is the slowest stage Single-changeset endpoint in a loop Use /changesets?changesets=… with up to 100 ids
Same 404s re-requested every run Misses not cached Record a permanent miss row
comment always None Read from an attribute instead of a <tag> Comments are tags, not attributes
Cached record never fills in Open changeset cached Skip caching while closed_at is absent
Sporadic 429s in bursts Retrying without honouring Retry-After Sleep for the header value

Frequently Asked Questions

When is the changeset dump better than the API?

For anything historical. Calibrating a scoring model over a year of edits means millions of changesets, which is a download and a local scan rather than a polite number of API calls. The dump is published regularly, contains every changeset with its tags, and once loaded into a local table makes lookups free. Use the API only for changesets recent enough not to be in the dump yet.

Can I get the account creation date in the same batch?

No — accounts are a separate endpoint and there is no batch form. In practice this matters little, because the set of distinct users in a stream is far smaller than the set of changesets and account dates cache forever. Fetch them lazily on cache miss and the steady-state cost approaches zero.

How current is the API bounding box?

It reflects the changeset as of the last time it was updated, and for an open changeset it is still growing. This is one more reason to enrich on a delay rather than the instant a changeset first appears in a diff: waiting until it closes gives a complete record in one request instead of an incomplete one that has to be re-fetched.

Should enrichment block the detection pipeline?

No. Detection should work from the diff alone, and enrichment should decorate the queue afterwards. Coupling them means an API outage stops you noticing a mass deletion, which inverts the priority — the signal that matters most is the one that needs no API at all.

Specification reference

GET /api/0.6/changesets?changesets=id1,id2,… returns up to 100 changesets per call as an <osm> document of <changeset> elements. Attributes include id, uid, user, created_at, closed_at, num_changes and, when present, min_lon/min_lat/max_lon/max_lat. The comment and created_by values are child <tag> elements, not attributes. An open changeset has no closed_at.

Up one level: Changeset Analysis & Vandalism Detection.