Preparing an OSM Import

An import is the only operation on this site whose main risks are not technical. The code is straightforward; what makes imports go wrong is doing them without permission, without discussion, without deduplication, or all at once.

The stages of an import, and how much of it happens before any code runs Four stages spread over time. The permission stage establishes that the source licence allows contribution, obtains a waiver if needed, and records the evidence, and happens before anything else. The plan stage documents the source, the tag mapping, the deduplication approach and the rollout, and is raised with the relevant community for discussion. The preparation stage converts and deduplicates the data and dry-runs the upload against the development instance. The rollout stage uploads area by area with pauses for feedback. Two of the four stages involve no data engineering at all. Two stages before anything technical begins permission licence or waiver evidence recorded plan documented, discussed community input prepare convert and dedupe dry run on dev roll out area by area pause for feedback Imports that get reverted almost always skipped one of the first two stages, not the third.
The engineering is the easy part; the first half of this timeline is what decides whether the import survives.

The Problem This Topic Solves

You have an authoritative dataset that would genuinely improve OpenStreetMap — a municipality’s building footprints, an operator’s stop locations, an agency’s address points — and you want to contribute it. Done well, imports have added enormous value to the map. Done badly, they create years of cleanup for local mappers and a lasting reluctance to accept the next one.

The failure scenario is well documented. A team converts a dataset, uploads it nationally over a weekend, and discovers afterwards that a third of the features already existed and are now duplicated, that the tagging used a scheme the local community abandoned years ago, and that nobody knows which changesets to revert because the upload was one enormous batch. The technical work was fine. Every one of those problems was a missing preparation step.

Prerequisites

Understand the upload mechanics from The OSM Editing API & Changeset Upload. Know the licence framework from OSM Licensing & ODbL Compliance. And have a working matcher from Matching OSM Features to External Datasets, because deduplication is conflation by another name.

Permission Comes First, and It Is Not a Formality

OpenStreetMap can only accept data whose rights holder permits its use under the project’s licence. Three situations qualify: the data is explicitly published under a compatible licence, the rights holder has granted a waiver, or the data is genuinely in the public domain in a way the project recognises.

Several things do not qualify, and all of them are common misunderstandings. Free availability on a website is not permission. An absence of an explicit prohibition is not permission. A permissive-sounding open data portal whose terms require attribution in a form incompatible with the project’s licence is not permission. And a dataset assembled by scraping another map is emphatically not permission — deriving from a proprietary map is the fastest way to have an import reverted and an account blocked.

Where a waiver is needed, it should be obtained in writing, recorded publicly, and referenced from the import’s documentation. That record is what lets somebody in five years confirm the data belongs there.

The Plan, and Why It Is Public

Every bulk import needs a written plan, published where the relevant community can read it, covering:

  • The source: what it is, who publishes it, its licence status and the evidence for it.
  • The tag mapping: every source field and what OSM tagging it becomes, including fields deliberately dropped.
  • The deduplication approach: how existing OSM features will be detected and what happens when one is found.
  • The rollout: which areas, in what order, with what pauses.
  • The revert plan: how the changesets are structured so a mistake can be undone.

Publishing it is not bureaucracy. It surfaces regional tagging conventions you did not know about, finds the local mappers who will review the work, and produces the page every changeset comment can link to. A plan discussed in advance turns a suspicious bulk edit into an expected one.

Three import failure modes and the preparation step that prevents each Three panels. Duplicate features result from uploading without deduplicating against what OSM already contains, leaving two objects for every real thing and a cleanup burden on local mappers. Wrong tagging results from mapping source fields to a scheme the local community does not use, producing data that is technically present but not found by any consumer. An unrevertible upload results from one enormous changeset, so a reviewer who finds one problem must choose between accepting it and destroying everything else. Three failures, three missing preparation steps Duplicates Uploaded without dedupe Two objects per real thing Cleanup falls on locals Prevented by: conflation Before any upload Wrong tagging Mapped to an unused scheme Data present, not findable Consumers never see it Prevented by: discussion Local conventions differ No revert path One enormous changeset Reviewer cannot be selective All or nothing decision Prevented by: small batches Grouped by area All three are cheap to prevent and expensive to fix, and all three are visible to the community long before they are visible to you.
Every one of these is a preparation failure rather than an execution failure.

Who Reviews, and When

A plan that names no reviewers is a plan that will be reviewed after the upload. Identify, before the rollout begins, at least one active local mapper for each area and the channel they prefer, and give them the plan with enough time to read it. The cost is a few days; the alternative is discovering a regional convention through a revert.

Tag Mapping Is a Design Exercise

Converting source fields into OSM tags is where local knowledge matters most. Three principles help.

Map to what is used, not to what is documented. A tagging scheme can be documented on the wiki and effectively unused in a given country. Check what the surrounding data actually carries before committing.

Drop what you cannot justify. A source field with no clear OSM equivalent should not be forced into a made-up key. Dropping it, and saying so in the plan, is far better than introducing a key nobody will ever consume.

Never import identifiers as tags without a reason. An internal record identifier is meaningful to your organisation and noise to everybody else — unless it is a genuinely public reference that consumers will use, in which case the established ref:* namespace is the place for it.

The conversion itself is mechanical, and Converting a Shapefile to OSM XML with ogr2osm covers the tooling.

Deduplication Is Not Optional

The single most damaging import failure is uploading features that already exist. It doubles every affected object, breaks routing and rendering, and leaves a cleanup job that falls entirely on local mappers.

Deduplication is conflation run in a specific direction: for every record you intend to upload, find whether OSM already has it. The matcher from this section applies directly, with one important difference in how the output is used. Here the confident matches are the records you do not upload, the review cases are records that need a human before anything happens, and only the confident non-matches are candidates for creation.

A second, subtler case is a record that matches an existing feature but carries better information. That is not a creation and it may not be an update either — modifying somebody’s surveyed data with an external dataset’s attributes is a decision that belongs in the plan, not in the code. Deduplicating Addresses Before an OSM Import works through the address case, which is the most common and the most error-prone.

What each deduplication outcome means for an import, which is the reverse of an enrichment A grid of three matcher outcomes against what an enrichment does with them and what an import does with them. A confident match means an enrichment attaches the external attributes, while an import skips the record entirely because OSM already has it. A review case means an enrichment usually defers or drops it, while an import must route it to a human before anything happens. A no-match means an enrichment records a gap, while an import treats it as the only category eligible for creation. The same outcomes mean opposite things Enrichment does Import does Confident match attach attributes skip: OSM has it Needs review defer or drop human before upload No match record a gap the only creation case Pointing an enrichment pipeline at an import is how duplicates get created: its confident matches are exactly what must not be uploaded.
Reading this table the wrong way round produces two objects for every real thing, which is the classic import disaster.

Validation and Error Handling

Condition Root cause Detection Remediation
Import halted on licence grounds Permission assumed from availability Community or foundation query Establish permission in writing before any work
Duplicate features created No deduplication pass Two objects per real thing after upload Conflate against existing data; upload only non-matches
Tagging not consumed by anything Scheme documented but unused locally Consumers do not find the data Check surrounding data before fixing the mapping
Import reverted wholesale One enormous changeset A single revert removes everything Split by area into small, single-purpose changesets
Local mappers object after the fact No plan published Objections arrive during rollout Publish and discuss the plan before uploading
Geometry rejected by the API Invalid or unclosed rings in the source Upload errors naming specific objects Validate and repair geometry before conversion
Attribute noise in the map Internal identifiers imported as tags Keys nobody consumes appear everywhere Drop what has no consumer; use ref:* where it does

Rollout: Small, Slow and Reversible

The rollout is where a good plan becomes a good import.

Small means changesets of a few hundred objects, each covering one coherent area. That is the unit a reviewer can evaluate and the unit a revert undoes.

Slow means pausing between areas. The point is not caution for its own sake; it is that objections arriving after the first area can still change the plan, whereas objections arriving after a national upload can only produce a revert.

Reversible means the changesets are structured so that undoing one does not disturb the others, and that the mapping from source records to changesets is recorded. Rolling Back a Bad OSM Import covers what that takes.

Every changeset should carry a comment naming the import, linking to the published plan, and identifying a contact. A reviewer who can read what an edit was for and ask a question about it will do that; one who cannot has only the revert button.

Performance and Scale

Import preparation is not usually compute-bound — the datasets are modest by the standards of the rest of this site — but two things do scale badly.

Deduplication cost grows with the product of the source size and the OSM feature density in the area, and it is the same spatial join discussed throughout this section. Index once, partition by area, and cap candidates.

Review effort is the real constraint. A review queue of a few hundred is workable; one of fifty thousand is not, and an import whose deduplication routes most records to review has not been prepared, it has been deferred. Improving the matcher until the review queue is human-sized is part of preparation, not an optimisation.

Guides in This Topic

Frequently Asked Questions

Is publicly available data automatically importable?

No. Availability and permission are different things. Importing into OpenStreetMap requires the rights holder to permit use under the project’s licence, which means an explicitly compatible licence, a written waiver, or genuine public-domain status. An open data portal whose terms conflict with the project’s licence does not qualify, and data derived from a proprietary map never does. Establish this in writing before any technical work begins.

Do I really need to publish a plan?

For anything bulk, yes — and it is in your own interest rather than a formality. The plan surfaces local tagging conventions you would otherwise get wrong, identifies the mappers who will review the work, and becomes the page every changeset comment links to. An import that arrives without one is indistinguishable from an unreviewed automated edit, and gets treated accordingly.

What happens when a record matches an existing OSM feature?

It is not uploaded as a new object, and what happens instead is a decision that belongs in the plan rather than in the code. Options include leaving the existing feature untouched, adding only attributes it lacks, or routing it to a human. What must not happen is overwriting surveyed data with external attributes by default: a mapper who walked the street usually knows more than a dataset compiled centrally.

How large should an import changeset be?

A few hundred objects covering one coherent area. That is small enough for a reviewer to evaluate in a sitting and small enough that a revert undoes one thing. The technical ceiling is far higher, but changesets are the unit of review and revert, so an enormous one forces a reviewer who finds a single problem to choose between accepting it and destroying everything else in the same upload.

Should I import the source's internal identifiers?

Only when they are genuinely public references that consumers will use, and then in the established reference namespace. An internal record identifier is meaningful inside your organisation and noise on the public map, and once imported it is very hard to remove. If you need to track which OSM object came from which source record, keep that mapping in your own systems, where it belongs.

Up one level: OSM Conflation & Data Enrichment.