Attribute Enrichment from Authoritative Sources

Enrichment is the safe half of conflation: nothing is uploaded, nobody else is affected, and a mistake costs you a wrong column in your own warehouse. That safety is exactly why enrichment pipelines are built carelessly, and why the resulting datasets become impossible to reason about within a year.

How an enriched feature is layered, and why each layer stays separate Four layers making up one enriched record. The OSM layer holds the tags exactly as the map has them, never modified. The external layer holds attributes from the authoritative source in their own namespace, so an external speed limit never overwrites the OSM one. The resolved layer holds the value a consumer should use, produced by an explicit precedence rule. The provenance layer records, per value, where it came from, when, and through which match. A note observes that collapsing these into one set of columns makes every later question unanswerable. Four layers, and only the third is what consumers read OSM values Exactly as the map has them never modified External values Own namespace, own columns never overwrite Resolved value Produced by a precedence rule what consumers read Provenance Source, date and match per value answers every later question Collapsing the first two layers into one is convenient on day one and makes the dataset unexplainable by month three.
Keeping the layers apart costs storage that is free and preserves the ability to change the precedence rule later.

The Problem This Topic Solves

You have OSM data and an authoritative external source that knows something the map does not — verified opening hours, an operator’s official capacity, a regulator’s licence status, a survey’s speed limits. You want your own copy of the data to carry both.

The failure scenario is a warehouse nobody trusts. External values were written into the same columns as OSM values, because that was simplest. Six months later a consumer asks whether a particular speed limit came from the map or from the reference dataset, and there is no way to tell. A second consumer wants OSM’s value because the reference is known to be stale in one region, and it is gone. The pipeline is re-run and the values change, and nobody can say whether that is because the map improved or because the matcher did.

Prerequisites

Have a working matcher from Matching OSM Features to External Datasets and an understanding of the licence position from Deciding if a Derived Database Triggers Share-Alike — combining OSM with an external source may produce a derived database, and the answer shapes what you may publish.

Namespace Separation Is the Whole Design

The single decision that determines whether an enriched dataset stays comprehensible is keeping external values in their own namespace. An external speed limit becomes ref_maxspeed, not maxspeed; an external opening-hours string becomes ref_opening_hours.

Three things follow from that, and each one is why the discipline pays.

Disagreement becomes visible. When both columns exist, a query comparing them is trivial, and the disagreement rate between OSM and an authoritative source is one of the most useful quality signals available — it finds both stale reference data and OSM features that need attention.

Precedence becomes a decision rather than an accident. With one column, whichever write happened last wins. With two, a resolved value is produced by a rule you wrote down and can change without re-running anything upstream.

Re-derivation becomes possible. Because nothing was overwritten, changing the precedence rule is a recomputation over stored columns rather than a full re-run of matching and fetching.

Precedence Rules Worth Writing Down

A precedence rule answers: when both sources have a value and they disagree, which one does a consumer see? Four patterns cover most cases.

Prefer OSM. Reasonable when the map is surveyed and the external source is compiled centrally. A mapper who walked the street usually knows more than a national dataset.

Prefer the external source. Reasonable when the source is genuinely authoritative for that attribute — an operator’s own record of its opening hours beats a guess from a passer-by.

Prefer the more recent. Requires both sides to carry a reliable timestamp, which OSM does at the object level but not per tag, so this is weaker than it looks.

Prefer neither; expose the disagreement. For attributes where being wrong is costly, resolving to null and flagging the conflict is more honest than picking.

The important part is that the rule is per attribute, not per dataset. An external source may be authoritative about licence status and useless about geometry.

Choosing a precedence rule per attribute, with the question each one answers A grid of four precedence rules against when each applies and what it risks. Preferring OSM applies when the map is surveyed locally and risks ignoring a genuinely authoritative source. Preferring the external source applies when that source owns the fact, such as an operator's own opening hours, and risks overwriting better local knowledge. Preferring the more recent value applies when both sides carry reliable timestamps and risks being misled by OSM's object-level rather than tag-level timestamps. Exposing the disagreement applies when being wrong is costly and risks leaving consumers with no value at all. Four rules, chosen per attribute rather than per source Applies when Risks Prefer OSM locally surveyed ignoring an authority Prefer external the source owns the fact overwriting local knowledge Prefer more recent both are timestamped OSM stamps objects, not tags Expose the conflict being wrong is costly consumers get nothing Applying one rule to a whole source is the mistake: a dataset can be authoritative about one attribute and worthless about another.
Writing the rule per attribute also documents what you believe about the source, which is useful when it changes.

Provenance Per Value

Recording where each value came from is the difference between a dataset that can answer questions and one that cannot. The minimum is, per enriched attribute: the source, the date it was fetched, and the match that connected the record to the feature.

That last part matters more than it looks. A value is only as good as the match that delivered it, so a stored match identifier lets a later review of the matcher propagate directly into a review of the values it produced. When a matching bug is found, the affected values are a query rather than an investigation.

Provenance is also what makes the licensing position defensible — the obligations in Recording OSM Data Provenance in a Pipeline apply to enriched data exactly as to raw extracts, and a per-value source record satisfies them almost incidentally.

Three questions a well-built enriched dataset can answer and a collapsed one cannot Three panels. The origin question asks where a particular value came from, which requires a per-value source record and is unanswerable once external and OSM values share a column. The alternative question asks what the other source said, which requires both values to still exist and is unanswerable after an overwrite. The change question asks why a value differs from last month, which requires stored fetch dates and match identifiers and is unanswerable when only the resolved value is kept. Three questions somebody will definitely ask Where from? Which source gave this value Needs per-value provenance Lost when columns merge Asked within weeks What else? What did the other say Needs both values kept Lost on overwrite Asked by the sceptic Why changed? Differs from last month Needs dates and match ids Lost when only resolved kept Asked after an incident Each answer costs a column to preserve and is impossible to reconstruct afterwards, which is why the decision belongs on day one.
The third question is the one that arrives during an incident, when reconstructing the answer is least affordable.

Validation and Error Handling

Condition Root cause Detection Remediation
Cannot tell a value’s origin External values written into OSM columns Provenance query returns nothing Separate namespaces; record source per value
Values change with no data change Precedence resolved at write time Reruns produce different outputs Resolve at read time from stored columns
A stale source overrides good data Precedence set per source, not per attribute Disagreement concentrated in one attribute Set precedence per attribute
Enrichment cannot be undone Values written in place No original value remains Keep OSM values untouched; derive the resolved one
Bad matches silently propagate Match identifier not stored Affected values cannot be identified Store the match id alongside every enriched value
Disagreement rate never examined Both values present but never compared No quality signal from the comparison Report disagreement per attribute every run
Licence position unclear Provenance absent from the output Cannot say what is derived from what Carry source and date through to every consumer

When Enrichment Is the Wrong Tool

Three situations look like enrichment problems and are not, and recognising them saves considerable effort.

The attribute belongs in OSM. If the external source is publishable, the information genuinely improves the map, and the licence permits it, the right destination may be OpenStreetMap itself rather than your warehouse. Enrichment keeps the value private to you; contributing it makes it available to everybody and means you no longer have to maintain the join. The decision is the one framed in Preparing an OSM Import, and it is worth asking before building a permanent enrichment pipeline around a fact that could simply be mapped.

The external source is the primary dataset. Where the external data is more complete, more current and more authoritative than OSM for your purpose, enriching OSM with it has the relationship backwards. The cleaner architecture uses the external dataset as the base and attaches OSM attributes to it, which changes the matching direction, the cardinality question and what a no-match means. Nothing about the techniques changes; the framing does, and getting it wrong produces a dataset whose coverage is limited by whichever side happened to be called the base.

The disagreement is the product. Sometimes the point is not to produce a resolved value at all but to find where the two sources differ — a quality audit, a change-detection job, a survey-targeting exercise. In that case the resolved value is unnecessary and the precedence rule is a distraction; the output is the comparison itself, and treating it as an enrichment pipeline that happens to log discrepancies buries the actual deliverable in a side effect.

Measuring Enrichment Quality

Enrichment has a quality signal that conflation alone does not: the disagreement rate between OSM and the external source on attributes both carry.

A very low rate suggests either that the source adds little or that the matcher is matching only the easy cases. A very high rate suggests either that the source is wrong, that the matcher is wrong, or that the two are measuring different things — an external “capacity” meaning licensed occupancy and an OSM capacity meaning parking spaces, for instance.

Either way the rate is worth tracking per attribute and per region over time. A sudden change almost always means something upstream moved, and it is the cheapest available alarm on an enrichment pipeline.

Performance and Scale

Enrichment cost is dominated by the matching that precedes it, not by the attribute attachment. Two structural choices keep it manageable.

Store matches, not enriched rows. A match table of feature identifiers and record identifiers is small and reusable. Regenerating enriched attributes from it is a join, which is far cheaper than re-running the matcher every time a source refreshes.

Resolve at read time. Computing the resolved value in a view or at query time, rather than materialising it, means a precedence change costs nothing and never leaves stale resolved values behind.

Both point at the same architecture: store the evidence, derive the answer. The alternative — materialising everything at write time — is faster to query and impossible to correct.

Guides in This Topic

Frequently Asked Questions

Why not write external values into the OSM tag columns?

Because it destroys three things at once: the ability to tell where a value came from, the ability to change the precedence rule later, and the ability to use the disagreement between the two as a quality signal. Storage is cheap and columns are free; the information you lose by collapsing them is not recoverable without re-running the whole pipeline, and sometimes not even then if the source has changed in the meantime.

Should precedence be set per source or per attribute?

Per attribute. A source can be authoritative about one thing and unreliable about another — an operator’s own record of its opening hours is better than anything a passer-by could observe, while its idea of where the building is may be a geocoded postcode. Applying one rule to a whole source guarantees that its good attributes and its bad ones are treated identically, which is exactly what you are trying to avoid.

What is the disagreement rate good for?

It is the cheapest quality alarm an enrichment pipeline has. Tracked per attribute and per region over time, a sudden change means something moved — the source refreshed with different semantics, the matcher regressed, or the map changed substantially in one area. A persistently high rate on one attribute usually means the two sides are measuring different things and the mapping needs revisiting rather than the data being wrong.

Should the resolved value be materialised or computed on read?

Computed on read, wherever query performance allows it. Materialising means every precedence change requires a rewrite and leaves the risk of stale resolved values sitting beside fresh inputs. A view or a query-time expression over the stored OSM and external columns costs a little at read time and makes the precedence rule something you can change in one place, review, and change back.

Does enrichment have licensing implications?

It can. Combining OpenStreetMap data with an external dataset in your own systems may produce what the licence calls a derived database, and what you are then permitted to publish depends on how the two are combined and what is distributed. The question is worth settling before the architecture is fixed, because a per-value provenance record — which you want anyway — is most of what a defensible answer requires.

Up one level: OSM Conflation & Data Enrichment.