Running OSM Validation in GitHub Actions Jump to heading#

A validation suite that only runs on somebody’s laptop protects nothing, and one that downloads a continental extract on every pull request protects nothing either because it will be turned off within a fortnight.

Prerequisites Jump to heading#

• A repository with the validation rules, per Writing Custom OSM Validation Rules in Python.
• A small committed fixture extract — a city, not a country — under a few megabytes.
• The check ordering from Continuous QA for OSM Pipelines.
• A runner with enough disk for the scheduled job’s extract, which the default runner often lacks.
• Somewhere to keep metric history between runs, since relative checks need it.

Conceptual minimum Jump to heading#

The workflow splits along a single axis: what runs on every change, and what runs against real data.

The pull-request job answers “did this change break the checks?”. It runs against a committed fixture, takes under two minutes, and blocks the merge. Its correctness depends on the fixture being representative enough to exercise the rules — which means a fixture chosen for its awkwardness, containing the multipolygon with a missing role and the relation that nests, rather than a tidy city centre.

The scheduled job answers “is the real data still healthy?”. It runs against an actual extract, takes as long as it takes, and does not block anything because there is nothing to block. Its output is a report and, on failure, an issue or an alert.

Two mechanics make both affordable. Caching the downloaded extract keyed on the upstream file’s date avoids re-downloading unchanged inputs, which on a several-gigabyte file is most of the job’s runtime. Artifacts carry the metrics document out of the run so the next scheduled execution can compare against it, which is the cheapest available history store for a pipeline that does not have a database to hand.

Runnable solution Jump to heading#

name: osm-validation

on:
  pull_request:
  schedule:
    - cron: "17 4 * * *"     # off the hour: shared runners are busiest at :00
  workflow_dispatch:

permissions:
  contents: read
  issues: write              # only the scheduled job needs this

jobs:
  fixture:
    name: Rules against the committed fixture
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements-dev.txt
      - name: Structural and rule checks
        run: |
          python -m osmqa.validate \
            --input tests/fixtures/awkward-city.osm.pbf \
            --rules rules/ \
            --metrics-out fixture-metrics.json \
            --fail-on structural,rules
      - name: Summarise
        if: always()
        run: python -m osmqa.summarise fixture-metrics.json >> "$GITHUB_STEP_SUMMARY"

  real-data:
    name: Statistical checks against a real extract
    if: github.event_name != 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 90
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12", cache: pip }
      - run: pip install -r requirements-dev.txt

      - name: Resolve the upstream extract date
        id: upstream
        run: |
          date=$(curl -sI "$EXTRACT_URL" | awk 'tolower($1)=="last-modified:"{print $4$3$5}')
          echo "date=${date:-unknown}" >> "$GITHUB_OUTPUT"
        env:
          EXTRACT_URL: ${{ vars.EXTRACT_URL }}

      # Key on the upstream date, not on a run number: an unchanged extract
      # is the common case and re-downloading it is most of this job.
      - name: Cache the extract
        id: cache
        uses: actions/cache@v4
        with:
          path: data/extract.osm.pbf
          key: extract-${{ steps.upstream.outputs.date }}

      - name: Download if the cache missed
        if: steps.cache.outputs.cache-hit != 'true'
        run: |
          mkdir -p data
          curl -fsSL --retry 3 -o data/extract.osm.pbf "$EXTRACT_URL"
        env:
          EXTRACT_URL: ${{ vars.EXTRACT_URL }}

      # Metric history lives in artifacts: the cheapest store available to a
      # pipeline with no database of its own.
      - name: Restore previous metrics
        uses: actions/download-artifact@v4
        continue-on-error: true
        with: { name: osm-metrics, path: history }

      - name: Validate
        id: validate
        run: |
          python -m osmqa.validate \
            --input data/extract.osm.pbf \
            --rules rules/ \
            --history history/metrics.json \
            --metrics-out metrics.json \
            --report-out report.md \
            --fail-on structural,critical

      - name: Publish the report
        if: always()
        run: cat report.md >> "$GITHUB_STEP_SUMMARY"

      # Only record history from a PASSING run. Metrics from a degraded run
      # teach the rolling band that the degradation is normal.
      - name: Record metrics as history
        if: success()
        uses: actions/upload-artifact@v4
        with:
          name: osm-metrics
          path: metrics.json
          retention-days: 90
          overwrite: true

      - name: Open an issue on failure
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            const body = require('fs').readFileSync('report.md', 'utf8');
            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `OSM validation failed (run ${context.runNumber})`,
              body,
              labels: ['osm-quality'],
            });

Step-by-step walkthrough Jump to heading#

1. Separate the two jobs by trigger. The if: github.event_name != 'pull_request' guard is what keeps a pull request from waiting on a continental download.
2. Commit an awkward fixture. A tidy extract passes everything; the useful fixture contains the cases the rules exist for, and it belongs in version control beside them.
3. Key the cache on the upstream file’s date. Keying on a run identifier never hits; keying on the date hits on every run where the extract has not changed, which is most of them.
4. Fail the fixture job on rule failures. This is the part that blocks a merge, and it should be strict because it runs against data you control.
5. Fail the real-data job only on structural and critical checks. A statistical wobble in real data on a Tuesday is not a reason to open an incident.
6. Restore history with continue-on-error. The first run has no previous artifact, and a workflow that fails on its own first execution gets deleted.
7. Record history only from passing runs. Otherwise a degraded run establishes the degradation as the new normal, which defeats the entire relative-check layer.
8. Write to the step summary. A report nobody has to click through to is read considerably more often than one they do.

Verification Jump to heading#

• A broken rule fails the pull request. Introduce a deliberate rule failure and confirm the merge is blocked.
• The cache hits. Run the scheduled job twice without an upstream change and confirm the second skips the download.
• History is compared. Confirm the second scheduled run reports deltas against the first.
• A failing run does not poison history. Force a failure and confirm no artifact is uploaded.
• The summary is readable. Open the run’s summary page and confirm the report renders without needing an artifact download.

Common errors and fixes Jump to heading#

Specification reference Jump to heading#

actions/cache restores an entry whose key matches exactly, falling back to restore-keys prefixes when no exact match exists; a cache entry is immutable once written under a key. Workflow runs triggered by schedule use the default branch’s workflow file. Writing to the file named by GITHUB_STEP_SUMMARY renders Markdown on the run’s summary page. See the GitHub Actions documentation for caching, events that trigger workflows, and job summaries.

Frequently Asked Questions Jump to heading#

Related Jump to heading#

• Continuous QA for OSM Pipelines — the parent topic.
• Setting Quality Thresholds That Fail a Build — what --fail-on critical actually evaluates.
• Generating an OSM Data Quality Report — what goes into the step summary.
• Writing Custom OSM Validation Rules in Python — the rules the fixture job runs.
• Mirroring OSM Downloads Behind a Local Cache — a better source than the public provider for repeated CI downloads.

Up one level: Continuous QA for OSM Pipelines.