Scheduling OSM Diff Sync with systemd Timers

Run the minutely OSM sync loop reliably on a Linux host — restarted after reboots, logged to journald, isolated when it fails, and never running two overlapping copies at once — using a systemd service and timer pair instead of a cron line.

Prerequisites

Conceptual minimum

A cron job answers only one question — when — and answers it badly for a long-running data task: it captures no structured logs, has no concept of a run that outlives its interval, restarts nothing when the host reboots, and cheerfully launches a second copy while the first is still working. systemd splits the job into two objects that each do one thing. A service unit (.service) describes how to run the process: which user, which working directory, what to do when it exits non-zero, how long it may run. A timer unit (.timer) describes when to activate that service, and — crucially — it is itself a supervised unit, so systemctl can tell you when it last fired and when it will fire next. Because the timer activates the service rather than forking a shell, every run inherits journald logging, cgroup accounting, and the restart policy for free. The relationship up to the loop it schedules is covered in the parent minutely update pipeline guide; here the mechanism is the pairing itself.

The one hazard a naive timer still leaves open is overlap. If a sync run occasionally takes longer than the interval — a large catch-up after an outage, say — the timer will start a second run while the first is mid-apply, and two processes writing the same base extract and checkpoint is exactly the corruption the pipeline’s crash-safety was designed to prevent. The fix is a mutual-exclusion lock: wrap the loop in flock against a lockfile, and a second invocation exits immediately rather than racing the first. Type=oneshot with a timer, plus flock, gives a run-to-completion model with hard non-overlap.

systemd timer activating a lock-guarded diff-sync service with journald logging The timer fires on its cadence and activates the service. The service acquires an flock; if the lock is held by a prior run the new invocation exits with no overlap. The script logs to journald, and a non-zero exit triggers the Restart policy while the failure stays isolated to this unit. Timer activates a lock-guarded service; the lock is what prevents overlap .timer unit OnUnitActiveSec = 60s .service unit Type=oneshot User=osm activates flock guard sync.lock --nonblock sync loop run one cycle acquired Exit — no overlap lock held by prior run busy journald journalctl -u

Runnable solution

Three files. The invoked script wraps the loop in a lock and logs to stdout (which journald captures); the service describes how to run it; the timer says when. Install the units under /etc/systemd/system/.

The wrapper script, /opt/osm-sync/run-sync.sh:

bash 
#!/usr/bin/env bash
# Run exactly one guarded sync process. flock ensures no overlap: if a prior
# run still holds the lock, --nonblock makes this invocation exit at once.
set -euo pipefail

LOCKFILE=/run/osm-sync/sync.lock
mkdir -p "$(dirname "$LOCKFILE")"

exec flock --nonblock "$LOCKFILE" \
    /opt/osm-sync/.venv/bin/python -m osm_sync.serve \
        --store /var/lib/osm-sync/checkpoint.json \
        --poll-seconds 60

The service unit, /etc/systemd/system/osm-diff-sync.service:

ini 
[Unit]
Description=OSM minutely diff sync (one guarded cycle batch)
Documentation=https://osm-data-processing.org/osm-replication-diff-sync/building-a-minutely-update-pipeline/
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=osm
Group=osm
WorkingDirectory=/opt/osm-sync
ExecStart=/opt/osm-sync/run-sync.sh

# Failure isolation: retry a crashed run a few times, then stop and stay failed.
Restart=on-failure
RestartSec=30s
StartLimitIntervalSec=300
StartLimitBurst=4

# Do not let a wedged run block the next tick forever.
RuntimeMaxSec=50s

# journald captures stdout/stderr with this identifier.
StandardOutput=journal
StandardError=journal
SyslogIdentifier=osm-diff-sync

# Light sandboxing; the extract dir is the only writable path.
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/osm-sync /run/osm-sync
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

The timer unit, /etc/systemd/system/osm-diff-sync.timer:

ini 
[Unit]
Description=Fire the OSM diff-sync service every minute
Documentation=https://osm-data-processing.org/osm-replication-diff-sync/building-a-minutely-update-pipeline/

[Timer]
# First run 90s after boot, then 60s after each run *finishes* (not starts),
# so a slow run naturally spaces out the next rather than stacking up.
OnBootSec=90s
OnUnitActiveSec=60s
# Anchor to wall-clock minutes as an alternative cadence if you prefer:
# OnCalendar=*:*:00
AccuracySec=5s
Persistent=true
Unit=osm-diff-sync.service

[Install]
WantedBy=timers.target

Enable and start the timer (not the service — the timer owns the schedule):

bash 
sudo systemctl daemon-reload
sudo systemctl enable --now osm-diff-sync.timer

Step-by-step walkthrough

  1. flock --nonblock in the wrapper is the non-overlap guarantee. The first run acquires sync.lock; while it holds the descriptor, any second invocation fails to acquire and — thanks to --nonblock — exits immediately instead of queuing. exec replaces the shell so the Python process inherits the lock for its whole lifetime.
  2. Type=oneshot tells systemd the service runs to completion and exits, which is the right model for a timer-driven task: the unit is active (exited) between ticks, not a daemon systemd tries to keep alive.
  3. Restart=on-failure with StartLimitBurst=4 is failure isolation. A transient crash is retried up to four times in five minutes; a persistently failing run then lands in the failed state and stops retrying, so a genuine outage surfaces in systemctl status instead of spinning silently forever.
  4. RuntimeMaxSec=50s caps a single activation just under the timer interval, so a wedged run is killed rather than blocking the host — the flock still prevents the next tick from overlapping the one being killed.
  5. OnUnitActiveSec=60s measures the interval from when the previous activation finished, so runtimes and interval add up rather than colliding. OnBootSec=90s delays the first run until after the network is up. Swap in OnCalendar=*:*:00 if you want runs anchored to wall-clock minute boundaries instead.
  6. Persistent=true makes systemd run a missed activation immediately after boot if the host was down when a tick was due — the timer equivalent of anacron, so a rebooted host resumes syncing without waiting a full interval.
  7. StandardOutput=journal and SyslogIdentifier route every log line the loop prints into journald under a searchable identifier, replacing cron’s silent-or-emailed output with structured, queryable logs.
  8. ProtectSystem=strict plus ReadWritePaths makes the whole filesystem read-only to the service except the checkpoint directory and the lock directory, so a bug or a compromised dependency cannot write outside the data dir.

Verification

Confirm the schedule and a healthy run:

  • The timer is armed. systemctl list-timers osm-diff-sync.timer shows a NEXT firing time in the future and a recent LAST. If NEXT is blank, the timer is not enabled.
  • The service succeeds. systemctl status osm-diff-sync.service reports Active: inactive (dead) between runs and status=0/SUCCESS for the last invocation. A oneshot that just finished shows active (exited) briefly.
  • Logs are flowing. journalctl -u osm-diff-sync.service -n 50 --no-pager shows the loop’s applied seq …, lag …s lines. journalctl -u osm-diff-sync.service -f follows them live.
  • Non-overlap holds. Start two runs by hand — sudo systemctl start osm-diff-sync.service twice quickly — and the journal shows the second exiting without doing work because flock refused the lock.
  • Reboot recovery. After sudo reboot, list-timers shows the timer re-armed and, with Persistent=true, a catch-up run fired shortly after boot.

Common errors and fixes

Symptom Root cause One-line fix
Timer never fires Enabled the .service, not the .timer systemctl enable --now osm-diff-sync.timer
Two runs write the base at once Missing or blocking lock Use flock --nonblock in the wrapper as shown
status=203/EXEC ExecStart script not executable chmod +x /opt/osm-sync/run-sync.sh
Run killed at 50s every time Catch-up backlog exceeds RuntimeMaxSec Raise RuntimeMaxSec, or batch catch-up separately
Read-only file system writing checkpoint Path not in ReadWritePaths Add the checkpoint dir to ReadWritePaths=
Unit stuck in failed, no retries Hit StartLimitBurst systemctl reset-failed osm-diff-sync.service
No logs in journald Output redirected elsewhere Set StandardOutput=journal; reload the daemon

Specification reference

A timer unit activates its paired service on the schedule declared in [Timer]. OnUnitActiveSec= fires relative to when the unit was last activated, OnCalendar= fires on wall-clock expressions, and Persistent=true runs a missed timer immediately after boot. Monotonic timers such as OnBootSec= and OnUnitActiveSec= are documented alongside calendar events in the official systemd.timer manual, and the service-side directives Type=oneshot, Restart=, and RuntimeMaxSec= in the systemd.service manual.

Frequently Asked Questions

Why use a systemd timer instead of a cron job for OSM diff sync?

A timer activates a service unit, so every run inherits journald logging, a restart policy, resource accounting, and a defined behaviour after reboot — none of which cron provides. You also get systemctl list-timers to see the last and next firing, and Persistent=true to catch up a missed run after downtime. cron only decides when to fork a shell and leaves logging, supervision, and overlap entirely to you.

How do I guarantee two sync runs never overlap?

Wrap the loop in flock --nonblock against a lockfile. The first run holds the lock for its whole lifetime because the shell execs into the Python process; a second invocation that starts while the first is still running fails to acquire the lock and exits immediately. Combined with OnUnitActiveSec=, which measures the interval from when the previous run finished, this makes overlap impossible even when a run outlasts its interval.

What does Type=oneshot mean for a timer-driven sync?

It tells systemd the service runs to completion and exits, rather than being a long-lived daemon to keep alive. Between ticks the unit sits as inactive or active-exited, and the timer re-activates it each interval. This is the correct model for a task that does a bounded amount of work — apply the pending diffs — and then stops, as opposed to a resident process you would model with Restart=always.

How do I read and follow the sync logs?

Because the service sets StandardOutput=journal with a SyslogIdentifier, use journalctl -u osm-diff-sync.service to read the history, add -f to follow live, and -n 50 to see the last fifty lines. Filter by time with --since "10 min ago". Every line the sync loop prints — applied sequence, lag, back-off warnings — is captured there with no extra plumbing.

Up one level: Building a Minutely Update Pipeline.