A systemd timer with OnCalendar=daily did not fire because the machine was powered off at the scheduled time. What does Persistent=true in the [Timer] section change about that, and how does systemd know a run was missed?

## The default behaviour, and why it surprises people A timer unit only computes elapse times while it is loaded and active. If the box is powered off, suspended past the moment, or the timer was masked, the moment passes with nothing to observe it. When the machine comes back the timer arms itself and computes the *next* elapse from now. The 02:30 backup that was due during last night's maintenance window simply never happened, and nothing in the journal says so, because no unit ran. That is the correct default for a lot of jobs — a metrics push or a cache warm has no value once its moment has passed — and the wrong one for backups, certificate renewal, log shipping and anything else whose value is cumulative. ## What Persistent= does ```ini [Timer] OnCalendar=daily Persistent=true RandomizedDelaySec=30min ``` With `Persistent=true`, systemd keeps a record of when the timer last triggered its unit. The record is a stamp file whose modification time carries the timestamp: - system timers: `/var/lib/systemd/timers/stamp-<unit>.timer` - user timers: `~/.local/share/systemd/timers/stamp-<unit>.timer` When the timer unit starts, systemd reads that stamp and asks whether the calendar expression would have elapsed between then and now. If it would have, the timer triggers its service immediately rather than waiting for the next scheduled moment. After that the ordinary schedule resumes. Because the stamp lives under `/var/lib`, it survives reboots but is scoped to that machine. Wiping `/var/lib/systemd/timers` — or rebuilding the host from an image — resets the state, and the first start after that is treated as "never run", which will fire the job once. ## It catches up once, not N times This is the most common misunderstanding. `Persistent=` is not a queue of missed occurrences. A daily timer on a laptop that was closed for a fortnight does not run fourteen backups on wake; it runs one. The semantics are "a run was owed, so run now", which is what you want for idempotent maintenance jobs and is exactly how anacron behaves for `/etc/cron.daily`. ## Only for calendar schedules `Persistent=` is meaningful only alongside `OnCalendar=`. A monotonic timer such as `OnBootSec=15min` has no notion of a missed occurrence: its reference point is the current boot, so after any downtime it simply starts counting again. Pairing `Persistent=true` with a purely monotonic timer buys nothing. ## The boot stampede, and the knob for it A host that comes up after downtime with several persistent timers armed will fire all of the owed jobs at once, on top of the rest of boot. Across a fleet recovering from the same power event, every host does that simultaneously. `RandomizedDelaySec=` is the mitigation: it spreads each trigger over a random window rather than putting the whole owed workload on the first seconds of uptime. ## Suspend, and the other directive `Persistent=` handles the machine having been *off*. A related but distinct case is the machine being *suspended*: `WakeSystem=true` asks systemd to use an alarm clock that can wake the system from suspend at the elapse time, on hardware that supports it. They solve different problems — one resurrects an owed run after the fact, the other prevents the miss by waking the machine — and a laptop maintenance timer often wants a considered choice between them. ## Verifying it ```bash systemctl list-timers backup.timer ls -l --time-style=full-iso /var/lib/systemd/timers/ ``` `list-timers` prints the `LAST` elapse and how long ago it `PASSED`, which tells you whether the catch-up you expected actually happened; the stamp file's mtime is the same fact at the source. If you need to rehearse the catch-up path, removing the stamp file makes the next start of the timer treat the job as owed. ## Choosing Ask one question of each scheduled job: if this run is skipped entirely, does the work still need doing later? Backups, certificate renewal, log rotation and reconciliation say yes — those get `Persistent=true`. A five-minute health push says no; catching it up an hour late just adds a misleading data point.