Four hundred servers each run the same systemd timer with OnCalendar=*-*-* 03:00:00, and every night they hit the same internal package mirror at once. Which [Timer] directives spread that load, and which one does not do what people expect?

## The failure mode A schedule written as an exact wall-clock time is a synchronization primitive. Four hundred hosts with NTP-corrected clocks and `OnCalendar=*-*-* 03:00:00` will start their jobs within milliseconds of one another. Whatever they touch — a package mirror, an artifact store, an object-storage bucket, a database read replica — sees a spike four hundred times its steady state, and the classic symptoms follow: connection pool exhaustion at the target, timeouts, retries that amplify the spike, and a job that succeeds when you test it on one host and fails at 03:00 on the fleet. ## The directive that actually spreads ```ini [Timer] OnCalendar=*-*-* 03:00:00 RandomizedDelaySec=45min ``` `RandomizedDelaySec=` takes a time span and delays each elapse by a random amount between zero and that span. The choice is made per host, so no coordination is needed: the fleet's arrivals become roughly uniform across the window. Sizing it is straightforward arithmetic — take how long one run takes and how much concurrency the target can absorb, and pick a window wide enough that expected concurrency stays under that ceiling. Four hundred hosts, a two-minute job, and a target comfortable with twenty concurrent clients wants a window of at least about forty minutes. ## Making the offset stable By default the delay is re-drawn on each elapse, so a given host lands at a different minute every night. `FixedRandomDelay=yes` changes that: the offset is derived deterministically from the machine ID and the unit, so each host keeps the same slot indefinitely. Reach for it when you want the timing to be reproducible — for correlating a nightly job with a graph, for staggering hosts against a cache that warms per shard, or simply so that "host 47 runs at 03:19" stays true and debuggable. ## The knob that does not do this `AccuracySec=` is regularly proposed as the answer and is not. It specifies the accuracy with which the timer shall elapse — the window, starting at the nominal time, inside which systemd may actually fire it — and it defaults to one minute. Its purpose is power management: by allowing slack, systemd can coalesce several timers into one wakeup and let the CPU sleep longer, which matters on laptops and dense virtualized hosts. Two consequences follow: 1. The default one-minute slack does not meaningfully de-synchronize a fleet — everyone is still aiming at the same minute. 2. Its intent is the *opposite* of spreading. Coalescing pulls firings together so they share a wakeup, rather than pushing them apart. If you genuinely need a timer to fire close to its nominal instant, `AccuracySec=1us` narrows the window, at the cost of denying systemd the batching. That is a legitimate use — it is just not a load-distribution tool. ## Alternatives, and when they are better - **Templated schedules.** Generating a distinct `OnCalendar=` minute per host from configuration management gives you exact, auditable placement, at the cost of managing the mapping and re-balancing when hosts come and go. Randomized delay is usually the better default; templating wins when the distribution has to be provably even or aligned to something else. - **Monotonic timers.** `OnBootSec=`/`OnUnitActiveSec=` inherently stagger, because hosts boot at different moments. Good when the requirement is really an interval; useless when the job must land in a nightly window. - **Fix the target.** Spreading the clients hides a server that cannot absorb burst. If the mirror falls over at four hundred concurrent pulls, the delay window buys time but the capacity question is still open. ## The interaction with catch-up runs A fleet recovering from a power event brings up many hosts within the same minute, and any timer with `Persistent=true` that is owed a run wants to fire immediately. That is the same stampede in a different costume, and `RandomizedDelaySec=` is the same mitigation — worth stating explicitly in an interview, because it shows you have thought past the steady state to the recovery path. ## Verifying ```bash systemctl list-timers mirror-sync.timer ``` Run it on a handful of hosts and compare the computed next elapse; with a randomized delay in place the times should visibly differ. On the target side, the honest confirmation is the arrival-rate graph flattening from a spike into a plateau.