skip to content

In cluster-wide cron with per-tick claims, why must each claim be keyed by the scheduled tick rather than the node's current clock reading?

level: seniorimportance: should knowfreq 44%

answer

  1. unique constraint needs identical values
  2. rounding straddles a boundary
  3. derive the key from the schedule
  4. explicit zone, stored in UTC
  5. skew changes when, not what

basics

~20 s

Node clocks and wake-up times differ, so keys built from 'now' can differ across nodes, and each distinct key gets its own winner. A key computed from the cron expression, in UTC, is identical on every node for the same tick.

solid answer

~50 s

Nodes never read exactly the same time, and they do not wake at exactly the same moment: one reads `02:00:29.9`, another `02:00:30.1`, and a node delayed by load may wake at `02:01:00.3`. A claim keyed by a rounded or truncated current time can therefore yield **different keys for the same tick**, and a unique constraint only rejects identical keys, so the job runs twice, or a late run steals the next tick's key. The fix is to derive the key from the **schedule**: compute the fire instant from the cron expression and claim `(job, scheduled_at)` in **UTC**. Skew then changes only *when* a node attempts the claim, not *what* it claims. Two fleet-specific traps remain: nodes with different time-zone settings compute different instants for the same expression, and a local-time schedule meets a skipped and a repeated hour on daylight-saving days. Keep skew bounded anyway, since a fast node still claims early.

code

sql · 4 lines
sql
-- every node runs this at the tick; the primary key (job_name, scheduled_at)
-- lets exactly one insert succeed, the others get a unique-key violation and skip
INSERT INTO job_runs (job_name, scheduled_at, owner_node, status)
VALUES ('nightly-billing', TIMESTAMP '2026-09-17 02:00:00', 'node-3', 'RUNNING');

go deeper

for a junior

Remember that different machines never agree exactly on the time, so a run should be identified by when it was scheduled, not by when some node started it.

for a middle

Explain how keys built from each node's clock diverge across a rounding boundary or a late wake-up, and why a key derived from the cron expression is identical everywhere.

for a senior

Cover the fleet traps: mixed node time zones, the skipped and repeated local hour, late wake-ups under load, and why skew must still be bounded and alerted on.

for a principal

Set the organisation-wide rule: schedules carry an explicit zone, claims are stored in UTC, and local-time business schedules come with a written transition-day policy owned by the job's team.

## What a tick identity is In cluster-wide cron with per-tick claims, every node tries to insert a claim record, and a **unique key** guarantees only one insert succeeds. That guarantee is only as good as the key: the store rejects a second claim **only if its key is identical** to the first. So every node must compute exactly the same key for the same scheduled run. That key is the run's **tick identity**, and it should name the moment the run was *scheduled for*, not the moment any particular node noticed it. ## How keying by 'now' breaks A tempting shortcut is to key the claim by the node's current time, rounded or truncated to the job's granularity. That fails in several ways: - **Readings straddle a boundary.** With rounding to the nearest minute, a node reading `02:00:29.9` keys `02:00` while one reading `02:00:30.1` keys `02:01`. Two keys, two winners, two runs. - **Late wake-ups.** A node delayed by load or a pause wakes at `02:01:00.3` for the 02:00 run and truncates to `02:01`. It wins a key that belongs to the *next* tick, so the real 02:01 run is either blocked or, if another node claimed first, this node's 02:00 work is silently dropped. - **Skewed clocks.** A node whose clock is off by more than the distance to a boundary lands on a neighbouring key even when it wakes on time by its own reckoning. - **Mixed granularity.** A job whose schedule changes from minutely to every five minutes produces keys that no longer line up with any cron-defined instant. In every case the unique constraint works perfectly; it is simply being given different values. ## Keying by the scheduled instant The robust key is computed from the **cron expression**: "the fire instant this node is about to act on", expressed in **UTC** and stored with full precision. Every node evaluating the same expression in the same zone gets the same answer regardless of when it wakes or how its clock is offset. ```sql -- one row per job per scheduled instant, in UTC CREATE TABLE job_runs ( job_name VARCHAR(100) NOT NULL, scheduled_at TIMESTAMP NOT NULL, owner_node VARCHAR(100) NOT NULL, status VARCHAR(20) NOT NULL, PRIMARY KEY (job_name, scheduled_at) ); ``` A scheduler loop that advances from the last tick it handled, rather than from the current time, also keeps a late wake-up attached to the tick it was meant for. ## Daylight-saving and time-zone traps across a fleet A fleet adds time-zone problems that a single machine does not have. | Trap | What happens | Mitigation | |---|---|---| | Mixed node time zones | Nodes defaulting to different zones compute different instants for `0 2 * * *`, hours apart, so both claims succeed and the job runs twice a day | Evaluate every schedule in an explicit zone stored with the job, never the operating system's default | | Spring-forward day | A local time inside the skipped hour does not exist; implementations disagree on whether to fire late or skip | Schedule in UTC, or define the rule for the skipped hour explicitly | | Fall-back day | A local time inside the repeated hour maps to two UTC instants, so a UTC key yields two distinct ticks | Decide whether both occurrences run; if only one should, pick the first | | Mixed library versions | Nodes with different time-zone rule data disagree after a government changes its rules | Roll rule updates fleet-wide and prefer UTC schedules | The simplest policy is to **schedule in UTC**. When a business genuinely needs local time, such as "02:30 in the store's region", convert each local fire time to UTC once, with a written rule for the two transition days. ## Clock skew still matters Keying by schedule fixes *what* is claimed, not *when* nodes act. Each node still decides when to wake using its own clock: 1. A node running minutes fast claims the tick early and may start billing before the day's last transactions have landed. 2. A node running slow wakes after the claim is taken and simply idles, which is harmless. 3. A store can refuse a claim whose tick is still in the future by the store's own clock, which caps the damage a fast node can do. Keep node clocks synchronised and alert on offset, even though the key no longer depends on it. ## Benefits beyond correctness - **Auditability**: each run row states exactly which scheduled instant it served, so "did billing run for the 16th?" is a lookup. - **Gap detection**: a scheduled instant with no row is a missed tick, detectable by query. - **Idempotent side effects**: the same tick identity can key downstream work, so a duplicate run finds its work already done.

  • Half of a fleet runs with a UTC system time zone and half with a local zone. What happens to a daily 02:00 cron using per-tick claims?
    Each half computes a different fire instant for '02:00', several hours apart, so the halves produce two different keys, both claims succeed, and the job runs twice a day. Evaluate the schedule in an explicit zone stored with the job, never in whatever zone each node's operating system reports.
  • Why does keying by the scheduled tick not remove the need to keep node clocks synchronised?
    The key fixes what is claimed, but each node still decides when to wake using its own clock. A node minutes fast claims and runs the job before its intended time, possibly before its input data is complete. Keep skew small and alerted on, and let the claim store refuse a tick that is still in the future by its own clock.
  • How does tick-keying help an operator after an incident?
    Each run row names exactly which scheduled instant it served, so gaps and duplicates are queryable: an expected instant with no row is a missed tick, and rejected claims show how many nodes tried. That turns 'did billing run for the 16th?' into a lookup instead of a search through logs.

saying these in an interview costs you the question

  • Truncating the current time to the minute is a safe claim key.
  • Synchronised clocks agree exactly, so skew never changes a key.
  • Each node can evaluate cron in its own default time zone.
  • Keying by scheduled tick removes any need for clock sync.
  • Daylight saving only matters for jobs inside the repeated hour.