Kubernetes documents that a CronJob may create a Job twice for one schedule point, or not at all. Given that guarantee, how would you design a nightly billing batch that must charge each customer exactly once?

## Start by naming the guarantee Three independent sources of non-determinism stack up: 1. **The CronJob controller** reconciles rather than firing timers, so a controller restart, a clock adjustment, or a control-plane outage can cause a schedule point to be **missed** entirely, or (rarely) a duplicate Job to be created. The documentation states the guarantee as "about once per schedule" — not exactly once. 2. **The Job controller** retries failed Pods up to `backoffLimit`. A Pod that did half the work and then lost its node is retried from scratch. That is at-least-once *execution*. 3. **The kubelet and node** can evict, preempt or kill a Pod mid-write, and a network partition can hide a completed side effect from the caller. So the platform gives you at-least-once with a possibility of zero. Any "exactly once" property must be constructed above it. Saying this explicitly is the first thing an interviewer is listening for. ## Make the run addressable Everything hinges on a **deterministic key for the logical run**. Compute it from the period being billed — `billing-2026-08-12` — not from wall-clock `now()` inside the container. If the key is derived from the current time, a retry twenty minutes later and a catch-up run the next morning are different runs, and every duplicate-suppression mechanism downstream fails. Pass the logical date in explicitly, or derive it from the schedule with a documented rule for late starts. ## Enforce single execution where the truth lives `concurrencyPolicy: Forbid` prevents a second *Job* while one is active. It does not prevent a duplicate Job created across a controller restart, a manually triggered run, a run in a second cluster during a failover, or a zombie Pod whose node was partitioned but is still writing. Mutual exclusion must be enforced by the system of record: - **Unique-constraint claim** — `INSERT INTO batch_runs(run_key, started_at) VALUES (...)`; a duplicate key violation means "someone else owns this run", and the second copy exits 0. Durable and visible. - **Advisory lock** with a lease and heartbeat — good for long runs, but requires careful expiry so a partitioned holder cannot resurrect and keep writing (fence with a monotonically increasing token). Whichever you pick, the fencing token or claim row must be checked on **every write**, not only at startup, if the run is long enough for a partition to occur mid-flight. ## Make each item's effect idempotent Single execution of the *run* is not enough, because a run can die halfway and be retried. Push idempotency down to the item: - One row per `(customer, period)` with a unique constraint, so re-processing is a no-op or an upsert. - An **idempotency key** on every external call — most payment providers accept one and deduplicate server-side, which is the only way to be safe when the response to a charge is lost in the network. - Checkpoints: record each item as done in the same transaction as its effect, so a resumed run skips it. If the effect is external, use the outbox pattern — write intent transactionally, then have a separate process deliver it with retries and an idempotency key. At this point the number of times the Job runs stops mattering, which is exactly the property you want: correctness no longer depends on a scheduling guarantee you do not have. ## Handle the zero case, which is the harder half Duplicates are loud; a missed run is silent. Nothing fails, no Job exists, no alert fires. Defences: - **Heartbeat from the payload** — the job writes "billing for period X completed" to a monitoring system; alert when that record is older than a period plus a margin. This survives the CronJob being suspended, deleted, misconfigured or stuck. - **Bounded catch-up** — a sane `startingDeadlineSeconds` (a few intervals) so a brief outage recovers automatically, without an unbounded backlog and without tripping the controller's 100-missed-schedules guard, which sticks and stops the schedule permanently. - **A reconciliation pass** — a second, later job that finds unbilled periods and processes them. With idempotent items this is safe to run always, and it converts "scheduler reliability" into "eventual consistency", which is a much cheaper property to guarantee. ## Set the Job's own budgets - `activeDeadlineSeconds` — a hang would otherwise hold the lock and, with `Forbid`, suppress every subsequent run indefinitely. - A small `backoffLimit` — with idempotent items, retries are safe but should not mask a systematic failure; fail loudly after two or three. - `restartPolicy: Never` so each attempt keeps its own Pod and logs, plus `ttlSecondsAfterFinished` and centralised log shipping so evidence outlives cleanup. - Explicit `timeZone`, and a documented decision about DST — a schedule near a transition can be skipped or repeated, which for a *daily* billing run is a correctness question, not a cosmetic one. ## Know when to leave CronJob behind CronJob's honest scope is "periodically kick off a task whose correctness does not depend on the kick". Once you need dependency graphs between steps, guaranteed backlog execution, per-run audit trails, human approval gates or exactly-once orchestration, the right answer is a workflow engine (Argo Workflows, Temporal, Airflow) or an event-driven consumer with a durable queue — while frequently still executing the steps as Kubernetes Pods. A principal-level answer states that boundary rather than bending CronJob toward guarantees it does not offer.