In a Kubernetes Job spec, what do the `backoffLimit`, `activeDeadlineSeconds` and `ttlSecondsAfterFinished` fields control, and what happens when each of them is reached?

## backoffLimit — the retry budget `.spec.backoffLimit` bounds how many times a Job retries before declaring defeat. Default is 6. What gets counted depends on the Pod template's restart policy. With `restartPolicy: Never`, each failed **Pod** increments the counter and the controller creates a fresh Pod for the next attempt. With `restartPolicy: OnFailure`, the kubelet restarts the container inside the same Pod and each **container restart** increments the counter. Either way, once the count exceeds `backoffLimit`, the Job receives the `Failed` condition with reason `BackoffLimitExceeded`, no further Pods are created, and any running Pods are terminated. Retries are not immediate. The controller applies exponential backoff starting at 10 seconds and doubling — 10s, 20s, 40s, 80s … — capped at 6 minutes. That matters for expectations: the default budget of six retries can stretch over roughly twenty minutes of wall clock. A subtlety worth naming: a Job with a low `backoffLimit` and a container that fails fast can exhaust the budget in seconds; a Job with `backoffLimit: 0` gets exactly one attempt. For Indexed Jobs, `backoffLimitPerIndex` gives each index its own budget so one poisoned shard cannot burn the whole Job's retries, and `maxFailedIndexes` bounds how many indexes may fail permanently. ## activeDeadlineSeconds — the time budget `.spec.activeDeadlineSeconds` is a wall-clock limit measured from when the Job starts, and it counts everything: Pod scheduling delays, image pulls, execution, and the backoff gaps between retries. When it expires the controller terminates all active Pods and sets `Failed` with reason `DeadlineExceeded`. Two properties are commonly tested. First, it **takes precedence over `backoffLimit`** — the deadline is enforced regardless of remaining retries, which makes it the reliable guard against a Job that hangs rather than fails. Second, the *same field name* also exists on the Pod spec, where it limits that individual Pod's lifetime rather than the Job's total. Setting it in the Job's `spec.template.spec` is a different control from setting it in `spec`, and confusing the two is a classic error. Use it as a safety net for anything that can hang: a batch waiting on a lock, a network call with no timeout, a queue consumer that never sees an empty queue. Set it comfortably above the p99 runtime — too tight and you convert slow-but-successful runs into pages. ## ttlSecondsAfterFinished — the retention budget Jobs are deliberately not self-deleting: their Pods are the record of what happened, and `kubectl logs` on a finished Pod is the primary debugging tool. But without cleanup, a CronJob firing every five minutes leaves thousands of Job and Pod objects in etcd, slowing `kubectl` and, on large clusters, pressuring the API server. `.spec.ttlSecondsAfterFinished` hands that to the TTL-after-finished controller: once the Job reaches a terminal state (`Complete` or `Failed`), the controller deletes it after the given number of seconds, and the deletion cascades to its Pods via owner references. A value of `0` deletes immediately on completion — convenient but it destroys your logs, so it is only sensible when logs are shipped elsewhere. There is an important interaction: modifying the TTL of an already-finished Job is undefined behaviour, and clock skew between nodes can make deletions early or late by a few seconds. CronJobs have a second, independent retention mechanism: `successfulJobsHistoryLimit` (default 3) and `failedJobsHistoryLimit` (default 1) keep the last N Job objects per outcome. If both are configured, whichever prunes first wins; a common production setup uses TTL for time-based retention and the history limits for count-based retention. ## How they compose Think of them as three independent budgets: | Field | Budget | Failure reason | Overridden by | |---|---|---|---| | `backoffLimit` | attempts | `BackoffLimitExceeded` | `activeDeadlineSeconds` | | `activeDeadlineSeconds` | wall clock | `DeadlineExceeded` | nothing | | `ttlSecondsAfterFinished` | retention | n/a (post-terminal) | manual deletion | A production-grade Job usually sets all three: a small retry budget so genuine failures surface quickly rather than looping, a deadline generous enough for the slow tail but tight enough to catch hangs, and a TTL long enough to debug (hours to a day) but short enough to keep etcd tidy. ## Diagnosing `kubectl describe job` prints the terminal condition and reason, which immediately distinguishes "it failed repeatedly" (`BackoffLimitExceeded`) from "it hung" (`DeadlineExceeded`). If the Job object is already gone, TTL or history limits removed it — that is the moment teams discover they needed centralised logging rather than relying on Pod logs.