How do you choose periodSeconds, timeoutSeconds, failureThreshold and initialDelaySeconds for a Kubernetes probe? Work through what those numbers mean for detection time and for false restarts under load.

## The fields - `initialDelaySeconds` (default 0) - wait this long after the container starts before the first check. - `periodSeconds` (default 10) - interval between checks. - `timeoutSeconds` (default **1**) - how long one check may take before it counts as a failure. - `failureThreshold` (default 3) - consecutive failures needed to act. - `successThreshold` (default 1) - consecutive successes needed to be healthy again; must be 1 for liveness and startup probes. The arithmetic every candidate should do out loud: with `periodSeconds: 10` and `failureThreshold: 3`, a hung process is restarted **20-30 seconds** after it hangs - the first failing check can land anywhere within the period, and each attempt can add up to `timeoutSeconds`. Halving the period halves detection time and doubles probe load. ## The 1-second timeout trap `timeoutSeconds: 1` is the most damaging default. A healthy service can exceed it during a stop-the-world GC pause, a cold JIT phase, a noisy neighbour on the node, or a burst of real traffic. Three such misses and the kubelet restarts a perfectly healthy container - dropping its warm caches and connections, pushing its traffic onto the remaining replicas, and making *them* more likely to miss their probes. That is the anatomy of a probe-induced cascading restart, and it always appears at peak load, the worst possible moment. Mitigations: raise `timeoutSeconds` to a few seconds for liveness, keep the health endpoint off the request-serving thread pool (or ahead of the queue), and never let the probe do real work. ## Liveness versus readiness tuning Because the consequences differ, the numbers should differ: - **Liveness** is destructive and irreversible: bias towards *false negatives*. Longer period, higher threshold, generous timeout. If unsure, prefer detecting a hang in 60 seconds over restarting a healthy Pod once a week. Many teams run no liveness probe at all for services that crash cleanly on fatal errors. - **Readiness** is cheap and reversible: bias towards *fast reaction*. A short period (2-5s) and low threshold (2) let an overloaded instance shed traffic immediately, and a `successThreshold` above 1 prevents flapping straight back into rotation. ## Startup probe versus initialDelaySeconds A large `initialDelaySeconds` on liveness to survive a slow boot also blinds you for that long after *every* restart, and it does nothing for the variance of a genuinely slow start. A `startupProbe` sized to the worst-case cold start is the right tool; then `initialDelaySeconds` on liveness can be 0. ## Probe load Total probe rate is `replicas x containers / periodSeconds`. At 500 replicas and a 1-second period that is 500 checks per second, plus whatever each check touches. If the endpoint queries a cache, a dependency or the disk, probes become a meaningful share of the service's own load - and an exec probe additionally forks a process each time. Cheap, constant-time health endpoints are not a nicety. ## A worked default For a typical HTTP service: - `startupProbe`: httpGet, period 10, failureThreshold 30 - five minutes of boot allowance. - `livenessProbe`: httpGet `/healthz` with local-only checks, period 10, timeout 3, failureThreshold 3 - roughly 30-second detection. - `readinessProbe`: httpGet `/readyz` including required dependencies and warm-up, period 5, timeout 2, failureThreshold 2 - about 10 seconds to leave rotation. Then validate: force a hang in staging and measure real detection time; run a load test and confirm no probe failures at peak; and after any incident, check whether probes helped or amplified it. ## Signals that the tuning is wrong Restarts clustering at peak traffic, `Unhealthy` events mentioning timeouts, restart counts rising across *all* replicas simultaneously, or Pods flapping in and out of endpoints. All four point at probe configuration rather than at the application.