A container starts normally, serves traffic for about a minute, is then killed and restarted, and after several cycles the pod reports CrashLoopBackOff — yet the application logs show no error before each death. How do you investigate?

## Reading the pattern "Runs fine, then is killed, no error in the logs" is a strong fingerprint. A process that fails on its own terms almost always leaves something behind — a stack trace, a fatal log line, a non-zero exit from its own code path. Silence plus a hard stop means an external signal. The two external killers are the kernel OOM killer (SIGKILL, `Reason: OOMKilled`, exit 137) and the kubelet acting on a failed liveness probe. The kubelet's kill is a SIGTERM followed by SIGKILL after the grace period, so the exit code is 143 or 137 with `Reason: Error`. The decisive evidence is in the events: ``` Warning Unhealthy kubelet Liveness probe failed: Get "http://10.1.4.7:8080/healthz": context deadline exceeded (Client.Timeout exceeded ...) Normal Killing kubelet Container api failed liveness probe, will be restarted ``` The periodicity is a clue too: if the time from start to kill is roughly `initialDelaySeconds + failureThreshold × periodSeconds`, the probe is the cause, and the constant interval across restarts confirms it. ## The probe fields and their defaults - `initialDelaySeconds` (0) — how long before the first check. - `periodSeconds` (10) — how often. - `timeoutSeconds` (**1**) — how long the kubelet waits for a response. This default is the single most common cause of spurious restarts. - `failureThreshold` (3) — consecutive failures before acting. - `successThreshold` (1 for liveness; must be 1). So the default configuration kills a container after three consecutive checks that each take longer than one second, spread over roughly 30 seconds. Any endpoint whose tail latency crosses a second — a JVM in a long GC pause, a thread pool exhausted by real traffic, an event loop blocked by a synchronous call — will trip it exactly when the system is already under stress. That is the worst possible time to remove capacity, and it is how a load spike turns into a rolling restart of every replica. ## Reachability problems The kubelet performs an HTTP probe against the **pod IP** on the container port. Failures follow from: - The app listening on `127.0.0.1` (very common defaults in some frameworks) — connection refused from the kubelet. - A probe `port` naming a container port that does not exist, or a numeric port the app does not serve. - A path that returns 3xx or requires authentication — the kubelet treats 200–399 as success, anything else as failure, and it sends no credentials. - HTTPS with `scheme: HTTP` or vice versa. - A separate admin/management port that the app only opens later in startup. An `exec` probe has its own trap: the command must exist in the image (distroless images have no shell), and each invocation costs a process — a heavy exec probe every few seconds can itself add load. ## Liveness versus readiness versus startup These answer different questions and confusing them causes real outages: - **Liveness** — "is this process unrecoverably stuck?" Failure means *restart me*. It should be shallow, local, and cheap: can the HTTP server accept a connection and return, is the main loop alive. It must **not** check downstream dependencies, because a shared dependency's blip then restarts the entire fleet simultaneously, adding a thundering herd of cold starts to an existing incident. - **Readiness** — "should I receive traffic right now?" Failure removes the pod from Service endpoints without restarting it. Dependency checks belong here, where the consequence is proportionate. - **Startup** — "has boot finished?" While a startup probe is running, liveness and readiness are suspended. This is the correct tool for an application with a long or variable boot (large caches, JIT warm-up, schema migrations) — far better than a large `initialDelaySeconds`, which either under-protects slow boots or delays detection of a genuinely wedged process forever. ## Investigation steps 1. `kubectl describe pod` — Unhealthy/Killing events, and the probe configuration as applied. 2. Compare the restart interval against the probe arithmetic. 3. Exec into a *currently running* replica and call the probe endpoint yourself, ideally repeatedly and while the pod is taking traffic: `kubectl exec -it pod -- curl -sv -m 1 http://127.0.0.1:8080/healthz`. Measure latency, not just status. 4. Reproduce under load — the failure often only appears with concurrency, GC pressure or a cold cache. 5. Check whether all replicas restart at the same moment. Synchronised restarts across the fleet point at a shared dependency inside the liveness check. 6. As an experiment, temporarily remove or relax the liveness probe. If the container then runs indefinitely, the probe was the killer; if it still dies, look elsewhere (memory, a supervisor inside the container, a node-level issue). ## Fixing it well Raise `timeoutSeconds` to something realistic (2–5s), raise `failureThreshold` so transient blips do not count, move dependency checks out of liveness into readiness, add a `startupProbe` for slow boots, and make the health endpoint cheap and independent of business traffic — ideally served from a path that does not queue behind the same thread pool. A liveness probe should fire rarely; if yours is restarting containers regularly, it is being used as a workaround for a bug rather than as a last-resort recovery mechanism.