You pushed a commit to the repository Flux watches and nothing changed on the cluster. How do you work out where delivery stopped, using the flux CLI and the resource statuses?

## Think in stages, not in one agent Because Flux is several controllers, "nothing happened" is never one thing. The chain is: commit exists in the remote → source-controller fetches it and publishes an artifact with that revision → kustomize-controller builds the configured `path` from that artifact → it applies the result → the applied objects become healthy. Every arrow can stall independently, and every stage writes its outcome into a status condition. A structured walk finds the break in about two minutes; guessing at logs takes an hour. ## Stage 1 — did the source see the commit? ```bash flux get sources git kubectl -n flux-system describe gitrepository fleet ``` You want two things: `Ready=True`, and a revision that contains your commit SHA (as of Flux 2.x the revision reads like `main@sha1:<commit>`). Common findings here: - **Not Ready.** The message names it: authentication failure (the referenced Secret is wrong or the deploy key was rotated), host unreachable, unknown ref. Credentials are the number-one cause after a key rotation nobody logged. - **Ready but an older revision.** Either the source's `interval` has not elapsed, or you pushed to a different branch than `spec.ref.branch`, or the repo you pushed to is not the URL in `spec.url` — a fork, a mirror, a second remote. Confirm with `git rev-parse HEAD` against the reported revision instead of trusting the branch name. - **Ready with the right revision.** Fetching is fine; move on. Note that a fetch failure leaves the *previous* artifact in place, so the cluster stays at the last good state rather than degrading — good behaviour, but it means the cluster looks fine while being silently stale. That is exactly what alerting on source readiness is for. ## Stage 2 — did the Kustomization act on it? ```bash flux get kustomizations kubectl -n flux-system describe kustomization apps ``` Things that show up here: - **Suspended.** `flux suspend` sets `spec.suspend: true` and reconciliation stops silently. Somebody suspended it during an incident and never resumed. The CLI marks it, so look before you theorise. - **Blocked on a dependency.** The condition says it is waiting for another Kustomization; the real problem is one layer down. - **Build failure.** A kustomize error — a missing file referenced in `resources:`, a bad patch, a `postBuild` substitution with no value. Nothing is applied and, importantly, nothing is pruned. - **Apply failure.** Server-side apply conflicts, admission webhook rejection, or RBAC denial when the Kustomization runs under a restricted `serviceAccountName`. - **Health-check failure.** It applied, but a named object in `healthChecks` never became healthy within `timeout`. The manifests are live; the workload is broken. Now you are debugging the app, not Flux. - **Wrong path.** Everything is green and the revision matches — because the Kustomization builds a `path` that does not contain the file you changed. Green status, no effect. - **Just early.** The Kustomization's `interval` has not elapsed since the artifact updated. ## Stage 3 — force it, to separate broken from slow ```bash flux reconcile kustomization apps --with-source ``` The CLI does this by annotating the object with `reconcile.fluxcd.io/requestedAt`, which the controller watches; `--with-source` reconciles the referenced source first, so you exercise fetch and apply in one command. If everything converges instantly, your problem was latency, and the fix is a `Receiver` webhook from the Git host so pushes trigger reconciliation instead of waiting on polling. If it fails, you now have a fresh, timestamped error instead of a stale one. ## Stage 4 — the specific cause ```bash flux logs --level=error --all-namespaces flux diff kustomization apps --path=./apps/production flux tree kustomization apps ``` `flux logs` aggregates controller logs so you do not have to know which controller to tail. `flux diff` builds the manifests locally and diffs them against the live cluster — the direct answer to "did my change even alter the rendered output?", which catches wrong-path and patch-not-applied cases immediately. `flux tree` shows the inventory of objects a Kustomization owns, which answers "is this object even managed by Flux?". And `flux check` verifies the controllers themselves are running and version-compatible — worth one run when *everything* is stale rather than one app. ## The one non-Flux cause worth keeping in mind If the object changed and then changed back, you are not looking at a delivery failure but at a fight over ownership: something else in the cluster is modifying a field Flux owns, and each side reverts the other on its own loop. The tell is a value that flips rather than a value that never arrives, and the resolution is to decide who owns the field — either stop the other actor or stop managing that field in Git. ## The habit to demonstrate Name the stages, check them in order, and say what each status *means* rather than what command you would type. Interviewers are listening for whether you know that a healthy Kustomization can sit on a stale artifact, and that green Flux resources do not imply a healthy application unless you asked Flux to check.