GitOps says Git is the single source of truth for desired state. In a live Kubernetes cluster, which parts of a running object does that claim not really cover, and how do you stop the agent reporting them as drift forever?
answer
- Git owns declared fields, not whole objects
- who is entitled to change this at runtime?
- autoscaler and manifest both writing replicas
- status is a report, never desired state
- narrow the exclusion, not the detection
basics
~20 sGit owns only the fields you actually declare. Values written after admission — API-server defaults, injected sidecars, autoscaler-managed replica counts, operator patches — and everything under status have other owners, so you either stop declaring that field or exclude it from the diff.
solid answer
~50 sThe honest form of the claim is that Git is the source of truth for the fields you declare, not for every byte of the live object. Several things legitimately write to a running object: the API server fills in defaults, mutating admission adds sidecars and annotations, controllers write `status`, an autoscaler owns the replica count, and operators patch fields on objects they manage. If your manifest also declares one of those fields, you have two writers for one field and the application flaps between synced and out of sync forever. There are two clean fixes and one bad one. Clean: stop declaring the contested field so the other controller owns it outright — the usual answer for autoscaled replicas — or tell the agent to ignore that specific field path on that specific resource. Bad: turn off drift detection for the whole application, which hides real unauthorised changes along with the noise.
go deeper
Know that a live object contains more than what you wrote — defaults and controller-written status — and that the desired state in Git covers the fields you declared.
Explain the concrete conflicts: an autoscaler and a manifest both setting replicas, an injected sidecar, an operator patching its own objects. Say what the symptom looks like — an application flipping between synced and out of sync on the same field.
Show the ownership decision. Ask who is entitled to change the field at runtime, remove controller-owned fields from the manifest, and scope any diff exclusion to a single path with a documented reason rather than muting the application.
Own the boundary as a standard: publish which field classes teams may exclude, require exclusions to be reviewed and attributed, and decide where the estate uses autoscaling in the first place, since every autoscaled field is a field Git deliberately does not control.
## What 'single source of truth' actually asserts The slogan is about intent, not about bytes. A live object is a merge of contributions from several writers, and only some of them are humans committing YAML. Being precise about ownership per field, rather than per object, is what separates a candidate who has run a GitOps estate from one who has read about it. ## Who else writes to your objects - **The API server's defaulting.** Submit a manifest and get back an object with fields you never wrote. These usually cause no trouble, because agents compare against your declared fields rather than the full live object. - **Mutating admission.** Service meshes and policy systems inject containers, volumes, annotations and labels at creation time. The live Pod template genuinely differs from what you committed. - **Autoscalers.** A HorizontalPodAutoscaler owns the replica count of the workload it targets. If the manifest also pins `replicas`, the two take turns writing that field. - **Operators and controllers.** A controller managing a custom resource may patch fields on the objects it creates, including ones you also declare if you have adopted its output into your manifests. - **Cluster-assigned values.** Service cluster IPs, node ports and similar allocations are assigned once and cannot be re-declared freely. - **`status`.** Never desired state at all. It is the controller's report of observed reality, and treating it as drift is a beginner error. ## The signature of the problem An application that oscillates between synced and out of sync on a short cycle, on the same field, with nobody touching it. Two writers, one field. With automatic correction enabled the two actually fight: the agent reasserts the committed value, the other controller re-asserts its own, and the workload churns. That churn can be genuinely harmful — repeated rollouts, evictions, or an autoscaler whose decisions are constantly undone during a traffic spike. ## Deciding ownership Ask one question per contested field: *at runtime, who is entitled to change this?* - If the answer is a controller reacting to live conditions — load, node capacity, certificate expiry — **the controller owns it and Git must stop declaring it.** For an autoscaled workload, remove `replicas` from the manifest entirely. The initial value comes from the autoscaler's minimum, and Git keeps owning everything else about the workload. - If the answer is "a human, through review", **Git owns it** and the other writer is a bug or a misconfiguration to fix at its source. - If the answer is "an admission webhook adds this and it is not really configuration", **exclude that path from the diff** so the agent stops comparing it. The exclusion must be as narrow as you can make it: one field path, on one resource, ideally with a comment saying which system owns it and why. Excluding a whole object, or disabling drift detection for the application, converts a cosmetic annoyance into a real blind spot — an attacker or a careless operator changing something else on that object now goes unreported. ## Why writing the live value back to Git is the wrong fix A tempting shortcut is to have automation commit the observed replica count back into the repo so the diff closes. This turns a control loop into a commit stream: every scaling decision becomes a repository change, the commits race the autoscaler, and the repository history fills with noise that hides real intent. It also does not converge — by the time the commit lands, load has moved on. Ownership has to move to one side of the boundary, not be continuously reconciled across it. ## The Kubernetes mechanism underneath Kubernetes itself has a model for multiple writers to one object, in which each writer's field ownership is tracked so conflicting writes are surfaced rather than silently clobbered. That machinery is the cluster's answer to the same problem, and it is worth knowing that a GitOps agent applying manifests participates in it. The GitOps-level decision, though, is upstream of any of that: decide who is entitled to own the field, and configure the loop so only one party asserts it. ## How to say this in an interview "Git is the source of truth for declared intent. A running object also carries defaults, injected configuration, and fields owned by controllers reacting to live conditions. So we decide ownership field by field: anything a controller must own is removed from the manifest, anything injected is excluded from the diff by exact path, and everything else Git owns and self-heals." That sentence tells an interviewer you have debugged a perpetually out-of-sync application at least once.
- Why is committing the autoscaler's current replica count back to Git the wrong fix?It converts a control loop into a commit stream. Every scaling decision becomes a repository change that races the autoscaler and is stale by the time it merges, while the history fills with noise that hides real intent. Ownership has to sit on one side: either the autoscaler owns replicas and the manifest omits the field, or you do not autoscale that workload.
- If you tell the agent to ignore a field, what have you given up?Visibility. Any change to that field — legitimate, accidental or malicious — is now invisible to drift detection. That is why the exclusion should name one field path on one resource rather than an object or an application, and should carry a comment identifying the system that owns it, so the next person knows it was a decision and not a workaround.
- A mutating webhook injects a sidecar container. Is that drift?Not usually. Your manifest never declared the injected container, so a diff limited to declared fields sees nothing to reconcile. It becomes drift only if your manifest also describes the Pod's container list in a way that conflicts, at which point you exclude the injected path explicitly rather than trying to reproduce the webhook's output in Git.
saying these in an interview costs you the question
- Claims every field of a live object comes from Git
- Declares replicas in Git while an autoscaler manages them
- Treats status fields as drift to be corrected
- Silences perpetual drift by disabling detection for the app
- Commits observed live values back into the repository