skip to content

When should an admission policy reject a Kubernetes manifest instead of silently patching it?

level: juniorimportance: must knowfreq 70%

answer

  1. who owns the correct value
  2. behaviour-neutral versus behaviour-changing
  3. a mounted file is not an environment variable
  4. the patch never reaches the repository
  5. deny when the safe form is a redesign

basics

~20 s

Patch when the fix is behaviour-neutral and the platform owns the value, like adding a missing default. Reject when the compliant form changes what the process sees, such as moving a secret from an environment variable into a mounted file.

solid answer

~50 s

Decide by asking whether the fix is behaviour-neutral. If there is exactly one correct value and the platform owns it - a missing owner label, `automountServiceAccountToken: false` on a workload that never calls the API server - a patch is fine, and denying only generates tickets. If the compliant form changes what the process sees, reject. The canonical case is the rule that secrets must not arrive as container environment variables: the safe shape is a mounted secret file, but the container calls `os.Getenv`, so rewriting `env.valueFrom.secretKeyRef` into a volume either crashes the app at start or leaves it reading an empty string. That is not a repair, it is a redesign, and only the team that owns the code can make it. Deny with a message naming the safe form, and keep mutation for defaults nobody would argue about.

go deeper

for a junior

Know that admission can either reject a request or rewrite the object before it is stored, and be ready to give one concrete example of each.

for a middle

Justify the choice from the field itself: who owns the correct value, and whether the compliant form changes what the running process reads.

for a senior

Expect to defend the rule to the team you blocked - why rewriting a secret reference is a redesign rather than a fix, and what your denial message has to tell them.

for a principal

Own the estate-wide posture: which classes of violation may ever be auto-repaired, who signs off on that, and how a patched field is made visible instead of silent.

## Two shapes of admission decision An admission policy that inspects a Kubernetes object has exactly two levers. It can **validate**: return allow or deny and leave the object byte-for-byte as submitted. Or it can **mutate**: return a patch that the API server applies before the object is persisted, so what lands in the cluster is not what the author sent. Every rule you write picks one, and the pick is not a matter of taste - it decides who ends up owning the correctness of the workload. A useful way to hold it: a denial is a message to a human, a patch is a change to a machine. Both leave the cluster compliant. Only one leaves the author knowing anything. ## The test: is the fix behaviour-neutral? Three questions settle almost every case. 1. **Is there exactly one correct value, and does the platform own it?** A missing owner label, a default that exists so the platform can bill, page or clean up, `automountServiceAccountToken: false` on a workload that never talks to the API server - the author had no opinion, and asking them to type it is friction with no learning attached. Patch it. 2. **Does the fix change the interface the process sees?** If the compliant shape alters what the container reads, which file exists, what a start-up path finds, or how the app must be written, no policy engine has the standing to make that change. Deny it. 3. **Does the author need to learn this?** If the same team will submit the same shape next quarter for a different service, a silent patch has taught nobody and you will be patching it forever. ## The canonical deny: secrets as environment variables A common rule is that secrets must not be injected as container environment variables - `env[].valueFrom.secretKeyRef` and `envFrom[].secretRef`. The reasoning is sound. The resolved value ends up in the container's process environment, where it is inherited by every child process, readable by anything that can dump `/proc`, and routinely captured by crash handlers and debug endpoints. It is also fixed at container start, so rotating the underlying Secret requires a restart. The safe form is a mounted secret file that the process reads from a path, whose contents the kubelet refreshes in place. A mutating policy *could* express that rewrite: delete the env entry, add a volume, add a volumeMount. It would be a disaster. The container is written against `os.Getenv("DB_PASSWORD")`. After the patch there is no such variable; the process either fails loudly at start-up or, worse, treats the empty string as a default and connects to nothing, or to something it should not. The mutation is not a repair; it is a behaviour change made by an engine that has never read the application code. So this rule denies, and the denial message says what the safe form is. ## The canonical patch: a default nobody argues about Contrast a rule that sets `automountServiceAccountToken: false` on Pods whose workloads never call the API server. Nothing in the container reads it. Nothing about the process changes. The field exists because the default is permissive, not because the author chose it. Denying every manifest that omits a field the author has no view on turns a security control into a typing exercise for hundreds of teams, and the first thing those teams ask for is a blanket exemption. ## What silent repair costs, even when it is right - **The source manifest keeps the flaw.** The file in the repository is what gets reviewed, promoted and applied elsewhere. The cluster is fixed; the artifact is not. - **Portability.** Applied to a cluster where the policy is absent or scoped differently, the same manifest is live and non-compliant. - **Invisibility.** A reviewer approving the pull request sees the unsafe shape and approves it, because the shape works in practice. - **Reversibility.** A patched field can be overwritten by another mutator or missed when the policy's scope changes, and nothing shouts. So even when you choose to patch, make the patch legible: stamp an annotation naming the rule that fired, so the field is traceable to a policy rather than looking like something a colleague typed. ## The answer an interviewer is listening for Not "mutate where you can, deny where you must" as a slogan, but a criterion applied to a concrete field: who owns the correct value, and does the compliant form change what the process sees. A candidate who can name one rule that must deny - and say precisely why the automatic fix would break the app - has answered the question.

  • A team argues that mutating is always kinder than blocking. What do you tell them?
    Kinder in the moment, worse over time. The patch never reaches their file, so the same shape ships to the next cluster and gets re-reviewed as correct. Mutation is right when the value was never theirs to choose; when it is, a denial with a clear message is the only thing that transfers the knowledge.
  • Is there a case for patching and denying under the same rule?
    Yes. Patch the safe default now to protect the fleet, and deny only the shapes the patch cannot make safe - for example set the missing field automatically, but refuse a manifest that sets it to the unsafe value explicitly, because there the author did have an opinion. What you must not do is patch and stay silent.
  • Why is a Secret in an environment variable worse than the same Secret in a file?
    The resolved value sits in the container's process environment, is inherited by every child process, and turns up in crash dumps and debug output. It is also fixed at container start, so rotating the Secret needs a restart, while a mounted secret's contents are refreshed in place by the kubelet.

saying these in an interview costs you the question

  • Says mutation is always safer because nothing gets blocked
  • Treats a mounted secret file as a drop-in replacement for an env var
  • Thinks the patched object is advisory rather than what is stored
  • Cannot name one rule that must deny rather than repair
  • Believes a silent patch counts as telling the author

context