Much real Kubernetes Ingress behaviour — path rewriting, timeouts, body-size limits, sticky sessions — is configured through controller-specific annotations rather than fields in the Ingress spec. Why is that, and what problems does it create in a shared cluster?

## Why the spec is so small The Ingress API was scoped to the intersection of what every implementation could support: match a hostname and path, send to a Service, terminate TLS. Anything richer differed too much between an in-cluster NGINX and a cloud L7 load balancer to standardise. Rather than block, implementations used the one extension point Kubernetes gives every object: `metadata.annotations`, a free-form `map[string]string` the API server stores without interpretation. The result is that a mature Ingress often has a three-line `spec` and a large annotation block. The spec describes where traffic goes; the annotations describe how it is actually handled. ## What breaks **No schema, no validation, no discovery.** The API server checks nothing beyond key syntax. Misspell `nginx.ingress.kubernetes.io/proxy-body-size` and you get no error, no warning and no effect — you learn about it when a large upload fails in production. A value in the wrong unit or format is usually ignored or, worse, silently coerced. There is no `kubectl explain` for annotations, so discovery means reading a specific controller's docs for a specific version. **No portability.** Annotations are namespaced by vendor prefix for a reason: they mean nothing to another controller. Moving from NGINX to an Envoy-based or cloud controller means auditing every annotation, finding equivalents that may not exist, and re-testing. Because none of this is in `spec`, a manifest can look portable and not be. **Version drift.** Controller releases change annotation behaviour, add validation that rejects previously tolerated values, or gate features behind new configuration. Upgrading the controller can therefore change traffic behaviour with no change to any manifest — an unusually invisible risk. Pin controller versions, read release notes for annotation changes, and stage upgrades. **Interaction with path matching.** The classic example: `Prefix: /api` forwards the full path, so making the backend see `/` requires `rewrite-target`. Doing that with capture groups requires switching the path to `ImplementationSpecific` with a regex, which abandons the standardised matching semantics. One missing feature thus drags the object into fully controller-specific territory. ## The multi-tenancy problem This is the part senior candidates should reach unprompted. Every Ingress served by a controller is compiled into **one shared data-plane configuration**. RBAC that lets a team create Ingresses in their own namespace therefore lets them contribute to a global config. Most annotations are safely scoped to their own server block, but a class of them is not: - **Snippet annotations** (`configuration-snippet`, `server-snippet`, `stream-snippet`) inject raw proxy configuration. Historically these have enabled cross-tenant reads of other services' secrets and, in the worst cases, remote code execution in the controller — which typically runs with cluster-wide read access to Secrets. - **`auth-url` / `auth-snippet` and rewrite regexes** have produced injection issues where crafted values escaped their intended context. - **Global-effect annotations** such as some rate-limit or SSL settings can influence behaviour beyond the owning namespace depending on controller version. The standard hardening set: run the controller with snippet annotations disabled (`allow-snippet-annotations: false`, the safer default in current releases), enable `annotation-value-word-blocklist`, keep the controller patched, run it with a minimal ServiceAccount and its own namespace, and — most importantly — gate Ingress creation through admission policy that allows only a known list of annotation keys and value patterns. ## Practical containment 1. **Allowlist annotations in admission control.** ValidatingAdmissionPolicy, Kyverno or Gatekeeper rejecting unknown or dangerous keys is the single highest-value control, because it converts "anyone can write proxy config" into "anyone can use approved knobs". 2. **Template Ingresses centrally.** A Helm chart or platform abstraction where teams set a hostname and a few named options, and the platform renders the annotations, removes both the typo class of bug and the injection class. 3. **Lint in CI.** Check annotation keys against the controller version in use so typos fail at review time rather than at runtime. 4. **Prefer typed alternatives.** Where a controller offers a CRD for the same behaviour, prefer it — it has a schema. Strategically, the Gateway API exists partly to solve this, turning rewrites, header manipulation and traffic splitting into validated fields, so new work increasingly belongs there while Ingress remains frozen. 5. **Test behaviour, not manifests.** Because annotations are unvalidated, the only reliable verification is a request-level test asserting the resulting behaviour — a large upload actually succeeds, the rewritten path actually arrives. The honest summary for an interview: annotations were a pragmatic escape hatch that let the ecosystem ship, and they work, but they trade away validation, portability and isolation — which is exactly why the community's next API made them fields.