Two Ingresses claiming the same host were both admitted within one second - why did the uniqueness rule pass both?

## What actually happened, step by step Two teams applied an Ingress for the same hostname at nearly the same moment. 1. Request A arrives. The API server calls policy. The rule lists known Ingresses, finds no claim on that host, allows. 2. Request B arrives, overlapping in time. The rule runs again - a separate evaluation, possibly on a different replica of the engine - lists known Ingresses, and still finds no claim on that host, because A has not been persisted yet, or has been persisted but has not reached the engine's view. 3. Both writes commit. Two Ingresses now claim one host. Nothing malfunctioned. Each decision was correct for the state it saw. The rule's premise - that the state it reads is the state the write lands on - is what failed. ## Why no amount of care in the rule fixes it This is a check-then-act race, the same shape as reading a value and writing it back without a compare-and-swap: - **There is no lock.** Admission gives the policy a chance to say no; it does not hand out a mutual-exclusion token over the cluster, over the namespace, or over the hostname. - **There is no serialisation.** Concurrent requests are handled concurrently, and a policy engine with one replica does not help: it still processes requests in parallel, and even a strictly sequential engine would only order the *decisions*, while the writes land afterwards and independently. - **A fresher read does not close it.** Reading the live API instead of a replicated copy shrinks the window; it does not remove it. The gap between *decide* and *persist* is structural. - **Optimistic concurrency does not cover it.** The API server does protect a single object: a conflicting update to the same object with a stale resourceVersion is rejected. But the invariant here spans two different objects with different names, and nothing in the storage layer knows they are related. The compact way to say all of this in an interview: **admission is not a transaction.** It has no isolation and no serialisation guarantee for anything wider than one object. ## Note what the API server itself does and does not enforce Host uniqueness across Ingresses is not an API server constraint. Names are unique per kind within a namespace, and that is the only uniqueness the storage layer will enforce for you. Everything else - one host, one owner; three load balancers, no more - is a convention your policy asserts, with the reliability of an assertion made by a reader who does not hold a lock. ## What you tell the platform team afterwards Be precise about the guarantee, because the team almost certainly believed it was stronger: - The gate rejects conflicts that were **visible at decision time**, which in practice is the overwhelming majority - people rarely collide inside the same second. - The gate cannot make a two-object invariant atomic, so a simultaneous collision passes and a collision during an engine restart passes. - The residual is handled by a **detective sweep**: something enumerates Ingresses on a schedule, finds duplicate hosts, and raises them with an owner. That is what makes the invariant eventually true. ## Making it actually hold, if it must If duplicate hosts are genuinely unacceptable rather than merely unwanted, stop asking a per-request checker to enforce a global invariant and give the invariant a single owner: - **Push it into a uniqueness the storage layer does enforce.** Model each claimed host as its own object whose *name* is derived from the hostname. Two claims for the same host are then two creates of the same name, and the second fails at the API server - real serialisation, done by etcd, not by a policy. Admission then checks only that a claim exists and belongs to the requester, which is a single-object check. - **Route allocation through one component.** Hostnames are handed out by one service that owns the ledger, and the gate merely verifies the allocation. Both replace *check for a conflict* with *win an exclusive create*, which is the only shape that survives concurrency. ## The general lesson Any rule of the form *this is allowed only if no other object already does X* is best effort at admission. Write it - it catches the ordinary case cheaply and gives good feedback at the right moment - but state its limits, back it with a sweep, and never let a design depend on it as a hard guarantee.