Noisy Neighbour Isolation

## Why the cluster-wide view cannot answer this A total is a sum over owners, and summing is exactly the operation that destroys the information you need. Aggregate bytes in, aggregate request rate and aggregate utilisation on a node all rise identically whether ten tenants each grew ten percent or one tenant tripled. "Busy" is therefore not a diagnosis; it is the same picture a healthily growing estate produces. **Contention attribution** — tying an observed slowdown on shared nodes to the owner who caused it — is the step that has to happen before any lever can be chosen, and it is the genuinely difficult part of this whole subject. It is difficult for a structural reason: the tenant that reports the incident is almost never the tenant that caused it. The victim has the tight deadline; the cause has the batch job. So the complaint tells you *when* and *where*, and nothing at all about *who*. ## What attribution actually requires 1. **The right time window.** Not the last hour and not a daily roll-up — the minutes on either side of the symptom's onset. A daily average hides a twenty-minute burst completely. 2. **The right machines.** Contention is local to the nodes both tenants' traffic lands on. An owner that is enormous on machines your complainant never touches is irrelevant to this incident. 3. **Per-owner breakdown.** Usage attributed to a quota principal — a client identity, an application, a team — rather than to the node as a whole. 4. **Several axes at once.** Bytes per second, requests per second, request-time share and connections held. Comparing owners on one axis produces a confident wrong answer roughly as often as a right one. ## The traps - **The loudest is not the costliest.** A tenant sending a very high rate of small requests, or one reading far-behind history that has to come off disk rather than out of memory, can occupy most of a node's request-handling capacity while sitting mid-table on a byte-rate chart. - **Retry storms look like causes.** A client that is being answered slowly or refused will retry, so its request rate climbs *after* the incident begins. Ranking owners by request rate during the incident can therefore promote a victim to prime suspect. Rank by what changed first, not by what is highest now. - **Identity collapse.** If every application in an organisation connects with the same credential or identity, per-owner numbers exist but all say the same thing. This is the single most common reason attribution is impossible, and it can only be fixed before the incident, never during it. - **No baseline.** Without a sense of what each owner normally consumes, "this tenant is using a lot" is unfalsifiable. The useful comparison is against that owner's own ordinary day. ## What varies between platforms This is where honest answers differ, and an interviewer listening carefully will notice: | Attribution granularity available | What you can conclude | |---|---| | Per client identity or application | Direct attribution to an owner | | Per connection or per source address only | Attribution to a fleet, then a mapping exercise to reach an owner | | Per namespace or per stream only | Attribution to a workload, which may be shared by several owners | | Aggregate per cluster only, as with some rented offerings | No attribution from the platform; you must infer from the client side | Because of that last row, attribution sometimes has to be reconstructed from what clients report about their own send and receive rates, correlated across teams. That is slower and less reliable, and it is a real cost of not being able to see inside the thing you rented. ## Making attribution possible before you need it The work that makes an incident tractable is all done in advance: - Give each application a distinct identity, and keep a mapping from identity to owning team. - Retain per-owner usage at a resolution fine enough to see a burst, for long enough to investigate one after the fact. - Keep an ordinary-day baseline per owner, so a deviation is visible as a deviation. - Record which workloads share which machines, so the search can be narrowed to the ones that actually interact. ## After attribution Only once an owner is named does the question of what to do become answerable, because every available response is aimed at a specific owner — a bound on what it may take, a change to what it sends, or, at the far end, moving it off this hardware entirely. Reaching for a lever before attribution means choosing a victim at random, and reaching for more capacity instead means paying to absorb a burst you have not yet understood and that will grow again.