Two moderation layers in the same stack each block about 90% of a 500-case attack corpus when measured in isolation. Why can you not conclude that dropping either one costs roughly ten points of coverage, and what measurement answers the question properly?

### The error, stated precisely Two per-layer catch rates are **marginal totals** over the same set of cases, and marginals do not determine the joint distribution. "Layer A blocks 450 of 500" and "layer B blocks 450 of 500" are two facts about two columns; the question "what happens if I delete a column" is a fact about the rows they share, which neither total contains. The two extremes fit the same numbers. If the layers agree case for case, they block the same 450 and miss the same 50: the stack catches 450, and dropping either one costs **zero** coverage. If they are maximally complementary, A's 50 misses are exactly the cases B catches and vice versa: the stack catches all 500, and dropping either costs **50** cases, ten points. Identical marginals, opposite removal decisions. Anything in between is possible, so the honest statement from two 90s alone is "the stack blocks between 450 and 500", which is not a basis for touching anything. ### The measurement that answers it The per-case matrix from the isolation runs — one boolean per (case, layer), never a per-layer aggregate. For a pair, collapse it into a contingency table: ``` B blocks B misses A blocks a b <- b = A's unique catch A misses c d <- d = the stack's residual ``` Read it as: `a + b + c` is stack coverage. `b`, the cases **A alone** caught, is what you lose by deleting A; `c` is what you lose by deleting B; `a` costs nothing either way, because B still covers those cases once A is gone. `d` — the cases nothing stopped — is the most valuable cell in the engagement and the only one that is a *finding* rather than accounting. With more than two layers, compute it per subset, because the quantity that matters is always "unique to this layer given the others that remain". ### What drives the overlap, and what it costs to learn Predict before you measure: layers that share a detection principle share blind spots. Two classifiers trained on overlapping public safety data, or an input and an output screen that both key on the same surface vocabulary, land near the all-agree corner, and redundancy is the default expectation rather than the surprise. Genuine complementarity comes from layers that differ in kind — one reading intent in the request, one reading the realised content of the reply. The matrix either supports the defence-in-depth argument for this specific stack or refutes it; nothing else does. The cost is that this number cannot be bought cheaply. You need a per-layer verdict on every case, which means the isolation runs: baseline plus one pass per layer, each paying generation and classifier calls across both the attack and the benign half. There is no way to compute the joint from end-to-end runs, because short-circuiting deletes exactly the `a` column — the cases the first layer caught are precisely the cases the second was never asked about. ### Where the numbers mislead **Small counts.** Unique catch is usually a handful of cases. On 500 cases, `b = 7` against `c = 4` is not "A contributes nearly twice B" — the difference is inside sampling noise, and ranking layers on it is reading a coin. Report the counts with an interval, and refuse to order layers whose intervals overlap. **Corpus relativity.** Unique catch estimates what removal would have cost *against the attacks you sourced*. A layer with zero unique catch is not proven useless; it is unexercised in the direction where it would have mattered. Say this on the same line as the number, not in a footnote. **Redundancy that the matrix cannot see.** A layer with zero unique catch still covers the case where its neighbour ships a bad model update, is misconfigured during a deploy, or its hosted provider has an outage. Availability and change-risk are not in a block-rate table, and a report that recommends deleting a layer on unique catch alone has handed the owner half the picture. **False positives.** A layer's cost is not only latency and spend; it is the benign traffic it stops. The benign half belongs in the same matrix, and the layer worth arguing about is the one with small unique catch and large benign impact. ### What you would check That the matrix was built from isolation runs and not reconstructed from end-to-end logs; that `not reached` is a distinct value never folded into `not blocked`; that no isolated run rewrote the text a later layer read; that each unique-catch count carries its interval and its corpus caveat; and that the residual cell — the cases nothing stopped — leads the report, because it is the only column that changes what the team builds next.