A teammate reads a high mutual information score as evidence that the signal causes fraud — what is wrong?

## The measure has no arrow in it **Mutual information** is defined as `I(X;Y) = H(X) - H(X|Y)`, the bits of uncertainty about one variable that observing the other removes. The algebra immediately gives the same value from the other side: `H(X) - H(X|Y) = H(Y) - H(Y|X)`. Equivalently, the symmetric form `I(X;Y) = sum over x,y of p(x,y) * log2( p(x,y) / (p(x) * p(y)) )` treats the two variables identically — swap their roles and every term is unchanged. Whatever a large score is telling you, it is telling you the same thing read in either direction. There is no computation you can perform on that number alone that recovers which variable came first. This is not fixed by reaching for a richer quantity. **Conditional mutual information** `I(X;Y|Z)`, which prices the bits `X` and `Y` share once a third variable is held fixed, is still symmetric in its first two arguments. It can tell you that a dependence disappears once you condition on something else — a genuinely useful fact — but it never labels one side cause and the other effect. ## Three patterns that all score high 1. **Direct influence.** The signal really is part of what produces the outcome. This is the case the teammate has in mind, and it is one of several. 2. **A shared upstream driver.** Some third factor moves both the signal and the label. Both columns then carry bits about that factor, so they carry bits about each other, with no influence running between them at all. Remove or hold the driver fixed and the dependence collapses. 3. **The reverse direction, recorded as a column.** A field that is written after the outcome is decided — a status, a queue assignment, a note added during a review — is downstream of the label. It shares bits with it precisely because it was derived from it. | Pattern | Typical score | What swapping the arguments reveals | |---|---|---| | Signal influences outcome | Moderate to high | Nothing; the number is identical | | Shared upstream driver | Can be very high | Nothing; the number is identical | | Column derived from the outcome | Near the label's entropy ceiling | Nothing; the number is identical | The third column is the whole point: the measure cannot separate the rows. ## Why the top of the ranking is the suspicious end The score is capped: `I(X;Y)` can never exceed `H(X)` or `H(Y)`, so its ceiling against a label is the label's own entropy. A column that was computed from the outcome sits right at that ceiling, because knowing it removes essentially all the label's uncertainty. Genuine predictive signals almost never do that. So a candidate that scores dramatically above everything else on the list is, more often than not, a record of the answer rather than a clue to it — and the ranking presents it as the best thing you have. The practical check is a **timeline check**, and it is a question about the data-producing process rather than about the numbers: - At the moment the score has to be produced, does this field already have a value, or is it still null? - Who writes it, and is that writer upstream or downstream of whatever decides the label? - Does the field's value ever change after the label is assigned? A time-ordered holdout — fit on earlier rows, score on later ones, with every field frozen to the value it had at the decision moment — turns those questions into a measurement. ## What does establish direction Nothing inside information theory does it alone. Direction comes from assumptions you bring: - **Time.** A value that provably exists before the outcome cannot have been derived from it. This is the cheapest and most commonly available assumption. - **Intervention.** Change the signal deliberately, at random, and see whether the outcome distribution moves. This is the strongest evidence and usually the most expensive to obtain. - **Domain structure.** Knowing how the pipeline is wired often rules out one direction outright. A dependence score is a filter that proposes candidates for those checks. It is not itself one of them. ## What an interviewer is listening for The expected answer is short and specific: the measure is symmetric, so it prices dependence and not direction; a high value is consistent with influence, confounding, or leakage; and the practically important case in a signal-ranking exercise is leakage, caught by asking when the value becomes available rather than by computing something further. A candidate who says "correlation is not causation" and stops has stated the slogan without the mechanism — what earns the point is naming the symmetry in the definition and the leakage signature at the entropy ceiling.