skip to content

Why is a single outcome's information content defined as -log2 p rather than as 1/p or 1 - p?

level: middleimportance: must knowfreq 48%

answer

  1. independent observations should add up
  2. independent probabilities multiply
  3. logs turn products into sums
  4. certainty has to score zero
  5. 10 bits plus 6 bits is 16

basics

~20 s

Because information from independent observations should add, while their probabilities multiply. Only a logarithm turns a product into a sum, and it also scores certainty as zero. Measures like 1/p multiply instead of adding, and 1 - p saturates.

solid answer

~50 s

The design requirement is **additivity**: seeing two independent things should give you the sum of what each one gave you. Independent probabilities multiply, so the measure must turn a product into a sum - and that is exactly what a logarithm does: `-log2(p1 * p2) = -log2 p1 + -log2 p2`. Concretely, an outcome at `p = 1/1024` (10 bits) together with an independent one at `p = 1/64` (6 bits) has joint probability 1/65536, whose surprisal is 16 bits, the sum. `1/p` fails the same test: 1024 and 64 give 65536 jointly, which is the product, not the sum of 1088. `1 - p` fails differently - it is capped at one, so one-in-a-thousand and one-in-a-million score almost identically. Requiring additivity over independent outcomes, continuity, and zero at certainty pins the measure to a logarithm; the base only fixes the unit.

code

pseudocode · 13 lines
pseudocode
function surprisal_bits(p):
    if p <= 0:
        return INFINITY          # an impossible outcome never occurs
    return -log2(p)

# two INDEPENDENT observations
separate = surprisal_bits(1/1024) + surprisal_bits(1/64)
# 10 + 6 = 16 bits

joint = surprisal_bits((1/1024) * (1/64))
# -log2(1/65536) = 16 bits

assert separate == joint   # this equality IS additivity

go deeper

for a junior

Remember the punchline: independent probabilities multiply and a logarithm turns multiplication into addition, so information from separate observations adds up the way a unit should.

for a middle

Walk the mechanics with numbers: show 10 + 6 = 16 bits on a joint probability of 1/65536, then show that 1/p gives the product instead and that 1 - p saturates near one.

for a senior

Demonstrate the precondition: state that additivity holds only under independence, and say what goes wrong when two signals are correlated and someone adds their bit figures anyway.

for a principal

Frame it as unit design: additivity, continuity and zero-at-certainty leave only the unit free, which is why a shared bit scale can be compared across teams and systems at all.

## The requirement that fixes the shape Before picking a formula, decide what the measure must do. For information about single outcomes there is one structural requirement that everything else follows from: > If two outcomes are **independent**, the information in seeing both should be the information in the first **plus** the information in the second. This is not an aesthetic preference. It is what makes the unit usable: if ten bits plus six bits did not come to sixteen bits, you could not compare, budget or reason about information at all. ## Probabilities multiply, information must add Independence means the joint probability is the product: `p(both) = p1 * p2`. So the measure has to convert a product on the inside into a sum on the outside. That operation is the logarithm: `-log2(p1 * p2) = (-log2 p1) + (-log2 p2)` Work it on numbers. Take an outcome at `p1 = 1/1024` and an independent one at `p2 = 1/64`. - `p1` alone: `-log2(2^-10) = 10` bits. - `p2` alone: `-log2(2^-6) = 6` bits. - Both: joint probability `1/65536 = 2^-16`, so `-log2(2^-16) = 16` bits - which is 10 + 6. ## Comparing the three candidate measures | candidate | value at p = 1/1024 | value at p = 1/64 | value on the joint outcome (1/65536) | sum of the two singles | additive? | |---|---|---|---|---|---| | `-log2 p` | 10 | 6 | 16 | 16 | yes | | `1/p` | 1024 | 64 | 65536 | 1088 | no - it multiplies | | `1 - p` | 0.99902 | 0.98438 | 0.99998 | 1.98340 | no - it saturates | The table makes both failures concrete: - **`1/p` reproduces the multiplication instead of removing it.** Joint value 65536 is the product of 1024 and 64, nowhere near their sum. It also scores a certain outcome as 1 rather than 0, so "nothing learned" is not the origin of the scale. - **`1 - p` is bounded above by 1.** A one-in-a-thousand outcome scores 0.999 and a one-in-a-million outcome scores 0.999999 - visually identical, though one is ten bits more surprising than the other. On the log scale those are 9.97 bits and 19.93 bits. ## What actually pins it to a logarithm Three modest requirements are enough to force the shape: 1. **Additive over independent outcomes** - the requirement above. 2. **Continuous in the probability** - a tiny change in `p` should not jump the measure. 3. **Zero at certainty** - `I(1) = 0`, so the scale starts where nothing is learned. Any function meeting all three must be `c * log(1/p)` for some positive constant `c`. The constant is not a second degree of freedom in disguise - it is the **choice of unit**, and choosing base 2 (the fair coin) is what makes the unit a bit. So the logarithm is not a convenience for handling small numbers; it is the only continuous additive answer, and everything left to choose after that is naming the unit. ## Independence is a precondition, not decoration Additivity is a statement about **independent** outcomes only. If two signals are correlated - one fires because the other did - then `p(both) != p1 * p2`, and adding their individual surprisals gives the wrong number for the pair. The measure itself is still fine: the surprisal of the joint outcome is `-log2 p(both)`, computed from the real joint probability. Two consequences worth stating out loud: - If the two signals almost always fire together, the pair carries barely more information than one of them alone, and naive addition badly overstates it. - If they are mutually exclusive, they never occur together at all, so the joint outcome has probability zero and asking for its surprisal is not meaningful. Confusing *independent* with *mutually exclusive* is the most common way this argument is mangled. Independent means the probabilities multiply; mutually exclusive means they cannot both occur, and their probabilities add only when you are asking about "one or the other". ## Why an interviewer asks this The question separates candidates who memorised a formula from candidates who know what the formula was built to do. The good answer is short and structural: information should add over independent observations, probabilities multiply, a logarithm is the bridge - and the base is just the unit. Anyone who can say that can also re-derive why certainty is zero, why rarity grows without bound, and why a bounded measure like `1 - p` could never work.

  • Two signals are strongly correlated. Does adding their surprisals still give the surprisal of seeing both?
    No. Addition is valid only when the joint probability is the product of the two, which is what independence means. For correlated signals you must use the real joint probability: `-log2 p(both)`. If they nearly always fire together, plain addition roughly doubles a figure that barely moved.
  • Does the additivity argument fix the unit as well as the shape of the measure?
    No. Additivity, continuity and zero-at-certainty force the form `c * log(1/p)` but leave the positive constant `c` free. That constant is exactly the unit choice - base 2 gives bits. Every comparison and ratio is unaffected by which constant you pick.
  • Why must certainty score zero rather than some baseline value?
    Because additivity demands it. Seeing a certain outcome alongside an informative one must leave the total unchanged, which only works if the certain outcome contributes zero. Any measure scoring certainty as 1 - such as 1/p - breaks the sum as soon as a guaranteed outcome appears.

saying these in an interview costs you the question

  • Says the logarithm is just a convenience for small probabilities.
  • Claims 1/p works equally well as a measure of surprise.
  • Asserts surprisals add for any two outcomes, independent or not.
  • Confuses independent outcomes with mutually exclusive ones.
  • Believes a certain outcome should score one rather than zero.
  • Thinks changing the log base changes which outcome is more surprising.