Why is a channel's capacity defined as a maximum over input distributions rather than fixed by the noise alone?

## Two ingredients, not one A channel hands you one thing: the probability of each output symbol given each input symbol. That law is fixed by physics and is not yours to change. But the amount of information that crosses the link per use is not determined by that law alone, because it also depends on **what you feed in**. The extreme case makes it obvious. A sender that transmits the same symbol every time conveys zero information across even a perfect channel, because the receiver already knew what was coming. The channel was capable of more; the input distribution wasted it. Any definition of "how much this channel can carry" therefore has to range over the sender's choices and take the best one. ## The definition The quantity being maximised is the **mutual information** between input and output: the number of bits by which observing the received symbol reduces your uncertainty about the transmitted one. Capacity is - `C = max over all input distributions p(x) of I(X; Y)`, measured in **bits per channel use**. Three things follow directly from that shape: - **Capacity is a property of the channel, not of a sender.** The maximisation removes the sender's choice by taking the best one, so two teams computing the capacity of the same channel get the same number. - **The maximising input distribution is part of the answer.** Knowing `C` tells you the ceiling; knowing the distribution that attains it tells you what a capacity-approaching design has to look like statistically. - **The unit is per use, never per second.** Converting to bits per second needs the symbol rate, which is a property of the equipment rather than of the channel's noise. ## The symmetric bit-flip case, worked For a binary channel that flips each transmitted bit independently with probability `p`, symmetry means the maximising input is the uniform one - zeros and ones equally likely - and the maximum works out to - `C = 1 - H(p)`, where `H(p) = -p log2(p) - (1-p) log2(1-p)` At `p = 0.1`: `H(0.1) = 0.1 x 3.3219 + 0.9 x 0.1520 = 0.469` bits, so `C = 0.531` bits per channel use. Read that as: each transmitted bit arrives carrying about half a bit of usable payload, because the receiver is left with 0.469 bits of residual uncertainty about what was sent. Three checks that catch a shaky understanding: | Flip probability `p` | Capacity | Why | |---|---|---| | 0 | 1 bit per use | Nothing is corrupted; every transmitted bit is payload | | 0.1 | 0.531 bits per use | `1 - H(0.1)`, with 0.469 bits of residual uncertainty | | 0.5 | 0 bits per use | Output is independent of input; observing it teaches nothing | | 0.9 | 0.531 bits per use | Invert every received bit and it is the `p = 0.1` channel | The last row is the one interviewers enjoy. A channel that corrupts nearly everything is not a bad channel - it is a *consistently* bad one, and consistency is exactly what a receiver can undo. What kills capacity is not the amount of corruption but the **unpredictability** of it, which peaks at `p = 0.5`. ## What the maximisation buys an engineer 1. **It makes the number comparable.** Because the sender's choice has been maximised away, capacities of different links are directly comparable, and the ceiling applies to designs nobody has built yet. 2. **It says where the design effort goes.** If the capacity-attaining input distribution is uniform and your modulation drives the input in a lopsided way, you are giving up rate before any code is chosen. 3. **It explains why capacity is not achieved by accident.** The ceiling assumes the sender is statistically optimal. A design that is sloppy about input statistics never reaches it regardless of how strong its code is. ## The misreadings to avoid The first is thinking capacity is a function of the error probability alone in a monotone way - "more errors, less capacity". The `p = 0.9` row breaks that: capacity is symmetric about `p = 0.5` and rises again on the far side. The second is conflating the maximisation with an optimisation you perform at runtime. You do not tune the input distribution while the link runs; the maximum is a mathematical definition, computed once, that says what the channel is worth. The third is dropping the unit. Because the quantity is per channel use, a channel with a high capacity and a slow symbol rate can carry less per second than a noisier channel driven faster. Capacity ranks channels per opportunity to transmit, not per wall-clock second.