Why does Mixtral 8x7B total 46.7B parameters instead of 56B?

## Reading Mistral's model names Mistral names its sparse models `NxMB`, where N is the number of experts per MoE layer and M gestures at the size class of the base dense model. It is a label, not an arithmetic expression. Candidates who multiply get 56B for Mixtral 8x7B and 176B for Mixtral 8x22B, and both are wrong. The published figures are: - **Mixtral 8x7B** — 46.7B total parameters, ~12.9B active per token, 32k context, Apache 2.0, released December 2023. - **Mixtral 8x22B** — 141B total parameters, ~39B active per token, 64k context, Apache 2.0, released April 2024. ## Why the totals are smaller than the multiplication A transformer layer is not one undivided block. In Mistral's sparse models only the feed-forward sublayer is replaced by a bank of eight expert feed-forward networks; the attention projections, the token embeddings, the output head and the normalisation parameters exist once per layer and are used by every token regardless of which experts fire. Duplicating only part of each layer eight times therefore multiplies only part of the parameter budget. That is the entire arithmetic reason the total lands at 46.7B rather than 56B. ## Why the active count is smaller still For each token at each layer a lightweight router picks a small subset of the experts — two of the eight in Mixtral's case. The other six experts contribute nothing to that token's forward pass. So the FLOPs per token correspond to a much smaller model than the parameter count suggests: about 12.9B active for 8x7B, about 39B for 8x22B. This is the whole selling point of the design — near-dense-large quality at near-dense-small compute. ## Which number you quote, and when This is where the practical judgment sits, and where interviewers push. - **Storage and memory footprint follow the total.** Every expert has to be resident and reachable, because the next token may route to any of them. A 46.7B model is a 46.7B download and a 46.7B residency problem, and that does not shrink because only two experts fire. - **Compute and per-token latency follow the active count.** Throughput per accelerator, and the cost of a token when you are billed by compute, track the ~12.9B figure. - **Benchmark comparisons need both.** Comparing Mixtral 8x7B against a dense 13B on quality alone flatters the dense model's memory profile; comparing against a dense 47B on quality alone flatters Mixtral's speed. State both axes. ## Where Mixtral sits in the line-up as of mid-2026 Mixtral was the model that made open MoE mainstream, and both variants remain Apache-2.0 and freely deployable. But the catalogue has moved on: the dense Mistral Small line (24B class, Apache 2.0 from Small 3 onward) generally matches or beats Mixtral 8x7B on quality while being far simpler to serve and far smaller to hold in memory, and it added multimodal input in later revisions. For a new self-hosting project, Mixtral is now usually the wrong default — you reach for it when you specifically want the MoE throughput profile at 8x22B scale, or when you have existing fine-tunes on it. Knowing that the flagship of eighteen months ago is no longer the default is itself part of the answer. ## Related naming traps The same misreading appears across the industry whenever a vendor publishes a sparse model with a total/active split, and vendors have not standardised which number goes in the name — some name the total, some the active count, some use the `NxM` style. The safe habit is to ignore the name and read the two published figures from the model card: total parameters and active parameters per token. If a model card gives only one number, find out which one it is before you size anything. ## Common wrong answers "It's 56B but they quantised it" — no, quantisation changes bytes per parameter, not parameter count. "The experts are shared between layers" — no, each MoE layer has its own bank of experts. "Only two experts are stored" — no, all eight are stored; only two run.