An MoE model has 670B total and 37B active parameters — which number sizes GPU memory?

## Two numbers, two budgets Every sparse model is quoted with a pair: total parameters and active (or activated) parameters per token. Reading them as one number is the classic mistake, in both directions — assuming you only need memory for the active slice, or assuming a 670B model must be slow. **Total parameters -> memory.** The weights of every expert must be resident on the accelerators before inference starts. There is no way around this in the normal case, because the router decides per token, per layer: the expert needed for the next token is not known until the previous layers have run. Nothing can be predictably left out. So memory footprint, and therefore the number of accelerators, tracks the total count. **Active parameters -> compute.** For any single token, only the router-selected experts multiply. The matrix multiplications performed per token — and the energy, FLOPs and a large part of the generation latency — track the active count, plus the always-dense parts (attention, embeddings, any shared expert). "Active parameters" is normally quoted inclusive of those dense parts, so 37B active means roughly the arithmetic of a dense 37B model. ## What this makes cheap and what it makes expensive - **Cheap:** cost per generated token, given that the machine is already up and busy. You are paying 37B-model arithmetic for a model that knows what a 670B parameter budget can hold. - **Expensive:** the entry ticket. You cannot serve the model at all until you have the memory for all 670B parameters, and that hardware is sitting there whether you send it one request per minute or thousands per second. This is why sparse models suit high-volume serving and suit low-volume self-hosting badly. It also explains a pricing observation people find confusing: providers can offer very large sparse models at per-token prices closer to those of far smaller dense ones, because per-token cost is set by active parameters while their capital cost is set by total. ## Where the memory actually goes Across a fleet, expert weights are usually **sharded** rather than replicated: with expert parallelism, each device holds a slice of the experts for each layer, and tokens are exchanged between devices — an all-to-all dispatch and combine per MoE layer — so each token reaches the device holding its expert. That divides the memory per device, but the *aggregate* memory still equals the total parameter count, plus per-device runtime overheads. Sharding changes who holds the bytes, not how many bytes exist. ## Does an MoE with 37B active behave like a dense 37B model? On speed, roughly yes. On quality, no — it is clearly stronger, because it has far more total capacity to store knowledge and specialized behaviour. Equally, it is weaker than a *dense* model of the same 670B total would be, if anyone could afford to train and serve one. The honest framing is that sparsity buys you a better quality-per-FLOP curve and a worse quality-per-byte-of-memory curve than dense. ## How to reason about a serving decision Ask two separate questions and do not let one answer the other: 1. *Can I fit it?* Driven by total parameters, plus room for runtime state. If the answer is no, nothing else matters. 2. *Can I serve my traffic on what I fit?* Driven by active parameters and how much work the hardware can do per second. A team that sizes by active parameters discovers at deploy time that the model does not load. A team that sizes by total parameters and then assumes matching slowness over-provisions compute and under-uses the accelerators they bought. ## The one-sentence version VRAM is sized by total parameters; FLOPs, latency and cost per token are sized by active parameters — the sparse model's whole point is that those two numbers are allowed to be far apart.