When is a sparse Mixture-of-Experts the wrong choice for a team self-hosting a model?

## Start from the bargain, not the hype Sparsity is a trade with a precise shape: **you spend memory to save compute.** The model holds every expert's weights so that any token can be routed anywhere, but each token only multiplies through a small fraction of them. The saving is denominated in FLOPs per token. The cost is denominated in resident bytes and in the machinery required to distribute them. That framing answers the question directly: sparsity is wrong wherever the thing you are short of is memory, and right wherever the thing you are short of is compute. ## Case 1 — the discount is per token and you have few tokens An internal service handling a few thousand requests a day still needs the full accelerator fleet to hold the model, twenty-four hours a day. Utilization is the amortization mechanism, and if there is nothing to amortize against, you are paying for a very large model's hardware to obtain a small model's arithmetic. This is the single most common way self-hosted sparse deployments disappoint: the team read the active-parameter number as the cost number. The honest comparison here is not sparse-versus-dense at the same total size. It is: what is the smallest model that clears my quality bar, and can I fit *that*? A distilled dense model — increasingly a standard production artifact — is often the right answer for low-volume internal work. ## Case 2 — quality per byte favours dense At equal total parameters, a dense model is stronger than a sparse one; at equal active parameters, the sparse model is stronger. Which comparison applies depends on what you are rationing. If your ceiling is a fixed memory budget — one accelerator, an edge box, a per-tenant footprint — you are rationing bytes, and the dense model is the one that gets more quality out of them. Sparse models win when you are rationing FLOPs or cost per token, and the memory is available. ## Case 3 — the interconnect is the hidden dependency Once experts are sharded across devices, every MoE layer performs an all-to-all: route tokens to the devices owning their experts, compute, gather results. That is a synchronous collective on the critical path of every layer. On a fabric designed for it, the overhead is manageable. On commodity networking, or across nodes that were never intended to act as one accelerator, the model becomes communication-bound and the FLOP savings evaporate. Teams that budget for GPUs and not for the fabric between them discover this after purchase. ## Case 4 — the operational tax Running a sparse model well means owning expert placement and sharding strategy, per-expert load telemetry, hot-expert replication when routing skews, and a serving stack that implements all of it competently. None of this exists for a dense model, where the only knobs are the usual ones. For a small platform team, that complexity is a real and recurring cost, and it is fair to weigh it against a modest efficiency loss from a dense alternative. ## Where sparse clearly wins So the argument stays balanced: high-volume serving where cost per token dominates the bill; frontier quality that is simply unreachable at a dense model's per-token compute; and any setting where you have the memory and want the largest possible knowledge capacity per unit of arithmetic. That is why nearly every frontier model shipped by mid-2026 is sparse — those labs are exactly in the regime where the trade pays. The mistake is assuming their regime is yours. ## How to decide, concretely 1. **Fit** — can you hold total parameters at all, on hardware you can actually get? If not, the discussion is over. 2. **Utilization** — estimate tokens per day against the always-on cost of that hardware. Low utilization erases the per-token advantage. 3. **Fabric** — is the interconnect adequate for a per-layer all-to-all across your expert shards? 4. **Alternatives at the bar** — what is the smallest model, dense or sparse, that clears your evaluation? Measure against your own task suite, not a leaderboard. 5. **Team** — who owns expert placement and routing telemetry at 3 a.m.? A principal-level answer names the bargain, applies it to the specific constraint the organization actually faces, and is willing to conclude either way. "Sparse because everyone else is sparse" is not a decision.