A dense model runs every parameter for every token. A mixture-of-experts model does not: a small routing network looks at each token and selects a handful of expert subnetworks to process it, leaving the rest idle. The model can hold far more total parameters while doing far less arithmetic per token.
That trade is why mixture architectures became common at the frontier. It also creates a specific set of operational problems that dense models do not have.
What changed in 2026
- The architecture became mainstream. Mixture designs moved from a frontier-lab curiosity to a common choice across both closed and open-weight releases.
- Expert granularity increased. Designs shifted toward more, smaller experts with finer-grained routing rather than a few large ones.
- Serving infrastructure specialized. Expert-parallel deployment, where different experts live on different devices, became a supported pattern in major serving stacks.
- Memory expectations got corrected. Widespread confusion about active versus total parameters gave way to clearer guidance that memory tracks the total.
Active versus total parameters
|
Dense model |
Mixture-of-experts model |
| Parameters used per token |
All |
A small fraction |
| Parameters that must be in memory |
All |
All |
| Arithmetic per token |
Proportional to size |
Proportional to active size |
| Memory requirement |
Proportional to size |
Proportional to total size |
| Training difficulty |
Standard |
Harder; routing must be balanced |
| Serving simplicity |
Simple |
Expert placement matters |
This table is the whole practical story. A mixture model advertising a small active parameter count is telling you about compute, not memory. Every expert has to be loaded because any token might route to it, so self-hosting requires memory for the full parameter count. Teams who plan capacity from the active figure discover this at deployment.
The load balancing problem
The router is trained alongside the model, and left to itself it develops favourites — routing most tokens to a handful of experts while others receive almost nothing. That is bad twice over: the popular experts become a throughput bottleneck while the neglected ones remain undertrained and contribute little.
Training therefore includes an auxiliary objective pushing the router toward even utilization. Getting that balance right is the central difficulty in training these models, and an imbalanced router shows up as both worse quality and worse hardware utilization.
At serving time the same imbalance manifests as uneven load across devices when experts are distributed. If one device holds the experts that receive most of the traffic, it saturates while others idle — which is a scheduling problem of the kind covered in AI workload scheduling.
For anyone consuming these models through an API, none of this is visible or actionable. It matters when you self-host, where it determines your hardware requirement and your achievable throughput. The broader architecture background is in mixture of experts explained.
Common mistakes
- Sizing memory from active parameters. Memory tracks the total, always.
- Assuming mixture means cheap to run. Cheap per token in compute, expensive in memory.
- Ignoring expert placement when distributing. Uneven placement produces uneven load.
- Comparing models on total parameters alone. Active count predicts speed; total predicts memory. Both matter.
- Expecting dense-equivalent quality at equal active size. A mixture model with a given active count typically outperforms a dense model of that size, and comparisons should account for the memory it consumes.
FAQ
Why not just use a dense model?
For a given compute budget per token, a mixture model can hold much more knowledge. The tradeoff is memory and training complexity.
How many experts activate per token?
Design-dependent, and typically a small number out of many. Finer-grained designs with more, smaller experts have become common.
Does routing add latency?
The routing computation itself is small. The latency risk is uneven load across devices in a distributed deployment.
Can I fine-tune a mixture model?
Yes, with the usual parameter-efficient methods, and routing behavior can shift during adaptation in ways worth monitoring.
Where to go next
For the architecture, read mixture of experts explained. For memory planning, GPU memory estimate for LLMs, and for serving efficiency, AI workload scheduling.