Mixture of experts, without the hype

Many of today's largest language models are mixtures of experts (MoE). Their announcements often quote two numbers, such as "400 billion parameters, 17 billion active," and the second number is the one that explains why they are affordable to run. The idea has been around for decades, but it became central to large language models only in the last few years.

MoE is also surrounded by misconceptions: that each expert is a specialist in a topic, that MoE models are cheap to host, or that "active parameters" tells you everything about performance. This explainer covers how MoE layers work, what the router does, what experts actually learn, the engineering trade-offs, and what MoE means if you use or deploy these models.

Dense vs. sparse models

In a standard dense transformer, every parameter is used for every token. Double the parameters and you roughly double the computation per token.

A sparse mixture-of-experts model replaces some layers, usually the feed-forward (MLP) blocks, with a set of parallel sub-networks called experts, plus a small router. For each token, the router picks a few experts, often 1, 2 or a handful out of 8, 64 or more, and only those run. The model has many parameters in total, but each token uses only a fraction of them.

This decouples two things that are tied together in dense models: capacity (total parameters, which store knowledge and skills) and compute per token (active parameters, which determine speed and cost).

How routing works

For each token at each MoE layer:

1. The router, a small learned layer, scores every expert for this token.
2. The top-k experts by score are selected.
3. The token is processed by those experts.
4. Their outputs are combined, weighted by the router's scores.

def moe_layer(x, experts, router, k=2):
    scores = softmax(router(x))              # one score per expert
    top = argsort(scores)[-k:]               # pick the k best experts for this token
    weights = scores[top] / scores[top].sum()
    return sum(w * experts[i](x) for w, i in zip(weights, top))

Different tokens in the same sentence can go to different experts, and the same token can use different experts at different layers.

Key papers

• Shazeer et al. (2017), "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer," showed MoE layers with thousands of experts in language models.
• Lepikhin et al. (2020), "GShard," scaled MoE transformers across many devices for translation.
• Fedus, Zoph and Shazeer (2021), "Switch Transformers," simplified routing to a single expert per token (top-1) and scaled to over a trillion parameters.
• Jiang et al. (2024), "Mixtral of Experts," released an open-weights model with 8 experts per layer and top-2 routing: about 47 billion total parameters, about 13 billion active per token, with performance competitive with much larger dense models at the time.
• Dai et al. (2024), "DeepSeekMoE," used many smaller, fine-grained experts plus always-active shared experts to improve specialisation.

What experts actually learn

The name suggests a "maths expert," a "French expert" and a "code expert." Evidence mostly does not support that picture. The Mixtral paper analysed which experts were selected for different domains and did not find clear topic-based specialisation; routing patterns were more related to syntax and token types than to subject matter. Experts learn whatever division of labour reduces training loss, which tends to be fine-grained and hard to interpret.

So do not expect to "use the medical expert" or to remove experts to remove topics. Experts are an efficiency mechanism, not a modular knowledge base.

The engineering trade-offs

Load balancing

If the router sends most tokens to a few experts, those experts become bottlenecks and the rest go unused. Training adds a load-balancing objective that encourages even usage, and some designs cap how many tokens each expert can take per batch.

Memory: all experts must be loaded

Only some experts run per token, but all of them must be in memory, because any token might need any expert. A model with 47B total and 13B active parameters needs memory for 47B parameters. MoE saves compute, not memory.

Communication

At scale, experts are spread across many GPUs, and tokens must be sent to wherever their chosen experts live. This "all-to-all" communication can dominate run time and requires careful parallelism.

Training stability

MoE training can be less stable than dense training, with router collapse or uneven expert usage. Techniques such as router noise, auxiliary losses and careful initialisation address this.

Dense vs. MoE at a glance

Fine-tuning and quantizing MoE models

MoE models bring a few extra considerations when you adapt or compress them. Fine-tuning can disturb the router's learned balance, so many practitioners freeze the router or keep learning rates conservative, and adapter methods such as LoRA are often applied to attention layers and experts selectively. Quantization works well because most memory sits in expert weights, which tolerate low precision reasonably; some tools also offload rarely used experts to slower memory. In both cases, evaluate on your own tasks, because effects can differ from dense models of similar size.

What this means if you use or deploy models

• For API users: MoE is largely invisible. It is part of why large, capable models can be priced competitively. Compare models on your own tasks, not on architecture; see scaling laws in plain English for why capability per dollar keeps improving.
• For self-hosting: check total parameters for memory planning, and active parameters for speed estimates. A large MoE model may need multiple GPUs even if it runs fast once loaded. Quantization helps with memory; see quantization without tears.
• For local use: MoE models can be attractive on machines with lots of memory but modest compute, such as high-memory laptops, because speed depends on active parameters while memory depends on total parameters.
• For comparisons: do not compare an MoE model's active parameter count with a dense model's total count as if they were equivalent. Neither number alone predicts quality.

MoE is one of several efficiency ideas in modern models, alongside sparse attention for long contexts and speculative decoding for faster generation.

Frequently asked questions

What is a mixture-of-experts model?

A mixture-of-experts model replaces some of its layers with several parallel sub-networks called experts and a router that sends each token to only a few of them. This lets the model have many parameters while using only a fraction for each token.

What are active parameters?

Active parameters are the parameters actually used to process a single token, the shared layers plus the selected experts. They determine compute per token. Total parameters include all experts and determine memory requirements.

Do experts specialise in topics like maths or coding?

Not in an easily interpretable way. Analyses of open MoE models have found routing patterns linked more to syntax and token types than to subject areas.

Are MoE models cheaper to run?

They need less computation per token than a dense model of the same total size, which makes them faster and cheaper per token at scale. But they need memory for all parameters, so hosting them requires substantial hardware.