You’ve probably heard it like this: inside a Mixture of Experts model there’s an expert in maths, another in poetry and another in medicine, plus a boss who sends you to whoever knows your subject. It’s a nice explanation. And it’s wrong.
In the previous post I explained what an MoE is using a kitchen: a head chef, the router, hands each word to a few cooks, the experts. Now for the fun part: six things people say about MoEs that don’t survive a measurement. Each one comes with what the people who looked inside actually found.
Myth 1: “Each expert knows one topic”
The Mixtral authors asked exactly that: is there an expert for maths, another for biology, another for philosophy? They ran texts on each topic through the model and counted which experts each word went to. The splits came out almost identical: scientific papers, medical abstracts and philosophy texts used the same experts in the same proportions. Only one set of synthetic maths problems stood out a little.
What they did find is dumber and more interesting: experts split up pieces of text. The word self in Python almost always goes to the same expert. So does “Question” in English. And code indentation, the spaces at the start of a line, has its own fixed experts.
OpenMoE, a family of research models of up to 34 billion parameters, went further: the split depends mostly on which piece of word it is, and very little on the context. The piece “ed” goes to the same few experts in “preferred” and in “led”, even though they have nothing to do with each other.
There’s one exception that proves the rule: language. In OpenMoE, Simplified and Traditional Chinese prefer the same two experts, and Japanese and Korean share another one. But not because an expert “knows Chinese”: it’s because Chinese text is made of different pieces than English.
I thought I had an expert in jokes. Turns out I have one for commas.
Myth 2: “Mixtral 8x7B is eight 7B models glued together”
The name invites it: eight 7-billion models glued together, and a boss who picks two. If that were true, it would have 8 × 7 = 56 billion parameters (58 counting exactly, because Mistral 7B has 7.2 billion). It has 47 billion. Where’s the difference?
Mixtral comes from Mistral 7B, and a model like that has two kinds of parameters:
- The shared part, about 1.6 billion: each layer’s attention, the part that looks at the context, and the vocabulary, the table that turns words into numbers and numbers back into words.
- The kitchen, about 5.6 billion: the network in each layer that transforms each word on its own.
Mixtral doesn’t copy the whole model eight times: it copies only the kitchen. The shared part is there once, and the numbers add up:
| Shared part | Kitchens | Total | |
|---|---|---|---|
| Eight 7B models glued | 8 × 1.6 B | 8 × 5.6 B | 58 B |
| Mixtral 8x7B | 1.6 B | 8 × 5.6 B | 47 B |
| What it uses on each word | 1.6 B | 2 × 5.6 B | 13 B |
The last row is a bonus: the 13 billion “active” parameters from the previous post aren’t 2 × 7, but the shared part plus two kitchens.
One more detail that confuses people: there aren’t 8 big experts, but 8 per layer. With 32 layers that’s 256 small kitchens, and kitchen 3 in layer 1 has nothing to do with kitchen 3 in layer 2. “Expert 3” is a station number, not a character that walks through the model.
Myth 3: “The router picks one expert for your question”
Many people picture the boss reading your question, deciding “this is about programming” and handing the whole thing to one expert. That’s not how it works. The router decides for every word and in every layer: it picks 2 of 8 in the first layer, another 2 in the second, and so on up to layer 32.
There are 28 ways to pick 2 experts out of 8. With 32 layers, a single word can follow 28³² different paths through the model: about 2×10⁴⁶, a 2 followed by 46 zeros.
What does happen is that consecutive words tend to repeat experts. In Mixtral’s layer 15, the next word repeats the previous word’s first expert between 23.6% and 28.4% of the time, depending on the kind of text. By chance it would be 12.5%. It isn’t “one expert per question”, but it isn’t random either.
Myth 4: “The experts share the work on their own”
Leave the boss to it and you get what happens in plenty of kitchens: it always picks the same people. Shazeer and co-authors (Geoffrey Hinton among them) saw it back in 2017: the router ends up giving all the weight to a few experts, and the problem feeds itself, because the favourites train more, get better and get picked even more.
That’s why every MoE carries something to spread the load. The classic fix is an extra penalty during training when the split gets unbalanced. DeepSeek-V3 uses something else: a bias per expert that goes up when it’s idle and down when it’s overloaded, and is only used for choosing.
There’s a practical consequence. Many MoEs give each expert a quota, and when it’s full, the word goes uncooked in that layer. OpenMoE saw that this happens more to words near the end of a long conversation. DeepSeek-V3 says that, thanks to its balancing, it doesn’t drop any.
Myth 5: “If you don’t use a topic, remove its expert”
If experts aren’t about topics, there’s no “medicine expert” to remove. But can you prune experts at all? Yes, carefully. Lu and co-authors (2024) removed 2 of the 8 experts in every layer of Mixtral, choosing them with general text:
| What they removed | Parameters | Average of 8 tests (from 67.58) |
|---|---|---|
| 2 of 8 experts per layer | 24% fewer | 2.9 points lower |
| 4 of 8 experts per layer | — | 7.1 points lower |
Looks cheap, but there’s a catch: with that same pruning, accuracy on maths problems (GSM8K) fell from 58.61 to 41.02. What’s spare for general text was exactly what maths needed. To prune a maths model they had to choose the experts with maths data, and the two choices only matched in 4 of the 32 layers.
The rule: prune by measuring with your own data, never by name.
Myth 6: “The head chef keeps refining the split throughout training”
It sounds logical: the model learns for months, so the router would keep improving its split all that time. In OpenMoE it didn’t: the split for each piece of word was settled early, at the start of training, and then barely changed. Each piece kept going to the same experts until the end.
It’s one of the reasons the split ends up looking more like a filing cabinet of text pieces than a team of specialists.
In short: my cooks don’t specialise in dishes but in particular ingredients. And the split was settled on day one.
Myth or reality?
Summary
| Myth | What was measured |
|---|---|
| Each expert knows one topic | They split pieces of text (self, indentation, “ed”). Language is the exception, because it changes the pieces |
| 8x7B is eight 7B models | Only the kitchen is copied; the shared part is there once: 47 B, not 58 |
| One expert per question | 2 of 8 per word and per layer: about 2×10⁴⁶ paths |
| They share the work on their own | Without balancing, a few take everything and it gets worse by itself |
| Remove the expert you don’t use | You can prune, but by measuring: what’s spare in general was needed for maths |
| The split improves throughout training | In OpenMoE it was settled early and barely changed |
And the two myths from the other post: an MoE doesn’t fit wherever a small model fits, and it isn’t always faster. They’re in Mixture of Experts (MoE): the trick behind giant LLMs.
Sources
Checked on 5 October 2026, in each paper’s PDF.
- Mixtral: Jiang et al., arXiv:2401.04088. Table 1 (32 layers, 8 experts, 2 active), §5 (no topic split;
self, “Question” and indentation) and table 5 (expert repetition between consecutive words). - OpenMoE: Xue et al., arXiv:2402.01739, §4: routing by word piece, language preferences, early-settled routing and words dropped near the end of a sequence.
- Self-reinforcing imbalance: Shazeer et al., arXiv:1701.06538, §4.
- Bias-based balancing and no dropped words: DeepSeek-AI, DeepSeek-V3, arXiv:2412.19437, §2.1.2.
- Mixtral parameter breakdown: my own calculation from its table 1 and its published configuration. It gives 46.7 B in total and 12.9 B active, matching the official numbers.
- Expert pruning: Lu et al., arXiv:2402.14800: §3 and figure 4 (maths) and table 2 (average of 8 tests).
- Specialisation with fine-grained and shared experts (what DeepSeekMoE aims for): Dai et al., arXiv:2401.06066.
How this post was made: from my notes, with Claude’s help for research and writing. The 2×10⁴⁶ paths are my own calculation from Mixtral’s table 1. The drawings are my own.