What is Mixture of Experts? — one analogy

1The analogy

Walk into a bank branch with ten teller windows and the first thing you do is take a numbered ticket. The ticket tells you which window to go to, and until your business is done, the other nine windows are busy serving somebody else, with nothing to do with you. However big the branch is, only one or two windows are actually handling your business.

A mixture-of-experts model looks like this. The inside is split into several branches, and for every piece of text that comes in, a router picks which branch handles it. Add more windows and the branch can handle a wider range of business, but any one customer's wait doesn't grow in proportion to the number of windows.

The windows with nobody at them still have to keep their lights on and stay staffed, though. Rent gets paid on all ten regardless. That's exactly why this structure isn't free.

2In detail

A router picks the branch

For every single piece of text passing through the model, a small router scores it first. This piece goes to branch three and branch seven; the next piece goes to branch one and branch three — something like that. Out of dozens of branches, usually only about two get picked.

Whatever comes back from the chosen branches gets merged back into one. Rather than just adding them together, they're blended in proportion to the score the router assigned — the branch picked with more confidence gets more weight in the mix.

The router gets trained too. Early on it splits things almost randomly; as training goes on, it settles into a pattern based on which branch tended to produce better results for which kind of piece. Nobody hand-assigns a topic to each branch.

Large overall, but only a slice runs each time

An ordinary model uses every one of its internal values once for every piece of text that passes through. Double the model and the computation doubles too — answers get slower, and cost doubles along with it.

Mixture of experts breaks that link. Grow the number of branches to eight and only two still run each time, so the amount of knowledge the model holds grows while the work done per calculation barely moves. That's why a model built this way is usually described with two numbers: its total size, and the size actually active at any one time. Judging performance or hardware needs off the first number alone will lead you astray.

It looks like a cheap way to grow bigger, but it has clear limits, too. However many branches you add, the depth of computation any single piece goes through as it passes stays the same — growing this way doesn't make the model smarter in proportion to its growth.

The branches aren't topic specialists

"Experts" invites the picture of one branch handling math, another handling translation. Look closely and it's rarely that tidy. One branch might light up heavily around punctuation; another might respond to a particular language's grammatical endings; most split along lines that are hard to describe in plain language at all.

Even within a single sentence, different pieces go to different branches. A branch doesn't handle a whole sentence at once — text keeps splitting apart and merging back together piece by piece. That's also why you can't pull out one branch and use it on its own.

Every branch has to stay ready, occupied or not

Less computation doesn't mean less memory. There's no way to know ahead of time which branch a given piece will need, so every branch has to be loaded and ready to go at any moment. A model with a large total size still demands large hardware. That's also why it's hard to run one of these on a personal computer.

Crowding is another issue. Left alone, pieces pile up on whichever branches happen to be popular, so some branches stay constantly busy while others sit mostly idle. An idle branch gets less training and drifts further toward being unused. So training usually adds a penalty for piling too much onto one branch, along with a cap on how many pieces any one branch can take at once — pieces that get turned away once the cap is hit just skip that layer entirely.

3More precisely

In a mixture-of-experts model, what splits apart isn't a whole layer — it's a specific section within each layer. The rest of the computation runs the same way for every piece; only that one routed section differs piece by piece. The number of branches selected is usually fixed at around two.

The analogy breaks down in a place worth naming. At a bank, a customer finishes their business at one window, but inside the model, a single piece pulls a new numbered ticket at every layer it passes through — with dozens of layers, that means dozens of fresh routing decisions. And a bank teller knows their own job, but a branch carries no label like that; only after training finishes can a person look inside and guess at what a branch tends to handle. The router doesn't split pieces by meaning either — it splits by a numerical outcome, so two sentences that look nearly identical can still land on different branches. And picking a branch isn't a smooth calculation — it's picking a few and discarding the rest, which is part of why this structure tends to make training less stable.

4Try it yourself

5Common misconceptions

It's easy to think a mixture-of-experts model is lightweight, but actually only the computation is reduced — it still needs memory for its full size.
It's easy to think each branch specializes in one topic, but actually the split happens on its own during training, along lines that are hard for a person to read.
It's easy to think a bigger total size means proportionally smarter, but actually the size actually active at any moment is much smaller, so the two numbers need to be read together.

7One-line summary

In shortMixture of experts splits a model into several windows and calls up only the window a piece of text actually needs, growing its size while keeping the computation done each time down.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02