Your bookmarks

Loading your bookmarks

Latest updates

New courses, skills, book chapters, and site improvements will appear here.

Create free accountSign upDashboardDashboard

Mixture of Experts (MoE)

  • Training
  • Standard term
3 steps

A Mixture of Experts (MoE) is a model design split into many sub-networks, called “experts,” with a router that picks a few of them to handle each piece of input.

Each token, a word or piece of a word, runs through only the experts it is sent to. So the model can hold far more parameters, its learned numbers, than it uses on any one token. All the experts still sit in memory, because the next token may need any of them.

Where you’ll meet it

You want to download a model and run it on your own laptop, so your journal never leaves it. One listing describes a mixture-of-experts model with a large total parameter count and a much smaller active count, and the small number looks like it would fit.

What to do

Plan around the total count. Look up the published memory requirement for the exact version you'd download, and compare it with the memory your laptop has. A compressed copy, called a “quantized” model, needs less memory and gives up a little quality.

Easy to mix up

Subagents are several separate AI agents, each with its own task. A Mixture of Experts is one model, and its experts are parts inside it that a router picks for each piece of input.

Check yourself

A post says one expert in a mixture-of-experts model is the “cooking expert” that answers recipe questions. Which description is closer to how these models work?

Go deeperBalancing the router and placing the experts are the hard parts

Robert Jacobs, Michael Jordan, and colleagues proposed mixtures of experts in 1991. Noam Shazeer and colleagues at Google scaled the idea up for language models in 2017, with up to thousands of experts in one model. Mistral's Mixtral, which anyone could download and run from 2023, brought the design into widely used models.

Training needs care to keep the router from sending most tokens to a few favorite experts, which leaves the rest undertrained. Many designs add a penalty during training that grows when the router's choices pile onto a few experts.

On one machine, a mixture of experts tends to answer at close to the speed of a model the size of its active count. Spread across many machines, tokens have to travel to wherever their experts sit, so speed depends as much on how the experts are divided among the machines as on parameter counts.

Practice Modern AI

Or take the learning quiz on a mix of every set