Training / Standard term
Mixture of Experts (MoE)
A neural architecture with several parameter blocks called experts and a router that selects a subset for each token or input.
Sparse MoE models can increase total parameter capacity without using every expert for every token. Experts may specialize, overlap, or remain difficult to interpret; the name does not guarantee human-readable domains. Efficiency depends on routing, batching, memory, communication between devices, and implementation, not active parameter count alone.
Builder example
Total and active parameter counts provide incomplete deployment information. Measure output quality, memory footprint, throughput, latency, and hardware requirements for the actual implementation. A sparse model can be compute-efficient while costly to store or distribute.
Common confusion: Not every expert must sit on one device, and not every disclosed total parameter count is comparable. Distributed serving and architecture details determine the memory and communication cost.