Mixture of Experts: How LLMs Scale Efficiently

Key Takeaways

The Core Idea: Conditional Computation

In a standard dense transformer, every parameter is used for every token. A 70B parameter model performs 70B parameter computations per token. Mixture of Experts (MoE) breaks this: instead of one large feed-forward network, each transformer layer has multiple 'expert' feed-forward networks. A router network selects which experts to activate for each token, typically 2 out of 8, 16, or even 128 experts.

The result: a model with 400 billion total parameters might only use 50 billion 'active' parameters per token. You get the capability of a large model at the inference cost of a smaller one. This is the key insight that MoE exploits: not all knowledge is needed for all tasks, so routing inputs to specialized sub-networks reduces wasted computation.

How Routing Works

The router is a small linear layer that takes the token representation and produces a score for each expert. The top-k experts by score are selected, and their outputs are combined using a weighted sum (weights proportional to scores). The router is trained jointly with the rest of the model, it learns to send different types of content to different experts.

Load balancing is critical: if all tokens route to the same few experts, the others are wasted capacity. Auxiliary losses during training encourage the router to distribute load evenly across experts. DeepSeek's innovation of auxiliary-loss-free load balancing, using a bias term adjusted dynamically rather than a training loss, is a notable recent improvement.

Notable MoE Models

GPT-4 is widely believed to use MoE architecture, though OpenAI has never confirmed this. Mixtral 8x7B (Mistral) was the first widely adopted open-weight MoE model: 8 experts per layer, 2 active per token, 47B total parameters but 12.9B active, it outperformed Llama 2 70B on most benchmarks while being roughly 2× cheaper to run.

Meta's Llama 4 Maverick uses 128 experts (2 active), with 400B+ total parameters. DeepSeek V3 uses a 671B MoE with 37B active parameters, achieving GPT-4o-level performance at a fraction of the inference cost. Google's Gemini models are also believed to use MoE at scale.

MoE Tradeoffs and Challenges

MoE's benefits come with real costs. Memory: all expert weights must be loaded into GPU memory even though only 2 are active per token. Serving a 400B MoE model requires 4× the GPU memory of a 100B dense model, even if inference compute is similar. This limits MoE's advantage when memory bandwidth rather than compute is the bottleneck.

Training instability: MoE models are harder to train than dense models. Expert collapse (one expert dominates), routing volatility (experts change frequently during training), and load imbalance are all failure modes that require careful engineering. The research community has made substantial progress, but MoE training remains more complex than dense training.