1. Why experts
The problem: To gain capacity, a dense block must grow — and every token pays for the whole block: ×20 parameters, ×20 FLOPs per token, even to predict “the”. The compute bill tracks capacity instead of tracking need.
The idea: MoE decouples the two: E subnetworks (experts) exist, only k activate per token. In the trace: 64 routed experts + 1 shared = 26B parameters, but (2+1) × 0.4 = 1.2B active per token — a ratio of ≈ 21×.
Why / at what price: Total capacity without the equivalent dense cost — that is the pitch. The immediate price: the 26B must reside in GPU memory at load time. The 21× ratio speaks of FLOPs — never of memory nor, as we will see, of latency.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
2. Router
The problem: Who decides which experts see which token? A hand-written switchboard — by language? by topic? — would be rigid and wrong; you need a per-token decision, learned with the rest of the model.
The idea: The router is a small projection: h_t → one logit per expert. Worked example over 3 experts: [2.1, 1.8, 0.2] → softmax [0.529, 0.392, 0.079]. These scores are decisions learned through the global loss — not human categories.
Why / at what price: The learned switchboard adapts without intervention. The price: illegibility — nothing guarantees an expert “is” math or code; observed specialization is statistical and can shift with the next rebalancing.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
3. Top-k and mixture
The problem: Running every expert “a little” would ruin the economics; keeping only one makes the choice brutal and the gradient fragile. How many experts to activate, and how to recombine their outputs?
The idea: Top-k decides: k = 2 experts per token, weights renormalized — 0.529 and 0.392 become 0.574 and 0.426 (dividing by 0.921). An always-on shared expert completes it: output = Σ weight × expert + shared + residual.
Why / at what price: Small k preserves the economics; the mixture keeps a gradient through two paths. The price: every token now hangs on a discrete decision — one flipped logit changes the whole compute path, less continuous behavior than a dense block.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
4. Capacity
The problem: 8 tokens, top-2, 4 experts: 16 assignments to seat. But a GPU tensor has fixed dimensions — an expert cannot accept “as many as needed”. What happens when demand exceeds room?
The idea: Capacity is a tensor dimension: C = ceil(T×k/E × f) = ceil(4 × 1.25) = 5 here. The trace shows it: E1 receives 6 requests for 5 seats — the 6th arrival is dropped, its output carrying only 0.426 of the intended mixture (logits assumed equal to t1’s).
Why / at what price: C bounds per-expert memory and compute — the tensor requires it. The price: the overflow policy (drop, reroute, queue) becomes an architecture choice that touches quality; and a generous f buys fewer drops at the cost of padding — f = 2 leaves 50% of slots empty.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
5. Load balancing
The problem: The trace gives 6/4/3/3: E1 saturates while E3 and E4 run at 60%. Taken to the extreme, a “lazy” router sends everything to one expert — and the others never learn anything again.
The idea: An auxiliary loss pushes the distribution toward uniform. It is a slider, not a switch: too weak, collapse onto one expert; too strong, it overrules the router even when E1 genuinely is the right choice.
Why / at what price: The right setting keeps every expert alive without crushing useful specialization. The price: one more loss term to watch, whose effect reads in load distribution — a systems metric — as much as in quality.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
6. Communication
The problem: The 4 experts live on 4 GPUs. Every top-2 token must travel to its experts and back — and the 20 reserved slots, 25% padding included, cross the network in both directions. Where does the latency go?
The idea: The all-to-all paces the step: two round trips per top-2 token, a rhythm set by the busiest expert (E1 at 100%). The 1.2B active parameters measure compute; the network bills the slots — used or not.
Why / at what price: Well placed — co-located experts, grouped batches — MoE keeps its promises. The generic price: latency is often communication-dominated; “active parameters” is an excellent FLOPs indicator and a very poor speed indicator.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Development from the course source
Chapter 14 — Mixture-of-Experts
14.1 Router and experts
Goal: give the model many possible feed-forward specialists without running every specialist for every token. Mixture-of-Experts is abbreviated MoE.
Intuition: A hospital has many specialists, but reception sends each patient only to the few most relevant doctors.
Step by step
-
Compute routing scores r = xW_router.
-
Select the top-k experts, for example the best 2.
-
Run only those experts on token representation x.
-
Combine their outputs using routing weights.
-
A shared expert, if used, processes every token and can learn common skills.
Worked example: scores for four experts are [0.1, 2.0, 1.5, −0.2]. Top-2 routing chooses experts 2 and 3. After a softmax over those two scores, their outputs are blended; experts 1 and 4 do no token-specific work for this token.
Why it matters: MoE can increase total parameter capacity while keeping active computation per token much smaller than running all experts.
Quick check: If there are 64 experts and top-2 routing, how many routed experts run for one token? Answer: 2, plus any always-on shared experts.
14.2 Load balance and communication
Goal: understand why routing is not free. If most tokens choose one expert, that expert becomes overloaded while others wait.
Intuition: A supermarket with ten cashiers is still slow if everyone queues at one register.
Step by step
-
Training may add balancing objectives.
-
Implementations may limit expert capacity.
-
On multiple devices, tokens must be sent to the devices holding their chosen experts.
-
Communication time can become a major cost.
Worked example: An architecture with impressive arithmetic savings may still be slow if routing causes heavy network traffic.
Why it matters: Real performance depends on hardware placement, batch size, kernels and communication, not parameter counts alone.
Quick check: What is load imbalance? Answer: work is distributed unevenly, leaving some experts overloaded and others underused.
14.3 SiLU, gated units and SiTU caveat
Goal: understand the baseline before a source-specific activation. The Sigmoid Linear Unit, SiLU, is SiLU(a)=a×sigmoid(a). A common gated unit multiplies SiLU(a) by another branch b coordinate by coordinate.
Intuition: One branch decides how open the gate is; the other branch carries the information.
Step by step
-
If a=0, sigmoid(a)=0.5 and SiLU(a)=0.
-
If a is strongly positive, SiLU(a) is close to a.
-
The element-wise product lets each coordinate control information flow.
Worked example: The supplied source names a SiTU activation. Unless an official equation or released implementation is available, treat its exact formula and claimed benefit as source-dependent rather than established fact.
Why it matters: Learning the baseline makes it possible to evaluate any proposed modification.
Quick check: In a gated unit, what does element-wise multiplication mean? Answer: multiply matching coordinates, not every coordinate with every other coordinate.
Complete worked case
Routing example reduced to three experts for the calculation: scores [2.1,1.8,0.2] give softmax about [0.529,0.392,0.079]. With top-2, experts 1 and 2 are active. If expert 1 has reached capacity, routing must apply the overflow policy.
Reading method: write the data, state every object shape, perform one transformation, and interpret the result before continuing.