Teacher guide — Mixture-of-Experts: routing and capacity

Duration: 120 minutes
Positioning: Trace router scores, top-k, shared/routed experts, capacity, and communication.
Expected evidence: Mixed: established mechanisms + source-reported Kimi K3-style choices.

Observable outcomes and preparation

Run top-k routing.
Diagnose overload.
Separate total and active parameters.

Before class, the instructor runs the worked case and lab, prints the exercise packet, prepares a four-column board—assumption, prediction, observation, delta—and checks that every mathematical notation is paired with dimensions. The demonstration must not become slide reading.

Diagnostic

Explain in one sentence: Why experts. What observation would falsify your explanation?
Explain in one sentence: Router. What observation would falsify your explanation?
Explain in one sentence: Top-k and mixture. What observation would falsify your explanation?

Teaching decision: if two of three answers remain nominal or lack a validity condition, rebuild the vocabulary with a numerical example before any formula. A fluent but unfalsifiable answer does not count as mastery.

Timed plan

Time	Activity	Observable evidence
0–10 min	Individual diagnostic, then pair comparison	Three answers and one named uncertainty
10–25 min	Situation and vocabulary	Annotated input → state → output diagram
25–55 min	Develop the mechanism on the board	Shapes, assumptions, and intermediate calculation visible
55–75 min	Worked case with deliberate errors	Reasoned correction, not only the right number
75–95 min	Causal lab: predict, change one variable, run	Prediction / observation / delta table
95–112 min	Exercises 1 and 2 with peer correction	Retained artifact and applied rubric
112–120 min	Exit ticket and transfer	Mechanism, boundary, next experiment

Teaching notes

Chapter 14 — Mixture-of-Experts

14.1 Router and experts

Goal: give the model many possible feed-forward specialists without running every specialist for every token. Mixture-of-Experts is abbreviated MoE.

Intuition: A hospital has many specialists, but reception sends each patient only to the few most relevant doctors.

Step by step

Compute routing scores r = xW_router.
Select the top-k experts, for example the best 2.
Run only those experts on token representation x.
Combine their outputs using routing weights.
A shared expert, if used, processes every token and can learn common skills.

Worked example: scores for four experts are [0.1, 2.0, 1.5, −0.2]. Top-2 routing chooses experts 2 and 3. After a softmax over those two scores, their outputs are blended; experts 1 and 4 do no token-specific work for this token.

Why it matters: MoE can increase total parameter capacity while keeping active computation per token much smaller than running all experts.

Quick check: If there are 64 experts and top-2 routing, how many routed experts run for one token? Answer: 2, plus any always-on shared experts.

14.2 Load balance and communication

Goal: understand why routing is not free. If most tokens choose one expert, that expert becomes overloaded while others wait.

Intuition: A supermarket with ten cashiers is still slow if everyone queues at one register.

Step by step

Training may add balancing objectives.
Implementations may limit expert capacity.
On multiple devices, tokens must be sent to the devices holding their chosen experts.
Communication time can become a major cost.

Worked example: An architecture with impressive arithmetic savings may still be slow if routing causes heavy network traffic.

Why it matters: Real performance depends on hardware placement, batch size, kernels and communication, not parameter counts alone.

Quick check: What is load imbalance? Answer: work is distributed unevenly, leaving some experts overloaded and others underused.

14.3 SiLU, gated units and SiTU caveat

Goal: understand the baseline before a source-specific activation. The Sigmoid Linear Unit, SiLU, is SiLU(a)=a×sigmoid(a). A common gated unit multiplies SiLU(a) by another branch b coordinate by coordinate.

Intuition: One branch decides how open the gate is; the other branch carries the information.

Step by step

If a=0, sigmoid(a)=0.5 and SiLU(a)=0.
If a is strongly positive, SiLU(a) is close to a.
The element-wise product lets each coordinate control information flow.

Worked example: The supplied source names a SiTU activation. Unless an official equation or released implementation is available, treat its exact formula and claimed benefit as source-dependent rather than established fact.

Why it matters: Learning the baseline makes it possible to evaluate any proposed modification.

Quick check: In a gated unit, what does element-wise multiplication mean? Answer: multiply matching coordinates, not every coordinate with every other coordinate.

Running the worked case

Routing example reduced to three experts for the calculation: scores [2.1,1.8,0.2] give softmax about [0.529,0.392,0.079]. With top-2, experts 1 and 2 are active. If expert 1 has reached capacity, routing must apply the overflow policy.

Do not reveal the result at once. Ask learners to predict the next operation, its shape, and the expected sign. After each line ask: “What changed? What stayed fixed? Which assumption did we use?” A calculation error repaired with a causal chain is worth more than a guessed result.

Lab protocol

Write a qualitative and, where possible, numerical prediction before touching a control.
Change one variable only; retain a capture or record initial and final values.
Explain the delta through the mechanism, not “the tool did that.”
Test one boundary value and state where the model stops representing a real system.

Misconceptions

#	Observable misconception	Grounded correction	Probe
1	“Why experts guarantees the outcome without assumptions or measurement.”	MoE decouples the two: E subnetworks (experts) exist, only k activate per token. In the trace: 64 routed experts + 1 shared = 26B parameters, but (2+1) × 0.4 = 1.2B active per token — a ratio of ≈ 21×.	Ask for a counterexample, then restate the mechanism with its validity condition.
2	“Router guarantees the outcome without assumptions or measurement.”	The router is a small projection: h_t → one logit per expert. Worked example over 3 experts: [2.1, 1.8, 0.2] → softmax [0.529, 0.392, 0.079]. These scores are decisions learned through the global loss — not human categories.	Ask for a counterexample, then restate the mechanism with its validity condition.
3	“Top-k and mixture guarantees the outcome without assumptions or measurement.”	Top-k decides: k = 2 experts per token, weights renormalized — 0.529 and 0.392 become 0.574 and 0.426 (dividing by 0.921). An always-on shared expert completes it: output = Σ weight × expert + shared + residual.	Ask for a counterexample, then restate the mechanism with its validity condition.

Boundary to maintain: Activation names and exact expert counts attributed to a named model remain source-reported until primary evidence is attached.

Probing questions

If we remove or reverse Why experts, which output changes first, and what observation would show it?
If we remove or reverse Router, which output changes first, and what observation would show it?
If we remove or reverse Top-k and mixture, which output changes first, and what observation would show it?
If we remove or reverse Capacity, which output changes first, and what observation would show it?
If we remove or reverse Load balancing, which output changes first, and what observation would show it?
If we remove or reverse Communication, which output changes first, and what observation would show it?

Assessment

Level	Criterion
0	Repeats terms without connecting input, transformation, and output.
1	Describes the chain but checks neither shape nor assumption.
2	Executes the case, explains the result, and names one limitation.
3	Transfers to a new case, compares an alternative, and proposes a measurement that could invalidate the choice.

Exit threshold: level 2 on the worked case and at least one exercise; a memorized formula without interpretation remains level 1.

Observation and remediation protocol

During discussion, the instructor records evidence rather than impressions. Evidence of understanding contains a named object, a justified transformation, and a checkable consequence. If a learner gives the right result without a chain, ask for the preceding line. If the chain is coherent but the result is wrong, preserve the chain and isolate the arithmetic error. If vocabulary from another concept is used, compare both mechanisms in an input, state, output, cost, and boundary table. Remediation targets the first break only: vocabulary, shapes, operation, interpretation, or claim scope. After correction, use a neighboring case with a changed value; success on the same example does not prove transfer. For pair work, assign operator and verifier roles, then swap. The verifier does not supply the answer: they request an assumption, check the shape, and ask what observation could contradict the reasoning. The instructor retains the exit ticket and classifies the dominant break. The next session opens with a three-minute problem aimed at that break instead of repeating the whole lesson.

Differentiation

Support: provide shapes and the first transformation; let the learner complete interpretation and boundary.
Core path: worked case without result, lab with one assigned variable, diagnostic exercise.
Extension: change one assumption, compare two mechanisms, and define the metric that would decide between them.

Post-session follow-up

Within twenty-four hours, return each annotated exit ticket with one priority, the exact resource to reopen, and a mini-case different from the worked case. Revision requires three items: a written prediction, a retained trace, and one sentence explaining the delta. At the next session, sample two submissions: one that repaired the break and one that remains ambiguous. Discuss them anonymously, then state the criterion that separates them. Do not use completion rate as evidence of mastery. Evidence of remediation is a correct chain on a new case with a stated boundary. If the same break appears in more than one third of the group, repair the support or demonstration before blaming learners.

Follow-up closes only when the new artifact shows the causal chain, check, and boundary—not merely when a file has been submitted.

Sources and evidence boundary

Owner-supplied bilingual course packet, Chapter 14.
Shazeer et al., “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, ICLR (2017).
Fedus, Zoph & Shazeer, “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity”, JMLR (2022).
Dai et al., “DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models”, ACL (2024).
Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Scope: Mixed: established mechanisms + source-reported Kimi K3-style choices. These references support the session frame; they do not turn a reported product choice into an independently verified result.

Exit ticket

In no more than six lines: mechanism; calculation or trace; observation; boundary; evidence level; next experiment. The instructor marks one priority causal break for revision.