Exercises with solutions — Mixture-of-Experts: routing and capacity

General instruction: every answer must show data, transformation, result, one check, and one limitation. A bare number or copied definition is insufficient.

Starting data: Routing example reduced to three experts for the calculation: scores [2.1,1.8,0.2] give softmax about [0.529,0.392,0.079]. With top-2, experts 1 and 2 are active. If expert 1 has reached capacity, routing must apply the overflow policy.

Boundary to retain: Activation names and exact expert counts attributed to a named model remain source-reported until primary evidence is attached.

Exercise 1 — Calculated trace — Why experts

Reproduce and annotate the chain input → state → output. Increase the first expert score from 2.1 to 2.52 and keep [1.8,0.2] for the others. Recompute softmax and check whether top-2 changes.

Deliverable: a data → operation → result → interpretation table, plus two sentences about the changed value.

Worked solution

Routing example reduced to three experts for the calculation: scores [2.1,1.8,0.2] give softmax about [0.529,0.392,0.079]. With top-2, experts 1 and 2 are active. If expert 1 has reached capacity, routing must apply the overflow policy.

Solved variant: Weights move from about [0.529,0.392,0.079] to [0.631,0.307,0.062]. Experts 1 and 2 remain top-2, but expected load concentrates more on expert 1; capacity and overflow policy become more critical.

MoE decouples the two: E subnetworks (experts) exist, only k activate per token. In the trace: 64 routed experts + 1 shared = 26B parameters, but (2+1) × 0.4 = 1.2B active per token — a ratio of ≈ 21×. The router is a small projection: h_t → one logit per expert. Worked example over 3 experts: [2.1, 1.8, 0.2] → softmax [0.529, 0.392, 0.079]. These scores are decisions learned through the global loss — not human categories. The minimum check covers dimensions, sign, and order of magnitude. If the observed change contradicts the prediction, locate the first operation whose direction changes instead of fixing only the final line.

Rubric Exercise 1 — /10

Criterion	Points
Explicit data and shapes	2
Traceable calculation	3
Prediction before variation	2
Interpretation and check	2
Named limitation	1

Exercise 2 — Diagnose a seductive explanation — Top-k and mixture

A colleague claims: « Top-k and mixture proves the system will be accurate, fast, and stable in every context. »

Separate mechanism, assumption, observation, and conclusion.
Name two correct lesson elements and two unsupported extrapolations.
Propose a bounded experiment with controlled variable, metric, and stop threshold.
Rewrite the claim as one defensible sentence.

Reasoned solution

Top-k decides: k = 2 experts per token, weights renormalized — 0.529 and 0.392 become 0.574 and 0.426 (dividing by 0.921). An always-on shared expert completes it: output = Σ weight × expert + shared + residual. Capacity is a tensor dimension: C = ceil(T×k/E × f) = ceil(4 × 1.25) = 5 here. The trace shows it: E1 receives 6 requests for 5 seats — the 6th arrival is dropped, its output carrying only 0.426 of the intended mixture (logits assumed equal to t1’s). Activation names and exact expert counts attributed to a named model remain source-reported until primary evidence is attached.

The claim mixes a local relation with a global guarantee. A defensible version states only the observed mechanism, test conditions, and measured metric. Stop the test if shapes become invalid, the metric crosses the declared degradation threshold, or another variable changes.

Rubric Exercise 2 — /10

2 points per element: separation, lesson grounding, extrapolations, protocol, and rewrite.

Exercise 3 — Architecture decision and transfer — Load balancing

You must reproduce the worked case “Routing example reduced to three experts for the calculation: scores [2.1,1.8,0.2] give softmax about [0.529,0.392,0.079]. With top-2, experts 1 and 2 are active. If expert 1 has reached capacity, routing must apply the overflow policy.” under two conditions. Option A uses the full chain through “Load balancing.” Option B is a transparent baseline that retains “Why experts,” calculates the expected output directly, and does not use the compression or adjustment mechanism studied. Build a decision record containing:

the workload and dominant constraint;
each option’s mechanism, without slogans;
one quality, memory, or latency prediction;
one case where your preferred procedure loses;
an A/B protocol, metrics, and rollback threshold;
a bounded verdict: choose, defer, or reject.

Elements of a strong solution

An auxiliary loss pushes the distribution toward uniform. It is a slider, not a switch: too weak, collapse onto one expert; too strong, it overrules the router even when E1 genuinely is the right choice. The all-to-all paces the step: two round trips per top-2 token, a rhythm set by the busiest expert (E1 at 100%). The 1.2B active parameters measure compute; the network bills the slots — used or not. Mixed: established mechanisms + source-reported Kimi K3-style choices.

A strong answer does not make the newer mechanism the default winner. It retains a measurable baseline, sets thresholds before testing, and separates component cost from whole-system behavior. The verdict names what remains uncertain and the next evidence that could change it.

Rubric Exercise 3 — /15

Criterion	Points
Framing and baseline	3
Compared causal chains	4
Protocol and metrics	4
Rollback threshold	2
Bounded verdict	2

Extension

Repeat Exercise 3 after reversing the dominant constraint. If you optimized memory, impose a strict quality floor; if you optimized fidelity, halve the memory budget. Identify the first part of the verdict that changes and the evidence required.

Review before submission

Review the packet as if another group had to reproduce it without speaking to you. Are all starting values or assumptions present? Are shapes or roles stated before operations? Does the prediction truly precede the observation? Is the result translated into behavior rather than left as an isolated number? Did you test a boundary value and identify a stop condition? Does the procedure or architecture choice retain a measurable baseline and a rollback threshold set before the test? Finally, highlight one sentence that states what is established, one that remains a hypothesis, and one measurement that could change your verdict. If any element is missing, the work is not reproducible.

Reference appendix for correction

Chapter 14 — Mixture-of-Experts

14.1 Router and experts

Goal: give the model many possible feed-forward specialists without running every specialist for every token. Mixture-of-Experts is abbreviated MoE.

Intuition: A hospital has many specialists, but reception sends each patient only to the few most relevant doctors.

Step by step

Compute routing scores r = xW_router.
Select the top-k experts, for example the best 2.
Run only those experts on token representation x.
Combine their outputs using routing weights.
A shared expert, if used, processes every token and can learn common skills.

Worked example: scores for four experts are [0.1, 2.0, 1.5, −0.2]. Top-2 routing chooses experts 2 and 3. After a softmax over those two scores, their outputs are blended; experts 1 and 4 do no token-specific work for this token.

Why it matters: MoE can increase total parameter capacity while keeping active computation per token much smaller than running all experts.

Quick check: If there are 64 experts and top-2 routing, how many routed experts run for one token? Answer: 2, plus any always-on shared experts.

14.2 Load balance and communication

Goal: understand why routing is not free. If most tokens choose one expert, that expert becomes overloaded while others wait.

Intuition: A supermarket with ten cashiers is still slow if everyone queues at one register.

Step by step

Training may add balancing objectives.
Implementations may limit expert capacity.
On multiple devices, tokens must be sent to the devices holding their chosen experts.
Communication time can become a major cost.

Worked example: An architecture with impressive arithmetic savings may still be slow if routing causes heavy network traffic.

Why it matters: Real performance depends on hardware placement, batch size, kernels and communication, not parameter counts alone.

Quick check: What is load imbalance? Answer: work is distributed unevenly, leaving some experts overloaded and others underused.

14.3 SiLU, gated units and SiTU caveat

Goal: understand the baseline before a source-specific activation. The Sigmoid Linear Unit, SiLU, is SiLU(a)=a×sigmoid(a). A common gated unit multiplies SiLU(a) by another branch b coordinate by coordinate.

Intuition: One branch decides how open the gate is; the other branch carries the information.

Step by step

If a=0, sigmoid(a)=0.5 and SiLU(a)=0.
If a is strongly positive, SiLU(a) is close to a.
The element-wise product lets each coordinate control information flow.

Worked example: The supplied source names a SiTU activation. Unless an official equation or released implementation is available, treat its exact formula and claimed benefit as source-dependent rather than established fact.

Why it matters: Learning the baseline makes it possible to evaluate any proposed modification.

Quick check: In a gated unit, what does element-wise multiplication mean? Answer: multiply matching coordinates, not every coordinate with every other coordinate.

Sources and evidence boundary

Owner-supplied bilingual course packet, Chapter 14.
Shazeer et al., “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, ICLR (2017).
Fedus, Zoph & Shazeer, “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity”, JMLR (2022).
Dai et al., “DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models”, ACL (2024).
Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Scope: Mixed: established mechanisms + source-reported Kimi K3-style choices. These references support the session frame; they do not turn a reported product choice into an independently verified result.