# Exercises with solutions — Residual streams and attention over depth **General instruction:** every answer must show data, transformation, result, one check, and one limitation. A bare number or copied definition is insufficient. > **Starting data:** Depth scores [2,1,0] over {x₂₄,x₁₂,x₀} give softmax ≈ [0.67,0.24,0.09]. The mixture favors x₂₄, the most recent checkpoint, while the other two together retain 33%. > > **Boundary to retain:** AttnRes is presented as a course mechanism and source-reported case study; exact Kimi K3 details require a primary source. ## Exercise 1 — Calculated trace — Residual connection Reproduce and annotate the chain `x_{l+1}=x_l+F_l(x_l)`. Increase the third checkpoint score from 2 to 2.4 in [0,1,2]. Recompute depth-softmax weights. **Deliverable:** a data → operation → result → interpretation table, plus two sentences about the changed value.
Worked solution Depth scores [2,1,0] over {x₂₄,x₁₂,x₀} give softmax ≈ [0.67,0.24,0.09]. The mixture favors x₂₄, the most recent checkpoint, while the other two together retain 33%. **Solved variant:** Weights move from about [0.090,0.245,0.665] to [0.068,0.184,0.748]. Retrieval concentrates more on the third checkpoint while retaining nonzero contributions from the other two. Addition changes everything: x_{l+1} = x_l + F_l(x_l). Each layer ADDS its transformation to the stream instead of replacing it; if F ≡ 0, x passes through unchanged — the identity path guarantees a direct gradient to every layer. Under the assumption of roughly orthogonal contributions (‖F_l‖ = 0.5), the energy follows ‖x_l‖² ≈ 1 + 0.25·l: x₀’s share goes from 100% to 25% (l = 12), 14.3% (l = 24), 7.7% (l = 48). Diluted, never erased — addition is conservative. The minimum check covers dimensions, sign, and order of magnitude. If the observed change contradicts the prediction, locate the first operation whose direction changes instead of fixing only the final line.
### Rubric Exercise 1 — /10 | Criterion | Points | |---|---:| | Explicit data and shapes | 2 | | Traceable calculation | 3 | | Prediction before variation | 2 | | Interpretation and check | 2 | | Named limitation | 1 | ## Exercise 2 — Diagnose a seductive explanation — Depth states A colleague claims: « Depth states proves the system will be accurate, fast, and stable in every context. » 1. Separate mechanism, assumption, observation, and conclusion. 2. Name two correct lesson elements and two unsupported extrapolations. 3. Propose a bounded experiment with controlled variable, metric, and stop threshold. 4. Rewrite the claim as one defensible sentence.
Reasoned solution The solution: photograph along the way. Keep a few chosen x_j — {x₀, x₁₂, x₂₄} — as retrieval points along depth, the informational analogue of training checkpoints. Session 12’s mechanism, rotated 90°: a query from the current layer scores the retained states — scores [2, 1, 0] over {x₂₄, x₁₂, x₀} → softmax [0.665, 0.245, 0.090] — then mixes the values. Retrieval is chosen, no longer endured. AttnRes is presented as a course mechanism and source-reported case study; exact Kimi K3 details require a primary source. The claim mixes a local relation with a global guarantee. A defensible version states only the observed mechanism, test conditions, and measured metric. Stop the test if shapes become invalid, the metric crosses the declared degradation threshold, or another variable changes.
### Rubric Exercise 2 — /10 2 points per element: separation, lesson grounding, extrapolations, protocol, and rewrite. ## Exercise 3 — Architecture decision and transfer — Blocking You must reproduce the worked case “Depth scores [2,1,0] over {x₂₄,x₁₂,x₀} give softmax ≈ [0.67,0.24,0.09]. The mixture favors x₂₄, the most recent checkpoint, while the other two together retain 33%.” under two conditions. Option A uses the full chain through “Blocking.” Option B is a transparent baseline that retains “Residual connection,” calculates the expected output directly, and does not use the compression or adjustment mechanism studied. Build a decision record containing: - the workload and dominant constraint; - each option’s mechanism, without slogans; - one quality, memory, or latency prediction; - one case where your preferred procedure loses; - an A/B protocol, metrics, and rollback threshold; - a bounded verdict: choose, defer, or reject.
Elements of a strong solution Group: one checkpoint per block of 12 layers → 4 states, 128 MiB (÷12). Granularity becomes a slider — 48, 12, 4, or 1 states: from “nearly redundant” to “no choice left”. The bounded reading: 0.665 measures x₂₄’s influence in THIS module, at THIS layer, for THIS token. The residual path carries x₂₄ regardless, and perfectly readable weights can accompany a module that explains nothing — the [0.333…] case proves it. Mixed: established mechanisms + source-reported Kimi K3-style choices. A strong answer does not make the newer mechanism the default winner. It retains a measurable baseline, sets thresholds before testing, and separates component cost from whole-system behavior. The verdict names what remains uncertain and the next evidence that could change it.
### Rubric Exercise 3 — /15 | Criterion | Points | |---|---:| | Framing and baseline | 3 | | Compared causal chains | 4 | | Protocol and metrics | 4 | | Rollback threshold | 2 | | Bounded verdict | 2 | ## Extension Repeat Exercise 3 after reversing the dominant constraint. If you optimized memory, impose a strict quality floor; if you optimized fidelity, halve the memory budget. Identify the first part of the verdict that changes and the evidence required. ## Review before submission Review the packet as if another group had to reproduce it without speaking to you. Are all starting values or assumptions present? Are shapes or roles stated before operations? Does the prediction truly precede the observation? Is the result translated into behavior rather than left as an isolated number? Did you test a boundary value and identify a stop condition? Does the procedure or architecture choice retain a measurable baseline and a rollback threshold set before the test? Finally, highlight one sentence that states what is established, one that remains a hypothesis, and one measurement that could change your verdict. If any element is missing, the work is not reproducible. ## Reference appendix for correction # Chapter 15 — Residual dilution and Attention Residuals ### 15.1 Residual streams **Goal:** understand how deep Transformers preserve and update information. A residual connection adds a layer’s change to the current representation: x_next = x + F(x). **Intuition:** Each editor keeps the current draft and adds a revision instead of rewriting from a blank page. **Step by step** - Start x₀=[1,2]. - Layer change F(x₀)=[0.5,−0.2]. - Residual result x₁=[1.5,1.8]. - A later layer receives both the old path and the new change. **Worked example:** In a very deep network, repeatedly mixing everything into one stream may make some earlier signals difficult to recover. This is called residual dilution in the supplied narrative. **Why it matters:** Residual connections help optimization, but they do not guarantee every useful earlier feature stays equally accessible. **Quick check:** If x=[3,1] and F(x)=[−1,2], what is x+F(x)? Answer: [2,3]. ### 15.2 Attention over depth **Goal:** let a layer choose among earlier representations, not only accept the latest accumulated stream. Attention Residuals, abbreviated AttnRes, use attention-like weighting over depth. **Intuition:** A researcher compares several saved drafts and chooses the most relevant passages for the next revision. **Step by step** - Keep selected earlier layer outputs or block summaries. - Compute relevance scores for the current layer. - Normalize the scores into weights. - Form a weighted mixture and use it in the new computation. **Worked example:** three depth states are h₁=[1,0], h₂=[0,2], h₃=[3,1]. Weights [0.2,0.3,0.5] give 0.2h₁+0.3h₂+0.5h₃ = [1.7,1.1]. **Why it matters:** Attention over depth costs storage and computation. Blockwise AttnRes reduces this cost by keeping or attending to coarser checkpoints rather than every layer output. **Quick check:** What do AttnRes weights sum to after softmax? Answer: 1. ### 15.3 Evidence boundary **Goal:** avoid treating an architectural sketch as a released specification. Exact checkpoint spacing, equations and layer placement must be verified from the authoritative Kimi K3 source. **Intuition:** A building concept drawing explains the idea but is not the final engineering blueprint. **Step by step** - Mechanism-level lesson: learned access to earlier depth states may fight dilution. - Product-level details: exact block sizes and reported gains require primary evidence. **Worked example:** This distinction keeps the course useful even if future releases revise names or numbers. **Why it matters:** Scientific reading means tracking what is derived, what is measured and what is merely reported. **Quick check:** Which source should settle an exact implementation dispute? Answer: official code or an authoritative technical report for the exact model version. ## Sources and evidence boundary - Owner-supplied bilingual course packet, Chapter 15. - He et al., “Deep Residual Learning for Image Recognition”, CVPR (2016). - Elhage et al., “A Mathematical Framework for Transformer Circuits” (residual-stream view), Anthropic (2021). - Course source packet supplied by the owner; named-product details remain source-reported until primary verification. > **Scope:** Mixed: established mechanisms + source-reported Kimi K3-style choices. These references support the session frame; they do not turn a reported product choice into an independently verified result.