Teacher guide — Kimi Delta Attention: a bounded case study
Duration: 120 minutes
Positioning: Study a reported combination of delta correction, per-channel decay, and chunkwise computation without turning product numbers into established facts.
Expected evidence: Mixed: established mechanisms + source-reported Kimi K3-style choices.
Observable outcomes and preparation
- Explain per-channel decay.
- Assemble the conceptual KDA chain.
- Classify claims by evidence strength.
Before class, the instructor runs the worked case and lab, prints the exercise packet, prepares a four-column board—assumption, prediction, observation, delta—and checks that every mathematical notation is paired with dimensions. The demonstration must not become slide reading.
Diagnostic
- Explain in one sentence: Established mechanism. What observation would falsify your explanation?
- Explain in one sentence: Per-channel decay. What observation would falsify your explanation?
- Explain in one sentence: Delta correction. What observation would falsify your explanation?
Teaching decision: if two of three answers remain nominal or lack a validity condition, rebuild the vocabulary with a numerical example before any formula. A fluent but unfalsifiable answer does not count as mastery.
Timed plan
| Time | Activity | Observable evidence |
|---|---|---|
| 0–10 min | Individual diagnostic, then pair comparison | Three answers and one named uncertainty |
| 10–25 min | Situation and vocabulary | Annotated input → state → output diagram |
| 25–55 min | Develop the mechanism on the board | Shapes, assumptions, and intermediate calculation visible |
| 55–75 min | Worked case with deliberate errors | Reasoned correction, not only the right number |
| 75–95 min | Causal lab: predict, change one variable, run | Prediction / observation / delta table |
| 95–112 min | Exercises 1 and 2 with peer correction | Retained artifact and applied rubric |
| 112–120 min | Exit ticket and transfer | Mechanism, boundary, next experiment |
Teaching notes
Chapter 12 — Kimi Delta Attention
12.1 Per-channel decay
Goal: make forgetting more precise. A channel is one coordinate in a learned representation. One scalar α treats every channel alike; a vector of decay values can treat channels differently.
Intuition: Instead of one dimmer switch for an entire building, give each room its own dimmer.
Step by step
-
Let decay vector a=[0.9,0.2].
-
If a memory row is [10,10], channel-wise decay gives [9,2].
-
The first channel retains information; the second refreshes quickly.
-
The model learns decay controls from data.
Worked example: Different channels may specialize in patterns with different useful lifetimes, such as local syntax versus a longer-running topic signal.
Why it matters: Per-channel decay increases flexibility but adds implementation and optimization complexity.
Quick check: If a channel’s decay is near zero, what happens? Answer: its previous content is mostly forgotten before the new update.
12.2 What we can safely claim
Goal: separate mechanism from uncertain product specifications. Kimi Delta Attention, abbreviated KDA, is described as combining delta-style associative updates with finer learned decay and hardware-aware chunkwise computation.
Intuition: Treat architecture descriptions like a map with confidence labels: roads verified by primary sources are solid; rumors are dotted lines.
Step by step
-
Established foundation: matrix operations, causal prediction, recurrent state, gates and delta correction.
-
Source-dependent claim: the exact way a named Kimi model combines these components.
-
Unverified details such as exact layer counts, expert counts or benchmark numbers must be checked against an official report or released code.
Worked example: KDA’s educational value is clear even without trusting every product number: it shows how compressed memory can become more selective and correctable.
Why it matters: This course will describe the architecture at the level supported by the supplied source and mark uncertain details instead of inventing them.
Quick check: Should a reported benchmark be treated like a mathematical identity? Answer: No. It depends on model version, setup, hardware and measurement method.
Running the worked case
For vector gate [0.9,0.2], the first channel retains almost all its trace while the second forgets quickly. The same delta correction can therefore have different temporal effects by channel.
Do not reveal the result at once. Ask learners to predict the next operation, its shape, and the expected sign. After each line ask: “What changed? What stayed fixed? Which assumption did we use?” A calculation error repaired with a causal chain is worth more than a guessed result.
Lab protocol
- Write a qualitative and, where possible, numerical prediction before touching a control.
- Change one variable only; retain a capture or record initial and final values.
- Explain the delta through the mechanism, not “the tool did that.”
- Test one boundary value and state where the model stops representing a real system.
Misconceptions
| # | Observable misconception | Grounded correction | Probe |
|---|---|---|---|
| 1 | “Established mechanism guarantees the outcome without assumptions or measurement.” | First sort: KDA’s core can be studied without any product. Read-compare-correct (session 14) plus a forgetting gate (session 16) — every number in that part can be recomputed here, on the board. | Ask for a counterexample, then restate the mechanism with its validity condition. |
| 2 | “Per-channel decay guarantees the outcome without assumptions or measurement.” | Replace the scalar with a vector: a = [0.9, 0.2], S′ = diag(a)·S. Channel 1 retains — half-life ln(0.5)/ln(0.9) ≈ 6.6 tokens — while channel 2 flushes (≈ 0.43): a 15× ratio inside the same state. | Ask for a counterexample, then restate the mechanism with its validity condition. |
| 3 | “Delta correction guarantees the outcome without assumptions or measurement.” | The trace quantifies it: a β=1 correction on channel 2 → exactly [4,4]; two steps of a 0.2 gate later: 4×0.2² = 0.160. The correction succeeded AND the value vanished — “corrected” and “retained” are independent properties. | Ask for a counterexample, then restate the mechanism with its validity condition. |
Boundary to maintain: Exact Kimi K3 specifications are not claimed as verified in this lesson; the vendor technical report (Kimi Linear, arXiv:2510.26692, to verify) is cited without independent replication.
Probing questions
- If we remove or reverse Established mechanism, which output changes first, and what observation would show it?
- If we remove or reverse Per-channel decay, which output changes first, and what observation would show it?
- If we remove or reverse Delta correction, which output changes first, and what observation would show it?
- If we remove or reverse Chunkwise execution, which output changes first, and what observation would show it?
- If we remove or reverse Source-reported, which output changes first, and what observation would show it?
- If we remove or reverse Evidence habit, which output changes first, and what observation would show it?
Assessment
| Level | Criterion |
|---|---|
| 0 | Repeats terms without connecting input, transformation, and output. |
| 1 | Describes the chain but checks neither shape nor assumption. |
| 2 | Executes the case, explains the result, and names one limitation. |
| 3 | Transfers to a new case, compares an alternative, and proposes a measurement that could invalidate the choice. |
Exit threshold: level 2 on the worked case and at least one exercise; a memorized formula without interpretation remains level 1.
Observation and remediation protocol
During discussion, the instructor records evidence rather than impressions. Evidence of understanding contains a named object, a justified transformation, and a checkable consequence. If a learner gives the right result without a chain, ask for the preceding line. If the chain is coherent but the result is wrong, preserve the chain and isolate the arithmetic error. If vocabulary from another concept is used, compare both mechanisms in an input, state, output, cost, and boundary table. Remediation targets the first break only: vocabulary, shapes, operation, interpretation, or claim scope. After correction, use a neighboring case with a changed value; success on the same example does not prove transfer. For pair work, assign operator and verifier roles, then swap. The verifier does not supply the answer: they request an assumption, check the shape, and ask what observation could contradict the reasoning. The instructor retains the exit ticket and classifies the dominant break. The next session opens with a three-minute problem aimed at that break instead of repeating the whole lesson.
Differentiation
- Support: provide shapes and the first transformation; let the learner complete interpretation and boundary.
- Core path: worked case without result, lab with one assigned variable, diagnostic exercise.
- Extension: change one assumption, compare two mechanisms, and define the metric that would decide between them.
Post-session follow-up
Within twenty-four hours, return each annotated exit ticket with one priority, the exact resource to reopen, and a mini-case different from the worked case. Revision requires three items: a written prediction, a retained trace, and one sentence explaining the delta. At the next session, sample two submissions: one that repaired the break and one that remains ambiguous. Discuss them anonymously, then state the criterion that separates them. Do not use completion rate as evidence of mastery. Evidence of remediation is a correct chain on a new case with a stated boundary. If the same break appears in more than one third of the group, repair the support or demonstration before blaming learners.
Follow-up closes only when the new artifact shows the causal chain, check, and boundary—not merely when a file has been submitted.
Sources and evidence boundary
- Owner-supplied bilingual course packet, Chapter 12.
- Kimi Linear / Kimi Delta Attention technical report, arXiv:2510.26692 (à vérifier / to verify).
- Yang, Kautz & Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule”, ICLR (2025).
- Course source packet supplied by the owner; named-product details remain source-reported until primary verification.
Scope: Mixed: established mechanisms + source-reported Kimi K3-style choices. These references support the session frame; they do not turn a reported product choice into an independently verified result.
Exit ticket
In no more than six lines: mechanism; calculation or trace; observation; boundary; evidence level; next experiment. The instructor marks one priority causal break for revision.