Exercises with solutions — Kimi Delta Attention: a bounded case study

General instruction: every answer must show data, transformation, result, one check, and one limitation. A bare number or copied definition is insufficient.

Starting data: For vector gate [0.9,0.2], the first channel retains almost all its trace while the second forgets quickly. The same delta correction can therefore have different temporal effects by channel.

Boundary to retain: Exact Kimi K3 specifications are not claimed as verified in this lesson; the vendor technical report (Kimi Linear, arXiv:2510.26692, to verify) is cited without independent replication.

Exercise 1 — Calculated trace — Established mechanism

Reproduce and annotate the chain input → state → output. Increase the second gate channel from [0.9,0.2] to [0.9,0.24]. Compare the trace retained in that channel after one step with all other values fixed.

Deliverable: a data → operation → result → interpretation table, plus two sentences about the changed value.

Worked solution

For vector gate [0.9,0.2], the first channel retains almost all its trace while the second forgets quickly. The same delta correction can therefore have different temporal effects by channel.

Solved variant: For an initial trace of 1, the second channel retains 0.24 instead of 0.20, which is 20% more. The first channel stays at 0.9, so the vector gate changes memory channel by channel.

First sort: KDA’s core can be studied without any product. Read-compare-correct (session 14) plus a forgetting gate (session 16) — every number in that part can be recomputed here, on the board. Replace the scalar with a vector: a = [0.9, 0.2], S′ = diag(a)·S. Channel 1 retains — half-life ln(0.5)/ln(0.9) ≈ 6.6 tokens — while channel 2 flushes (≈ 0.43): a 15× ratio inside the same state. The minimum check covers dimensions, sign, and order of magnitude. If the observed change contradicts the prediction, locate the first operation whose direction changes instead of fixing only the final line.

Rubric Exercise 1 — /10

Criterion	Points
Explicit data and shapes	2
Traceable calculation	3
Prediction before variation	2
Interpretation and check	2
Named limitation	1

Exercise 2 — Diagnose a seductive explanation — Delta correction

A colleague claims: « Delta correction proves the system will be accurate, fast, and stable in every context. »

Separate mechanism, assumption, observation, and conclusion.
Name two correct lesson elements and two unsupported extrapolations.
Propose a bounded experiment with controlled variable, metric, and stop threshold.
Rewrite the claim as one defensible sentence.

Reasoned solution

The trace quantifies it: a β=1 correction on channel 2 → exactly [4,4]; two steps of a 0.2 gate later: 4×0.2² = 0.160. The correction succeeded AND the value vanished — “corrected” and “retained” are independent properties. Chunkwise execution carries state block to block and parallelizes the interior — shown EXACT in session 15 (o₅ = [3,5] with or without chunks). It is an execution strategy: it changes cost, never the numbers. Exact Kimi K3 specifications are not claimed as verified in this lesson; the vendor technical report (Kimi Linear, arXiv:2510.26692, to verify) is cited without independent replication.

The claim mixes a local relation with a global guarantee. A defensible version states only the observed mechanism, test conditions, and measured metric. Stop the test if shapes become invalid, the metric crosses the declared degradation threshold, or another variable changes.

Rubric Exercise 2 — /10

2 points per element: separation, lesson grounding, extrapolations, protocol, and rewrite.

Exercise 3 — Architecture decision and transfer — Source-reported

You must reproduce the worked case “For vector gate [0.9,0.2], the first channel retains almost all its trace while the second forgets quickly. The same delta correction can therefore have different temporal effects by channel.” under two conditions. Option A uses the full chain through “Source-reported.” Option B is a transparent baseline that retains “Established mechanism,” calculates the expected output directly, and does not use the compression or adjustment mechanism studied. Build a decision record containing:

the workload and dominant constraint;
each option’s mechanism, without slogans;
one quality, memory, or latency prediction;
one case where your preferred procedure loses;
an A/B protocol, metrics, and rollback threshold;
a bounded verdict: choose, defer, or reject.

Elements of a strong solution

The exact status: “reported”. The primary source is a vendor technical report (Kimi Linear, arXiv:2510.26692 — to verify): citable, informative, not independently replicated here. The information travels with its label, neither suppressed nor laundered. The habit: systematically separate (a) the verifiable equation, (b) the reported implementation choice, © the experimental result — and demand evidence proportional to each claim’s precision. Mixed: established mechanisms + source-reported Kimi K3-style choices.

A strong answer does not make the newer mechanism the default winner. It retains a measurable baseline, sets thresholds before testing, and separates component cost from whole-system behavior. The verdict names what remains uncertain and the next evidence that could change it.

Rubric Exercise 3 — /15

Criterion	Points
Framing and baseline	3
Compared causal chains	4
Protocol and metrics	4
Rollback threshold	2
Bounded verdict	2

Extension

Repeat Exercise 3 after reversing the dominant constraint. If you optimized memory, impose a strict quality floor; if you optimized fidelity, halve the memory budget. Identify the first part of the verdict that changes and the evidence required.

Review before submission

Review the packet as if another group had to reproduce it without speaking to you. Are all starting values or assumptions present? Are shapes or roles stated before operations? Does the prediction truly precede the observation? Is the result translated into behavior rather than left as an isolated number? Did you test a boundary value and identify a stop condition? Does the procedure or architecture choice retain a measurable baseline and a rollback threshold set before the test? Finally, highlight one sentence that states what is established, one that remains a hypothesis, and one measurement that could change your verdict. If any element is missing, the work is not reproducible.

Reference appendix for correction

Chapter 12 — Kimi Delta Attention

12.1 Per-channel decay

Goal: make forgetting more precise. A channel is one coordinate in a learned representation. One scalar α treats every channel alike; a vector of decay values can treat channels differently.

Intuition: Instead of one dimmer switch for an entire building, give each room its own dimmer.

Step by step

Let decay vector a=[0.9,0.2].
If a memory row is [10,10], channel-wise decay gives [9,2].
The first channel retains information; the second refreshes quickly.
The model learns decay controls from data.

Worked example: Different channels may specialize in patterns with different useful lifetimes, such as local syntax versus a longer-running topic signal.

Why it matters: Per-channel decay increases flexibility but adds implementation and optimization complexity.

Quick check: If a channel’s decay is near zero, what happens? Answer: its previous content is mostly forgotten before the new update.

12.2 What we can safely claim

Goal: separate mechanism from uncertain product specifications. Kimi Delta Attention, abbreviated KDA, is described as combining delta-style associative updates with finer learned decay and hardware-aware chunkwise computation.

Intuition: Treat architecture descriptions like a map with confidence labels: roads verified by primary sources are solid; rumors are dotted lines.

Step by step

Established foundation: matrix operations, causal prediction, recurrent state, gates and delta correction.
Source-dependent claim: the exact way a named Kimi model combines these components.
Unverified details such as exact layer counts, expert counts or benchmark numbers must be checked against an official report or released code.

Worked example: KDA’s educational value is clear even without trusting every product number: it shows how compressed memory can become more selective and correctable.

Why it matters: This course will describe the architecture at the level supported by the supplied source and mark uncertain details instead of inventing them.

Quick check: Should a reported benchmark be treated like a mathematical identity? Answer: No. It depends on model version, setup, hardware and measurement method.

Sources and evidence boundary

Owner-supplied bilingual course packet, Chapter 12.
Kimi Linear / Kimi Delta Attention technical report, arXiv:2510.26692 (à vérifier / to verify).
Yang, Kautz & Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule”, ICLR (2025).
Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Scope: Mixed: established mechanisms + source-reported Kimi K3-style choices. These references support the session frame; they do not turn a reported product choice into an independently verified result.