1. Established mechanism
The problem: A product announcement mixes everything: equations, assembly choices, benchmarks. If you cannot separate what is provable with a pencil from what is declared, you are teaching — or buying — a press release.
The idea: First sort: KDA’s core can be studied without any product. Read-compare-correct (session 14) plus a forgetting gate (session 16) — every number in that part can be recomputed here, on the board.
Why / at what price: That verifiable core travels everywhere: it stays true whatever happens to the product. The price: it says nothing about the real scale choices — how many layers, which ratios — and that is exactly where “reported” begins.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
2. Per-channel decay
The problem: A scalar α imposes one half-life on the whole state: holding the customer’s name for long AND flushing the weather fast is impossible with a single dial.
The idea: Replace the scalar with a vector: a = [0.9, 0.2], S′ = diag(a)·S. Channel 1 retains — half-life ln(0.5)/ln(0.9) ≈ 6.6 tokens — while channel 2 flushes (≈ 0.43): a 15× ratio inside the same state.
Why / at what price: The control is real and verifiable. So is the price: d_k gates per token instead of one — 128 values to produce and bound at d_k = 128 — and one more debugging axis: which channel holds what is no longer readable at a glance.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
3. Delta correction
The problem: Correcting and retaining are easily conflated: a team believes a “successful” write is a durable write. What becomes of an exact correction placed on a fast-forgetting channel?
The idea: The trace quantifies it: a β=1 correction on channel 2 → exactly [4,4]; two steps of a 0.2 gate later: 4×0.2² = 0.160. The correction succeeded AND the value vanished — “corrected” and “retained” are independent properties.
Why / at what price: The delta + forgetting composition avoids both raw accumulation (session 13) and total inertia. The price: a write’s fate depends on the channel it lands in — same β and values, opposite outcomes (0.160 versus 3.240 on channel 1).
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
4. Chunkwise execution
The problem: The announcement touts “chunkwise parallel prefill”. Should that count as a model-quality argument? Session 15 has the answer — you just have to apply it.
The idea: Chunkwise execution carries state block to block and parallelizes the interior — shown EXACT in session 15 (o₅ = [3,5] with or without chunks). It is an execution strategy: it changes cost, never the numbers.
Why / at what price: Real speed, unchanged quality — so “chunkwise” in an announcement is a throughput argument, not a quality one. The price of the confusion: comparisons crediting the mechanism with gains that came from elsewhere.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
5. Source-reported
The problem: The supplied document attributes layer counts, KDA/attention ratios, and performance figures to a “K3-style” architecture. You have neither the protocol nor a replication. What status do these facts get in class?
The idea: The exact status: “reported”. The primary source is a vendor technical report (Kimi Linear, arXiv:2510.26692 — to verify): citable, informative, not independently replicated here. The information travels with its label, neither suppressed nor laundered.
Why / at what price: Labeling lets you teach what is real without certifying it. The price: discipline — every sentence in the course carries its status, and a vendor report proves the claim exists, not that the result does.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
6. Evidence habit
The problem: In meetings the three registers fuse into one sentence: “KDA decays per channel, K3 stacks it 3:1, and it beats the baseline by X%”. Repeated as is, the sentence launders its last two thirds.
The idea: The habit: systematically separate (a) the verifiable equation, (b) the reported implementation choice, (c) the experimental result — and demand evidence proportional to each claim’s precision.
Why / at what price: The sort costs three seconds per sentence and prevents months of architecture built on an unreplicated benchmark. The social price: saying “reported, not replicated” in a meeting takes more courage than a slide of citations.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Development from the course source
Chapter 12 — Kimi Delta Attention
12.1 Per-channel decay
Goal: make forgetting more precise. A channel is one coordinate in a learned representation. One scalar α treats every channel alike; a vector of decay values can treat channels differently.
Intuition: Instead of one dimmer switch for an entire building, give each room its own dimmer.
Step by step
-
Let decay vector a=[0.9,0.2].
-
If a memory row is [10,10], channel-wise decay gives [9,2].
-
The first channel retains information; the second refreshes quickly.
-
The model learns decay controls from data.
Worked example: Different channels may specialize in patterns with different useful lifetimes, such as local syntax versus a longer-running topic signal.
Why it matters: Per-channel decay increases flexibility but adds implementation and optimization complexity.
Quick check: If a channel’s decay is near zero, what happens? Answer: its previous content is mostly forgotten before the new update.
12.2 What we can safely claim
Goal: separate mechanism from uncertain product specifications. Kimi Delta Attention, abbreviated KDA, is described as combining delta-style associative updates with finer learned decay and hardware-aware chunkwise computation.
Intuition: Treat architecture descriptions like a map with confidence labels: roads verified by primary sources are solid; rumors are dotted lines.
Step by step
-
Established foundation: matrix operations, causal prediction, recurrent state, gates and delta correction.
-
Source-dependent claim: the exact way a named Kimi model combines these components.
-
Unverified details such as exact layer counts, expert counts or benchmark numbers must be checked against an official report or released code.
Worked example: KDA’s educational value is clear even without trusting every product number: it shows how compressed memory can become more selective and correctable.
Why it matters: This course will describe the architecture at the level supported by the supplied source and mark uncertain details instead of inventing them.
Quick check: Should a reported benchmark be treated like a mathematical identity? Answer: No. It depends on model version, setup, hardware and measurement method.
Complete worked case
For vector gate [0.9,0.2], the first channel retains almost all its trace while the second forgets quickly. The same delta correction can therefore have different temporal effects by channel.
Reading method: write the data, state every object shape, perform one transformation, and interpret the result before continuing.