Kimi Delta Attention: a bounded case study

Study a reported combination of delta correction, per-channel decay, and chunkwise computation without turning product numbers into established facts.

Applied AI · advanced · Session 17

Mechanism map

MECHANISM (verifiable)                │  EVIDENCE STATUS
                                      │
S_{t−1} = [[2,3],                     │
           [5,1]]                     │
      │                               │
      ▼                               │
┌──────────────────────────┐          │  ✅ algebra the reader
│ ① PER-CHANNEL DECAY      │          │     can check
│   a = [0.9, 0.2]         │          │
│   S′ = diag(a)·S_{t−1}   │          │
└──────────────────────────┘          │
      │  S′ = [[1.80, 2.70],          │
      ▼        [1.00, 0.20]]          │
┌──────────────────────────┐          │  ✅ same rule as session 14
│ ② READ, COMPARE, CORRECT │          │
│  S = S′ + β k (v − S′ᵀk)ᵀ│          │
└──────────────────────────┘          │
      │                               │
      ▼                               │
┌──────────────────────────┐          │  ❌ source-reported
│ ③ CHUNKWISE EXECUTION    │          │     (layer counts, ratios,
│   “K3-style” assembly    │          │      performance figures)
└──────────────────────────┘          │
                                      │  ⚠ a more precise claim
channel-1 half-life ≈ 6.6 tokens      │    demands more precise
channel-2 half-life ≈ 0.43 token      │    evidence

The problem — Established mechanism

A product announcement mixes everything: equations, assembly choices, benchmarks. If you cannot separate what is provable with a pencil from what is declared, you are teaching — or buying — a press release.

The idea — Established mechanism

First sort: KDA’s core can be studied without any product. Read-compare-correct (session 14) plus a forgetting gate (session 16) — every number in that part can be recomputed here, on the board.

Why / at what price — Established mechanism

That verifiable core travels everywhere: it stays true whatever happens to the product. The price: it says nothing about the real scale choices — how many layers, which ratios — and that is exactly where “reported” begins.

Check: Take the delta rule from session 14 and the gate from session 16 without ever saying “Kimi”. What remains teachable, and why does that part depend on no product source?

The problem — Per-channel decay

A scalar α imposes one half-life on the whole state: holding the customer’s name for long AND flushing the weather fast is impossible with a single dial.

The idea — Per-channel decay

Replace the scalar with a vector: a = [0.9, 0.2], S′ = diag(a)·S. Channel 1 retains — half-life ln(0.5)/ln(0.9) ≈ 6.6 tokens — while channel 2 flushes (≈ 0.43): a 15× ratio inside the same state.

Why / at what price — Per-channel decay

The control is real and verifiable. So is the price: d_k gates per token instead of one — 128 values to produce and bound at d_k = 128 — and one more debugging axis: which channel holds what is no longer readable at a glance.

Check: With a = [0.9, 0.2], compute each channel’s half-life, then the per-token cost of a vector gate versus a scalar α, for d_k = 128. Is the extra control free?

Visual support — Per-channel decay

trace of 1, never rewritten        t :   0      1      2      3
channel 1 (a₁ = 0.9)                     1.000  0.900  0.810  0.729
channel 2 (a₂ = 0.2)                     1.000  0.200  0.040  0.008
                                                ▲
half-lives: 6.6 tokens versus 0.43 —     by t=1, channel 2 has
a 15× ratio inside the SAME state        already lost nearly all

The problem — Delta correction

Correcting and retaining are easily conflated: a team believes a “successful” write is a durable write. What becomes of an exact correction placed on a fast-forgetting channel?

The idea — Delta correction

The trace quantifies it: a β=1 correction on channel 2 → exactly [4,4]; two steps of a 0.2 gate later: 4×0.2² = 0.160. The correction succeeded AND the value vanished — “corrected” and “retained” are independent properties.

Why / at what price — Delta correction

The delta + forgetting composition avoids both raw accumulation (session 13) and total inertia. The price: a write’s fate depends on the channel it lands in — same β and values, opposite outcomes (0.160 versus 3.240 on channel 1).

Check: On a channel whose gate is a₂ = 0.2: a β=1 correction (k=[0,1], v=[4,4]) brings it to exactly [4,4]; two steps later, with no rewrite, only 4×0.2² = 0.160 remains. Did the correction fail? Separate “correct” from “retain” using those two numbers.

The problem — Chunkwise execution

The announcement touts “chunkwise parallel prefill”. Should that count as a model-quality argument? Session 15 has the answer — you just have to apply it.

The idea — Chunkwise execution

Chunkwise execution carries state block to block and parallelizes the interior — shown EXACT in session 15 (o₅ = [3,5] with or without chunks). It is an execution strategy: it changes cost, never the numbers.

Why / at what price — Chunkwise execution

Real speed, unchanged quality — so “chunkwise” in an announcement is a throughput argument, not a quality one. The price of the confusion: comparisons crediting the mechanism with gains that came from elsewhere.

Check: Chunkwise computation was shown exact in session 15 (o₅ = [3,5] with or without chunks). What does that exactness prove — and not prove — about the quality of a model using it?

The problem — Source-reported

The supplied document attributes layer counts, KDA/attention ratios, and performance figures to a “K3-style” architecture. You have neither the protocol nor a replication. What status do these facts get in class?

The idea — Source-reported

The exact status: “reported”. The primary source is a vendor technical report (Kimi Linear, arXiv:2510.26692 — to verify): citable, informative, not independently replicated here. The information travels with its label, neither suppressed nor laundered.

Why / at what price — Source-reported

Labeling lets you teach what is real without certifying it. The price: discipline — every sentence in the course carries its status, and a vendor report proves the claim exists, not that the result does.

Check: The supplied document states a KDA/full-attention layer ratio for a K3-style architecture. Write the sentence you would use in a meeting: it must convey the information without granting it verified-fact status.

Visual support — Source-reported

EVIDENCE LADDER — strongest to weakest

✅ redone on the board     “diag(a)·S decays row i by a_i”
✅ counted directly        “one gate/channel = d_k values per token”
⚠ cited report             “K3-style layer ratio”
   (not replicated)          (arXiv:2510.26692, to verify)
❌ not evaluated here      “+ X% long-context quality”

rule: repeating a claim = inheriting its burden of proof

The problem — Evidence habit

In meetings the three registers fuse into one sentence: “KDA decays per channel, K3 stacks it 3:1, and it beats the baseline by X%”. Repeated as is, the sentence launders its last two thirds.

The idea — Evidence habit

The habit: systematically separate (a) the verifiable equation, (b) the reported implementation choice, (c) the experimental result — and demand evidence proportional to each claim’s precision.

Why / at what price — Evidence habit

The sort costs three seconds per sentence and prevents months of architecture built on an unreplicated benchmark. The social price: saying “reported, not replicated” in a meeting takes more courage than a slide of citations.

Check: Sort these three statements: (a) diag(a)·S decays row i by a_i; (b) product P uses this mechanism; (c) it beats the baseline by X %. For each, what evidence would you require before repeating it?

Visual support — Evidence habit

sentence heard in a meeting:
“KDA decays per channel, K3 stacks it 3:1, it beats the baseline by X%”
                         │ sort │
                         ▼
(a) established  : per-channel decay — recomputable here
(b) reported     : the 3:1 ratio — vendor report, to verify
(c) not evaluated: the X% — no protocol, no replication

defensible sentence:
“the mechanism is verifiable; the ratio and the gain remain reported”

Worked case — full trace

For vector gate [0.9,0.2], the first channel retains almost all its trace while the second forgets quickly. The same delta correction can therefore have different temporal effects by channel.

S₀ = [[2,3],[5,1]]   row i = key channel i
VECTOR gate a = [0.9, 0.2]  →  S ← diag(a)·S  (row i × a_i)

── A. Per-channel decay, three steps with no write ────────────────
  t+1 : [[1.800, 2.700],[1.000, 0.200]]
  t+2 : [[1.620, 2.430],[0.200, 0.040]]
  t+3 : [[1.458, 2.187],[0.040, 0.008]]
  channel-1 half-life: ln(0.5)/ln(0.9) = 6.58 tokens
  channel-2 half-life: ln(0.5)/ln(0.2) = 0.43 token
  → ≈ 15× ratio between two channels of the SAME state   ✅ the whole point

── B. Same delta correction, different temporal effect ────────────
  at t+2, write on channel 2: k=[0,1], v=[4,4], β=1
    v̂ = S′ᵀk = [0.200, 0.040]   ;  e = [3.800, 3.960]
    S  = [[1.620, 2.430],[4.000, 4.000]]   ✅ exact correction
  two steps later, with no further write:
    channel 2 → 4 × 0.2² = 0.160   ❌ almost all of it is already gone
    (the same write on channel 1 would be 4 × 0.9² = 3.240)
  “corrected” and “retained” are therefore two distinct properties.

── C. Shape check ─────────────────────────────────────────────────
  a has d_k = 2 components, one per ROW of S. A gate of length d_v
  would put diag(a) on the right and switch off value dimensions,
  not addresses: different mechanism, identical symbols.
  Cost: d_k gates per token instead of 1 scalar.

── D. LABELLING THE CLAIMS (the exercise of this session) ─────────
  “S ← diag(a)S + βk(v−S′ᵀk)ᵀ”             ✅ VERIFIABLE — redo A and B
  “a vector gate costs d_k values/token, not 1”  ✅ VERIFIABLE
    — direct counting
  “the K3-style architecture stacks N KDA layers per full-attention
    layer”                                 ❌ REPORTED — vendor technical
    report (arXiv:2510.26692), not independently replicated here
  “KDA improves long-context quality”      ❌ EXPERIMENTAL RESULT
    — needs protocol, test sets, ablations, confidence intervals

RULE: what is computed here is taught as established. What comes from
the supplied document is taught as “reported”, label attached.

Sorting every claim in this session by evidence strength

Claim Status What it would take to validate it
S ← diag(a)·S then delta correction Established (algebra) redo the arithmetic: 2 → 1.80 → 1.62
A per-channel gate costs d_k values per token Established (counting) count the outputs of the gating network
K3-style mixes KDA and full-attention layers at a given ratio Source-reported independent replication of the cited report (arXiv:2510.26692)
KDA improves long-context quality Not evaluated here protocol, test sets, ablations, variance

Causal lab

Predict → change one variable → run → explain the delta

/interactives/curriculum/chunk-gate-memory.html?lang=en

Common errors

“The course describes the Kimi K3 architecture, so it is documented.”

The course reports what a supplied document attributes to a K3-style architecture. The primary publication is a vendor technical report (arXiv:2510.26692), cited but not replicated here. The 6.58 and 0.43 half-lives are checkable with a pencil; layer counts and ratios are not. “Source-reported” is not hedging, it is the actual status.

“A per-channel gate is strictly better than a scalar α, otherwise nobody would ship it.”

Two problems. The cost is real: d_k gates per token versus 1, plus bounding constraints on every channel. And “it ships in a product” is not comparative evidence — you would need a compute-matched ablation, which is absent here. The mechanism is teachable; its superiority is not.

Boundary, evidence, and sources

Exact Kimi K3 specifications are not claimed as verified in this lesson; the vendor technical report (Kimi Linear, arXiv:2510.26692, to verify) is cited without independent replication.

Evidence status: Mixed: established mechanisms + source-reported Kimi K3-style choices.

  • Owner-supplied bilingual course packet, Chapter 12.
  • Kimi Linear / Kimi Delta Attention technical report, arXiv:2510.26692 (à vérifier / to verify).
  • Yang, Kautz & Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule”, ICLR (2025).
  • Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Transfer challenge

The supplied document describes a K3-style architecture: a given KDA/attention layer ratio, announced gains.

  1. Write three sentences about this architecture — one per status: established, reported, not evaluated.
  2. For the “reported” sentence, list the two checks that would move it up one rung.
  3. Decide: on what basis could your team prototype KDA right now, without waiting for those checks?

Synthesis and exit ticket

  • Established mechanism
  • Per-channel decay
  • Delta correction
  • Chunkwise execution
  • Source-reported
  • Evidence habit

Ticket: mechanism · trace · observation · boundary · evidence · next experiment

Instructor notes: Frame the problem before naming the mechanism. Collect an initial prediction and retain it for the exit ticket.

Instructor notes: Connect every step to the next with a causal verb. Flag any merely decorative arrow.

Instructor notes: Write the full announcement on the board — equation, ratio, benchmark — and ask: “what can we verify from here, offline?”. The room’s spontaneous sort gives the session its outline; keep it posted.

Instructor notes: Open by writing the mechanism on the board WITHOUT the product name, and introduce the name only after the arithmetic. If the name comes first, everything after inherits its prestige — exactly the bias this session fights.

Instructor notes: Answer: the whole core remains — read-compare-correct, gates, half-lives: recomputable algebra, independent of any product. Expected wrong answer: believing that without the name only “theory” remains — every number in the trace can be redone with a pencil.

Instructor notes: Hook question: “with ONE forgetting dial, how do you keep the customer’s name and flush the weather?”. The felt impossibility creates the need for the vector — the exact limit of session 16’s scalar α.

Instructor notes: Have them compute both half-lives (6.58 and 0.43), then ask what a 128-gate vector would look like. The jump from the 2-channel toy to real scale must be made by them, not announced.

Instructor notes: Answer: half-lives ln(0.5)/ln(0.9) ≈ 6.58 tokens and ln(0.5)/ln(0.2) ≈ 0.43; cost: 128 gate values per token versus 1. No, not free: production, bounding within [0,1], and per-channel debugging. Expected wrong answer: forgetting the bounding constraint (stability, session 16).

Instructor notes: Have the room fill in the t=4 column (0.656 and 0.0016), then ask which channel they would pick for a first name, which for a sentence’s tone. Choosing by use case makes the vector concrete.

Instructor notes: Take bets: “a perfect correction on a fast channel — what is it worth two tokens later?”. Collect numeric bets before computing 4×0.2² — the gap between the bets and 0.160 is the beat.

Instructor notes: Have them recite the read-compare-correct chain from memory before adding forgetting. If session 14 is shaky, the KDA stack becomes a magic word instead of a composition.

Instructor notes: Answer: the correction did NOT fail — it landed exactly on [4,4]; retention is what is short (0.160 after two steps of a 0.2 gate). Correcting = the state reached at t; retaining = what survives at t+n. Expected wrong answer: “β should have been larger” — β changes nothing here.

Instructor notes: Ask who remembers session 15’s verdict (chunking is exact). If nobody, redo it in sixty seconds with o₅ = [3,5]: this beat only makes sense leaning on that result.

Instructor notes: Explicitly recall the session-15 result: chunking is exact, therefore it adds no quality. Have the group articulate why “chunkwise computation” in an announcement is not a performance argument.

Instructor notes: Answer: exactness proves chunkwise breaks nothing — same numbers — hence also improves nothing: any quality gain comes from elsewhere. Expected wrong answer: counting “parallel prefill” as a quality argument — it is a cost argument.

Instructor notes: Hand out three printed sentences — equation, ratio, benchmark — and have them pinned on a wall-mounted evidence ladder. The placement disagreements ARE the beat’s content; do not adjudicate too fast.

Instructor notes: Run it aloud: each person phrases a sentence about K3-style, the group votes “established / reported / not evaluated”. Correct the phrasing, not the person. This is language training, not a knowledge test.

Instructor notes: Model answer: “the vendor technical report states a ratio of N KDA layers per attention layer; that is reported, not independently replicated — I treat it as an architecture hypothesis”. Have the group vote on two or three phrasings; reject those that drop the attribution.

Instructor notes: Have two claims from outside the course placed on the ladder (one from the last press release the room read). The ladder is only learned once it works on fresh material.

Instructor notes: Role-play the meeting: one learner reads the fused sentence, another politely interrupts and sorts it into three statuses. Two minutes, then swap roles — the target skill is verbal, not conceptual.

Instructor notes: Close by applying the labelling to the course itself: ask which claim on YOUR slides is weakest in evidence. An instructor who welcomes that question teaches evidence discipline better than ten slides about it.

Instructor notes: Answer: (a) established — redoing the arithmetic suffices; (b) reported — require the cited technical report and its exact scope; (c) not evaluated — require protocol, test sets, ablations, and variance before repeating it. Expected wrong answer: demanding the same evidence level for all three.

Instructor notes: Have everyone rewrite the defensible sentence in their own words, then read three aloud. The target skill is a sentence speakable in a meeting, not a taxonomy.

Instructor notes: Walk line by line. Locate an inconsistency at the first faulty step, not only on the final line.

Instructor notes: Have learners assign the column-2 statuses claim by claim before revealing the column. The expected disagreement is on row 3: that is exactly where “reported” separates from “established”.

Instructor notes: Retain initial and final values. Do not allow simultaneous changes that make the delta impossible to attribute.

Instructor notes: For each claim, have the group produce the smallest counterexample before giving the correction.

Instructor notes: Separate verifiable mechanism, reported implementation choice, and experimental result. Evidence precision must match claim precision.

Instructor notes: Expected: (1) e.g. “diag(a)·S decays each channel” / “the report states a 3:1 ratio” / “the long-context gain is not evaluated here”; (2) independent replication + published protocol (test sets, budget, ablations); (3) the established mechanism is enough to prototype — the key point: you can build on (a) without certifying (b) or (c). Misconception to harvest: “until it is verified, nothing can be done”. Eight minutes, solo then debrief.

Instructor notes: Rebuild the chain without looking at the slides, then fill the ticket in at most six lines. Compare with the opening prediction and name what actually changed.