DeltaNet: correcting memory

Replace blind additive writing with read, compare, correct.

Applied AI · advanced · Session 14

Mechanism map

TOKEN t :  k=[1,0]   v=[9,9]        inherited state S = [[2,3],
                                                          [5,1]]
      │
      ▼
┌──────────────┐   ① READ      v̂ = Sᵀk = [2,3]
│   read       │───────────────────────────────┐
└──────────────┘                               ▼
                                 ┌───────────────────────┐
                                 │ ② COMPARE             │
                                 │   e = v − v̂ = [7,6]   │
                                 └───────────────────────┘
                                               │
      ┌────────────────────────────────────────┘
      ▼
┌──────────────────────────────────────────────┐
│ ③ CORRECT     S ← S + β k eᵀ     (β = 1)     │
└──────────────────────────────────────────────┘
      │
      ▼
   S = [[9,9],      read q=[1,0] → [9,9]   ✅ replaced
        [5,1]]      row 2 never touched    ✅ preserved

The problem — Limit of addition

Session 13, last write: k=[1,0] reused, read [11,12] — neither [2,3] nor [9,9]. An additive memory only knows how to reinforce: updating an association (“the price moved to 9”) yields a blend of both versions instead of a replacement.

The idea — Limit of addition

The diagnosis is structural: S ← S + k vᵀ never consults S. The write is decided without knowing what memory already holds — repeating information amplifies it, correcting it entangles it.

Why / at what price — Limit of addition

Naming the cause — the blind write — points to the remedy: read before writing. Addition’s remaining virtue is its simplicity: one operation, no state consulted; everything the fix adds will be paid for in per-token compute.

Check: Session 13 ended with S row 1 = [11,12] after rewriting k=[1,0] with v=[9,9]. What exact information was lost there, and why can plain addition never recover it?

Visual support — Limit of addition

blind addition (s.13)              delta rule (s.14)
k=[1,0] rewritten with [9,9]       same token
        │                                  │
        ▼                                  ▼
row 1 : [2,3] + [9,9]              reads v̂ = [2,3], e = [7,6]
      = [11,12]   ❌               writes k eᵀ → [9,9]   ✅

a blend of both versions           a clean replacement

The problem — Read before writing

To correct without blending, you must know what memory would answer BEFORE writing. Otherwise there is no way to tell “new information” (write hard) from “already known” (do nothing).

The idea — Read before writing

Memory predicts first: v̂ = Sᵀk. On S = [[2,3],[5,1]] with k=[1,0], v̂ = [2,3] — exactly what an aligned query would read. The write becomes conditional on the state, no longer only on the input.

v̂=Sᵀk

Why / at what price — Read before writing

That read is what makes correction possible — and it is one more multiplication per token, before anything is even written. The price: v̂ is only reliable if k aligns with what was written; a misplaced key “corrects” a prediction that never existed.

Check: On S = [[2,3],[5,1]], compute v̂ = Sᵀk for k=[0,1] then for k=[0.5,0.5]. In which case does the read match no written value, and what does that say about key choice?

The problem — Local error

The read gives v̂ = [2,3], the target is v = [9,9]. Writing all of v would superpose again — straight back to session 13’s problem. What exactly must be written to get from one to the other?

The idea — Local error

Only the difference: e = v − v̂ = [7,6]. It carries both what is missing (positive components) and what must be erased (negative ones). And if v̂ = v, then e = [0,0]: the redundant token writes nothing.

e=v−v̂

Why / at what price — Local error

The zero error is the first economy mechanism: no useless rewriting, no drift on repetitions. The price: e is local to the current key — it corrects what k reads, not the whole memory; an error seen through the wrong key stays invisible.

Check: A token arrives with v = [9,9] while memory already predicts v̂ = [9,9]. Write e, write k eᵀ, and state precisely what memory does at that step.

The problem — Delta update

e says what to correct; it must still land in the right place at the right strength. A correction spread everywhere would degrade the other rows; one that is too strong overshoots — case D reads [−5,−4].

The idea — Delta update

S ← S + β k eᵀ: k localizes (row 1 only), e carries the content, β sets the dose. With β=1 and ‖k‖=1: exact replacement, [2,3] → [9,9], row 2 untouched. With β=0.5: halfway, [5.5, 6].

S←S+βk(v−Sᵀk)ᵀ

Why / at what price — Delta update

After one step, (1 − β·‖k‖²) of the error remains: exact replacement needs β=1 AND a normalized key. The price: calibration — a mis-set β or ‖k‖ ≠ 1 overshoots or oscillates, like the beginner session’s learning rate… at token rate.

Check: With e = [7,6] and k = [1,0], give S after β=1 then after β=0.5. Then state the general rule: what fraction of the error remains after an update of strength β?

Visual support — Delta update

S ← S + β · k · eᵀ
         │    │    │
         │    │    └── WHAT to correct: e = v − v̂ = [7,6]
         │    └─────── WHERE to write: k = [1,0] → row 1 only
         └──────────── HOW MUCH: β = 1 → all; β = 0.5 → half

left after one step: (1 − β·‖k‖²) × e

The problem — Fast weights

S changes at every token during inference — so what exactly was “learned” in training? If everything moves all the time, the beginner session’s training/conversation distinction seems to collapse.

The idea — Fast weights

Two speeds coexist: S is a FAST weight, rewritten token by token and discarded at sequence end; the matrices producing k, v, β are SLOW parameters, frozen at inference. Training learns how to drive the memory, not its contents.

Why / at what price — Fast weights

This separation reconciles the two regimes: context writes itself into S without touching parameters. The price: debugging changes in kind — odd behavior can come from state (this very sequence) or from parameters (training), and the remedies have nothing in common.

Check: During inference S goes from [[2,3],[5,1]] to [[9,9],[5,1]] between two tokens. Did the matrices producing k, v and β move? Sort each object into “fast” or “slow” and justify.

Visual support — Fast weights

                PARAMETERS (slow)       STATE (fast)
                W_k, W_v, W_β           S
change…         during training         at every token
at inference    frozen                  rewritten constantly
scope           every context           this very sequence
analogy         the learned grammar     the ongoing conversation

The problem — Orientations

You open two implementations: one writes S + βk eᵀ and reads qᵀS; the other writes S + βe kᵀ and reads Sq. The formulas differ — is one of them wrong?

The idea — Orientations

Neither: the whole presentation transposes (rows ↔ columns) without changing the mechanism. What is immutable: read, compare, correct — and the DECLARED shapes: k(d_k), e(d_v), S(d_k×d_v) in one convention, transposed in the other.

Why / at what price — Orientations

Knowing this keeps you from “fixing” correct code. The price: convention freedom is a team trap — mixing both in one file produces silent bugs. Hence the house rule: declare the shapes next to every formula.

Check: A library writes S ← S + βe kᵀ instead of S ← S + βk eᵀ, and reads y = Sq. Is the mechanism different, or the same thing transposed? Which shape must be declared to settle it?

Worked case — full trace

If S reads [0.6,0.2] for target [1,0], error [0.4,−0.2] drives only the missing correction, unlike fully adding the target again.

State inherited from session 13 (two orthogonal writes):
  S = [[2,3],      row 1 ↔ key [1,0]
       [5,1]]      row 2 ↔ key [0,1]

── Case A: rewrite key [1,0] with v=[9,9], β=1 ──────────────────
  v̂ = Sᵀk = 1·[2,3] + 0·[5,1] = [2,3]
  e  = v − v̂ = [9−2, 9−3] = [7,6]
  k eᵀ = [[7,6],[0,0]]
  S  = [[2+7, 3+6],[5,1]] = [[9,9],[5,1]]
  read q=[1,0] → [9,9]    ✅ old value overwritten, not stacked
  read q=[0,1] → [5,1]    ✅ the other association is untouched
  (session 13 raw addition, for contrast: S = [[11,12],[5,1]] → ❌ [11,12])

── Case B: redundant token, v = [9,9] already stored ─────────────
  v̂ = [9,9] → e = [0,0] → k eᵀ = 0 → S unchanged   ✅ no rewrite

── Case C: partial β, β=0.5 from the starting state ──────────────
  S = [[2+0.5·7, 3+0.5·6],[5,1]] = [[5.5, 6],[5,1]]
  read q=[1,0] → [5.5, 6]   ❌ neither [2,3] nor [9,9]: halfway

── Case D: unnormalized key k=[1,1], target v=[1,0], β=1 ─────────
  v̂ = Sᵀk = [2,3]+[5,1] = [7,4];   e = [−6,−4]
  S = [[2−6, 3−4],[5−6, 1−4]] = [[−4,−1],[−1,−3]]
  read q=[1,1] → [−5,−4]   ❌ overshoot: ‖k‖² = 2, not 1
  fix: the β that cancels the error is 1/‖k‖² = 0.5

SHAPE CHECK: k(2×1) eᵀ(1×2) → 2×2 correction = shape of S.
β is a SCALAR; if it came out as a vector (2), βk is no longer an
intensity factor at all — the implementation is lying about shapes.

Additive writing versus the delta rule, on the same token stream

Situation Addition S ← S + k vᵀ Delta S ← S + βk(v−Sᵀk)ᵀ
Key [1,0] rewritten with [9,9] S row 1 = [11,12] (superposition) S row 1 = [9,9] (replacement)
Redundant token (v already stored) writes anyway, doubles the trace e=[0,0]: no write at all
What the write consults nothing: only k and v current state S, via v̂=Sᵀk
Breaking point saturation by accumulation unnormalized key or miscalibrated β

Causal lab

Predict → change one variable → run → explain the delta

/interactives/curriculum/linear-delta-memory.html?lang=en

Common errors

“The delta rule erases the old value and writes the new one.”

It erases nothing: it adds βk eᵀ. The exact replacement in case A only happens because β=1 and ‖k‖²=1. Case D, with k=[1,1], overshoots and reads [−5,−4] — the same formula, without normalization.

“β is a learning rate, so a hyperparameter you tune once.”

β is produced by the network at every token, like k and v: it is fast state, not a setting. It can be 1 on a correcting token and ~0 on the redundant token of case B, inside the same sequence.

Boundary, evidence, and sources

The delta rule reduces some interference; it does not create unlimited capacity, and stability depends on keys, gates, and normalization.

Evidence status: Established mechanisms; numerical simplifications are pedagogical.

  • Owner-supplied bilingual course packet, Chapter 9.
  • Schlag, Irie & Schmidhuber, “Linear Transformers Are Secretly Fast Weight Programmers”, ICML (2021).
  • Yang et al., “Parallelizing Linear Transformers with the Delta Rule over Sequence Length” (DeltaNet), NeurIPS (2024).
  • Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Transfer challenge

Replay case D with the normalized key k = [1/√2, 1/√2] ≈ [0.707, 0.707], target v = [1,0], β = 1, on S = [[2,3],[5,1]].

  1. Predict: will the read q = k after the update land exactly on [1,0]?
  2. Compute v̂, e, then the post-correction read.
  3. Conclude: what does ‖k‖ = 1 guarantee, and what does it NOT guarantee for the other keys?

Synthesis and exit ticket

  • Limit of addition
  • Read before writing
  • Local error
  • Delta update
  • Fast weights
  • Orientations

Ticket: mechanism · trace · observation · boundary · evidence · next experiment

Instructor notes: Frame the problem before naming the mechanism. Collect an initial prediction and retain it for the exit ticket.

Instructor notes: Connect every step to the next with a causal verb. Flag any merely decorative arrow.

Instructor notes: Re-display session 13’s S = [[11,12],[5,1]] and take a vote: “implementation bug or mathematical property?”. “Bug” usually wins — perfect: the beat shows the addition did exactly what it was told.

Instructor notes: Open by re-displaying S = [[11,12],[5,1]] from the previous session and ask “what should have happened?” before introducing any formula. The delta rule must land as an answer to a pain they already felt.

Instructor notes: Answer: attribution is lost — nothing says [11,12] = [2,3] + [9,9] rather than any other decomposition; a sum has infinitely many preimages, so addition is irreversible. Expected wrong answer: “subtract the old value” — you would need to still know it.

Instructor notes: Read both columns right to left: the same token produces two different memories. The only code difference is the prior read — have it underlined.

Instructor notes: Ask: “before correcting someone, what must you know?” — what they currently believe. The teaching analogy carries the beat: a memory is corrected like a learner, starting from its present answer.

Instructor notes: Have them compute v̂ = Sᵀk by hand for two keys, one of them misaligned. Do not comment on the second result: let the group discover that the read matches nothing stored.

Instructor notes: Answer: k=[0,1] reads [5,1], a genuinely stored value; k=[0.5,0.5] reads 0.5·[2,3] + 0.5·[5,1] = [3.5, 2] — no written value. Key choice decides whether “reading” means retrieving or blending. Expected wrong answer: renormalizing the blend to “recover” a value.

Instructor notes: Have e computed on two degenerate cases first: v̂ = v (nothing to do) and v̂ = 0 (write everything). The two extremes make the general case [7,6] immediately readable.

Instructor notes: Force the e=[0,0] case with a show of hands: “does memory write anything?” Half will say yes. That is the moment the difference from addition becomes visceral.

Instructor notes: Answer: e = [0,0], k eᵀ = [[0,0],[0,0]] — memory does NOTHING, and that is the desired behavior on a redundant token. Expected wrong answer: “it reinforces the association”, the reflex inherited from session 13’s addition.

Instructor notes: Announce “one formula, three dials” and have the roles identified before giving the breakdown: what localizes? what carries content? what sets the dose? The three answers (k, e, β) structure the beat.

Instructor notes: Elicit a prediction for S at β=0.5 before computing, then ask which β would cancel the error in one step. Move straight to case D: the answer “1” is wrong the moment ‖k‖≠1.

Instructor notes: Answer: β=1 → row 1 = [9,9]; β=0.5 → [5.5, 6]. Rule: (1 − β·‖k‖²) of the error remains — here ‖k‖² = 1, hence (1 − β). Expected wrong answer: generalizing (1 − β) to any key while forgetting ‖k‖²; case D refutes it two slides later.

Instructor notes: Hide the three captions and have them recovered. Then test the residual formula on case D: β = 1 but ‖k‖² = 2 → residual (1 − 2) = −1: the negative sign IS the overshoot.

Instructor notes: Flash question: “while you talk to an assistant, what is changing inside the machine?”. Collect answers loosely, then sort them into the two columns of the visual support.

Instructor notes: Draw two columns on the board, “changes every token” and “changes every gradient step”, and have them sort S, W_k, W_v, β, k. Fast-weight versus parameter confusion is what blocks sessions 16 and 17.

Instructor notes: Answer: no — W_k, W_v, W_β are frozen at inference. Sorting: S fast; k, v, β recomputed each token from slow parameters; the W_* slow. Expected wrong answer: filing β as a hyperparameter — the deck’s common error 2.

Instructor notes: Sort the objects the room named at the start of the beat into the table, then ask: “where does a chat correction live?”. Right column — a direct callback to beginner session 11.

Instructor notes: Show the two code snippets side by side for thirty seconds and vote: “same mechanism or not?”. Record the tally — it gets compared after the beat.

Instructor notes: Deliberately write the transposed formula on the board and see who objects. Close with a house rule: no formula is discussed unless its shapes are written next to it.

Instructor notes: Answer: the same mechanism transposed — declaring S’s shape, (d_k×d_v) or (d_v×d_k), settles it immediately; then check that write AND read use the same convention. Expected wrong answer: declaring the second code wrong because “the course formula” is the other one.

Instructor notes: Walk line by line. Locate an inconsistency at the first faulty step, not only on the final line.

Instructor notes: Have learners fill the final row before revealing it: that trade-off is what decides in production.

Instructor notes: Retain initial and final values. Do not allow simultaneous changes that make the delta impossible to attribute.

Instructor notes: For each claim, have the group produce the smallest counterexample before giving the correction.

Instructor notes: Separate verifiable mechanism, reported implementation choice, and experimental result. Evidence precision must match claim precision.

Instructor notes: Answers: v̂ = 0.707·[7,4] ≈ [4.95, 2.83], e ≈ [−3.95, −2.83]; since β·‖k‖² = 1, the read q = k lands exactly on [1,0] — case D’s overshoot disappears. BUT the correction writes into BOTH rows (k has no zero): the reads q=[1,0] and q=[0,1] get polluted. ‖k‖ = 1 guarantees exact replacement on THIS key; isolation of the others comes from orthogonality, not from the norm. Ten minutes, pairs.

Instructor notes: Rebuild the chain without looking at the slides, then fill the ticket in at most six lines. Compare with the opening prediction and name what actually changed.