Linear attention and fixed-size matrix memory

Move from a growing token notebook to a fixed state matrix, then measure reads and interference.

Applied AI · advanced · Session 13

Mechanism map

WRITE (token t)                       READ (query)
 k=[1,0]  v=[2,3]                        q=[1,0]
     │                                      │
     ▼                                      │
┌──────────────┐                            │
│    outer     │                            │
│ product k vᵀ │                            │
└──────┬───────┘                            │
       │ S ← S + k vᵀ                       ▼
       ▼                            ┌──────────────┐
┌──────────────────────┐    qᵀS     │     read     │
│  S  = [[2,3],        │ ─────────▶ │  y = qᵀS     │
│        [0,0]]        │            │  y = [2,3]   │
│  FIXED SIZE d_k×d_v  │            └──────────────┘
└──────────────────────┘
  every write and every read goes through S;
  S does not grow with the sequence

The problem — Matrix foundations

You are about to manipulate memories made of vectors and matrices. A shape error — multiplying a length-3 vector by a 2×2 matrix — does not produce a wrong result: it produces nonsense. And with NumPy broadcasting, some wrong shapes still execute and silently corrupt everything downstream.

The idea — Matrix foundations

A vector is an ordered list of d numbers; a matrix organizes rows and columns (d_k × d_v). Dimensions dictate which multiplications are valid: here k(2) and v(2) build S(2×2), and a length-3 key [1;0;2] is rejected before any computation.

Why / at what price — Matrix foundations

The shape check costs one line and runs before the computation: it turns silent corruption into an immediate refusal. The price of skipping it: sessions 14 to 18 stack these objects — an early wrong shape becomes untraceable there.

Check: S is 2×2 and you are handed k=[1,0,2]. Which multiplication fails exactly, which dimension must you fix first, and why must this check precede the calculation rather than follow it?

The problem — Dot product

A memory needs addresses: which write should a query pair with? Eyeballing vectors gives no usable number, and without a quantified alignment measure every read would have to re-read everything.

The idea — Dot product

The dot product compresses alignment into one number: q·k = Σqᵢkᵢ. Here q=[1,0] against k=[1,0] gives 1 (aligned); against k=[0,1] it gives 0 (orthogonal). That is the addressing mechanism: strong = relevant, zero = ignored.

q·k=Σqᵢkᵢ

Why / at what price — Dot product

Why it works: the orthogonality zero isolates addresses — without it, every query would read every write. The price: alignment is continuous; two nearby keys (q·k = 0.9) partially share an address, and that sharing becomes the final beat’s interference.

Check: For k=[0,1] and q=[1,0] the dot product q·k is 0. What will the query read at that address, why is this zero sought rather than suffered, and what would happen if every key were identical?

Visual support — Dot product

        k=[0,1]
          ▲
          │
          │
          └──────────▶ k=[1,0]   (also the direction of q)

q=[1,0]·k=[1,0] = 1   → address selected
q=[1,0]·k=[0,1] = 0   → address ignored
q=[1,1]·k=[1,0] = 1   → partial selection: q touches BOTH rows

The problem — Outer product

The dot product compares, but writes nothing. How do you file the value v=[2,3] “at the address” k=[1,0] inside a fixed-size structure — so that the right query finds it later?

The idea — Outer product

The outer product k vᵀ builds a matrix: the row is selected by k, the content is carried by v. For k=[1,0] and v=[2,3]: k vᵀ = [[2,3],[0,0]] — the value is filed on row 1, row 2 stays blank. Writes accumulate: S ← S + k vᵀ.

S←S+k vᵀ

Why / at what price — Outer product

An associative write in one O(1) operation, local to k’s direction: this is what makes fixed state possible. The price: the addition is blind — it never checks whether the address is already occupied. Writing twice on k=[1,0] adds the values instead of replacing them.

Check: Write k vᵀ for k=[0,1] and v=[5,1], then give the full S. Which row of S changes, which stays untouched, and which property of k explains that localized write?

Visual support — Outer product

            v = [ 2   3 ]

k = [1]     ┌─────────┐
    [0]     │  2   3  │  ← row lit by k₁ = 1
            │  0   0  │  ← row switched off by k₂ = 0
            └─────────┘

k vᵀ: every cell (i,j) = kᵢ × vⱼ

The problem — Reading state

The memory now holds several writes superposed in a single matrix. How does a query recover THE value that concerns it without scanning a list — since there is no list at all any more?

The idea — Reading state

Reading is a multiplication: y = qᵀS. Query q=[1,0] selects row 1 and reads [2,3] exactly; q=[0,1] reads [5,1]. Nothing is scanned: one operation, however long the past.

y=qᵀS

Why / at what price — Reading state

An O(1) read versus O(n) for the KV cache: the gain is structural. The price: y is always a combination of everything written, weighted by q·k. It is exact only when keys are orthogonal — the read cannot tell “stored value” from “blend”.

Check: After both orthogonal writes, predict y for q=[1,1] BEFORE computing, then check. Does the result match any actually stored value, and what does that teach you about what “reading” means here?

The problem — Fixed memory

A KV cache at 100,000 tokens blows up a service’s memory bill — and it grows again at the next token. Can you serve a long context with a memory whose size does not depend on length?

The idea — Fixed memory

S measures d_k × d_v, full stop: 1,000 or 100,000 tokens written, the matrix keeps the same size. The cache’s O(n) growth becomes a constant — at the price of a superposed summary instead of an exact trace.

Why / at what price — Fixed memory

The memory budget becomes predictable — exactly what gets billed in production. The price: information capacity is constant too; beyond roughly d_k distinct key directions, writes superpose. “Fixed size” also means “fixed capacity”, never infinite context.

Check: The sequence grows from 1,000 to 100,000 tokens. What happens to the size of S, what happens to a KV cache, and which of the two shows up on a production service’s memory bill?

Visual support — Fixed memory

one layer, one head, d = 128, BF16

              KV cache (2·d·2 B/token)      state S (d×d×2 B)
  1,000 t     500 KiB   ██                  32 KiB  ▏
100,000 t     ≈ 49 MiB  ████████████        32 KiB  ▏  (unchanged)

the cache pays for every token; S pays once

The problem — Interference

Third write: k=[1,0] reused with v=[9,9]. The read q=[1,0] returns [11,12] — neither the old value nor the new one. Who overwrote what? Nobody: both writes coexist, merged. What do you do with a memory that can no longer discriminate?

The idea — Interference

Interference is arithmetic, not random: row 1 of S holds [2,3] + [9,9] = [11,12], and the read returns exactly that sum. Nearby keys write into shared directions; their values blend in proportion to alignment.

Why / at what price — Interference

Seeing it as a sum makes it predictable and measurable — a controlled recall test is enough to detect it. The price remains: without correction or forgetting, an additive memory drifts with length. That is exactly the problem session 14’s delta rule attacks.

Check: The third write reuses k=[1,0] and the read returns [11,12] instead of [2,3] or [9,9]. Name precisely the quantity that was lost, then propose a measurement that would detect this interference in production without inspecting S.

Worked case — full trace

With zero S, k=[1,0] and v=[2,3], the write gives [[2,3],[0,0]]. Query q=[1,0] reads [2,3]. Trying to add a length-3 key [1,0,2] also reveals why shape checks are mandatory.

S = [[0,0],      initial state (2×2), empty
     [0,0]]

write 1: k=[1,0]  v=[2,3]
  k vᵀ = [[1*2, 1*3],   = [[2,3],
          [0*2, 0*3]]      [0,0]]
  S    = [[2,3],[0,0]]

write 2: k=[0,1]  v=[5,1]         ORTHOGONAL key
  S    = [[2,3],[5,1]]

read q=[1,0]:  y = qᵀS = [2,3]    ✅ value 1 recovered exactly
read q=[0,1]:  y = qᵀS = [5,1]    ✅ value 2 recovered exactly

write 3: k=[1,0]  v=[9,9]         REUSED key → collision
  S    = [[11,12],[5,1]]
read q=[1,0]:  y = [11,12]        ❌ neither [2,3] nor [9,9]: superposition

SHAPE CHECK: k(2) vᵀ(2) → S(2×2). A key [1,0,2] (3) is rejected.

Exact KV cache versus fixed matrix state

Criterion KV cache State S (linear attention)
Memory for n tokens grows as O(n) constant, d_k × d_v
Recall of one token exact approximate, interference-prone
Cost per new token O(n) (re-reads all) O(1) (single update)
Breaking point memory saturates on long context similar keys → blended values

Causal lab

Predict → change one variable → run → explain the delta

/interactives/curriculum/linear-delta-memory.html?lang=en

Common errors

“Fixed-size memory = infinite context.”

S has a constant size, therefore constant information capacity: past that point writes superpose instead of accumulating cleanly. The trace shows it at write three, where [2,3] and [9,9] merge into [11,12]. A long context is not a preserved context: S always accepts a write, even when it can no longer discriminate.

“Reading qᵀS returns the value that was written.”

Only when keys are orthogonal — which is the case for the first two writes, and precisely why they read back exactly. With similar or reused keys the read returns a weighted blend: that is the definition of interference, not an implementation bug. The useful question is therefore never “is the value there?” but “are the keys separated enough?”

Boundary, evidence, and sources

Fixed memory means neither perfect memory nor infinite context: capacity and interference remain bounded.

Evidence status: Established mechanisms; numerical simplifications are pedagogical.

  • Owner-supplied bilingual course packet, Chapter 8.
  • Katharopoulos et al., “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention”, ICML (2020).
  • Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Transfer challenge

Rerun the trace in dimension 3: k₁=[1;0;0], v₁=[2,3], k₂=[0;1;0], v₂=[5,1], then a third write k₃=[0.7;0.7;0], v₃=[4,0].

  1. Give the new shape of S and justify it.
  2. Predict which read (q=k₁ or q=k₂) degrades, then compute both.
  3. State the general rule: when is an extra write “free”?

Synthesis and exit ticket

  • Matrix foundations
  • Dot product
  • Outer product
  • Reading state
  • Fixed memory
  • Interference

Ticket: mechanism · trace · observation · boundary · evidence · next experiment

Instructor notes: Frame the problem before naming the mechanism. Collect an initial prediction and retain it for the exit ticket.

Instructor notes: Connect every step to the next with a causal verb. Flag any merely decorative arrow.

Instructor notes: Open with the silent-broadcasting anecdote: a (3,) against a (2,2) refuses, but some wrong shapes “work” and do nonsense. Ask who has lost an hour to this — the raised hands install the reflex better than a rule.

Instructor notes: Have dimensions written on the board before any operation. Explicitly reject one incompatible multiplication in front of the group: that reflex carries sessions 14 to 18, where shapes get harder to track mentally. Ask “what shape goes in, what shape comes out?” every time.

Instructor notes: Answer: k vᵀ (and qᵀS) fails — k has 3 components against S’s 2 rows; you fix d_k first, the projection that produces k. The check precedes the computation because a wrong shape can execute via broadcasting and corrupt downstream without an error. Expected wrong answer: “truncate k to two components”.

Instructor notes: Hook question: “what does a memory without addresses look like?”. Target answer: a pile where everything blends. Keep that image on the board — the final interference beat will reuse it verbatim.

Instructor notes: Ask for the sign of the dot product before computing. Two orthogonal vectors give zero, and that zero is what makes addressing possible: without it every query would read every write. Have a learner state that consequence rather than announcing it.

Instructor notes: Answer: the write filed on k=[0,1] is invisible to q=[1,0] — zero contribution. That zero is sought: it isolates addresses. If every key were identical, every read would return the sum of all values — a one-cell memory. Expected wrong answer: “q·k = 0 is a bug”.

Instructor notes: Have q=[1,1] placed on the drawing: it sits at 45° from both axes, so it reads both rows equally. The geometry announces the next check’s [7,4] result before any computation.

Instructor notes: Have them draw an empty 2×2 matrix and ask: “where do you file [2,3] so q=[1,0] finds it?”. The spontaneous proposals (cell (1,1)? the diagonal?) make the need for a rule felt — k vᵀ IS that rule.

Instructor notes: Many confuse dot and outer products. Have learners name each output shape — a scalar versus a matrix — before moving on, and write both side by side. Left untreated here, this confusion reliably resurfaces in session 14 on the delta rule.

Instructor notes: Answer: k vᵀ = [[0,0],[5,1]]; row 2 changes (k₂ = 1), row 1 stays untouched (k₁ = 0) — the localization comes from k’s zeros. Expected wrong answer: transposing and writing [5,1] as a column; have the 2×2 shape verified on the board.

Instructor notes: Have the four cells filled one by one with the kᵢ × vⱼ rule before showing the result. The row of zeros must come from them: it is what makes the write “addressed”.

Instructor notes: Ask first: “how many operations to read from a 100,000-entry list?”, then “and from a matrix?”. The O(n)/O(1) contrast must precede the formula, not illustrate it afterwards.

Instructor notes: Hide the result and take a show-of-hands prediction for y. The classic error is expecting the last written value instead of the sum the query reconstructs. Collecting one wrong prediction and repairing it publicly beats walking through the correct calculation.

Instructor notes: Answer: y = [7,4] = [2,3] + [5,1] — neither stored value. “Reading” means combining in proportion to q·k, not retrieving. Expected wrong answer: predicting [5,1], the last write — exactly the error the show-of-hands prediction should surface.

Instructor notes: Have the room estimate KV-cache memory at 100,000 tokens before showing the visual support. Estimates typically spread across three orders of magnitude — precisely the symptom this session treats.

Instructor notes: Connect to product cost: have them compute KV cache memory at 100,000 tokens, then compare with S, unchanged. The numeric contrast carries the whole session and sets up session 18. Do not supply the result: have them produce it.

Instructor notes: Answer: S stays at d_k × d_v; the KV cache is multiplied by 100 (O(n)); the cache is what shows up on the bill — see the visual support: 500 KiB → ≈ 49 MiB per head per layer, while S stays at 32 KiB. Expected wrong answer: “S grows a little too”.

Instructor notes: Have both multiplications redone on a calculator: 2×128×2×100,000 versus 128×128×2. Then announce that session 18 will redo this at real-model scale (32 layers, 8 heads) — same formulas, more factors.

Instructor notes: Replay the [11,12] read and ask: “who had the bug?”. Target answer: nobody — the addition did exactly its job. The discomfort of that answer is the beat’s entry point; do not defuse it too early.

Instructor notes: Have learners construct an interference case themselves by choosing their own similar keys. A collision they built is retained far better than one they were shown, and makes “keys separated enough” concrete ahead of the session 16 gates.

Instructor notes: Answer: the lost quantity is the distinction between the two writes — the sum is exact, the attribution is gone. Production measurement: a controlled recall test (insert known key→value pairs, measure exact-recall rate versus length and key similarity). Expected wrong answer: proposing to inspect S, unreadable in practice.

Instructor notes: Walk line by line. Locate an inconsistency at the first faulty step, not only on the final line.

Instructor notes: Have learners fill the final row before revealing it: that trade-off is what decides in production.

Instructor notes: Retain initial and final values. Do not allow simultaneous changes that make the delta impossible to attribute.

Instructor notes: For each claim, have the group produce the smallest counterexample before giving the correction.

Instructor notes: Separate verifiable mechanism, reported implementation choice, and experimental result. Evidence precision must match claim precision.

Instructor notes: Answers: S becomes 3×2 (d_k = 3, d_v = 2). BOTH reads degrade — q=k₁ reads [4.8, 3] instead of [2,3] and q=k₂ reads [7.8, 1] instead of [5,1]: k₃ overlaps both addresses and deposits 0.7 × [4,0] = [2.8, 0] on each. Rule: an extra write is free if and only if its key is orthogonal to every key you still intend to read. Expected misconception: “only the nearest read is affected”. Ten minutes, pairs.

Instructor notes: Rebuild the chain without looking at the slides, then fill the ticket in at most six lines. Compare with the opening prediction and name what actually changed.