KV cache, recurrent memory, MLA, and low rank

Compare three memory budgets and distinguish per-token compression, fixed state, and low-rank factorization.

Applied AI · advanced · Session 18

Mechanism map

TOKEN t: h_t (d_model = 4096)
      │
      ▼
┌──────────────────┐
│  W_DKV : d → c   │  latent compression (c = 512)
└────────┬─────────┘
         ▼
      c_t  (512 values)   ◀── THE ONLY THING CACHED
         │
   ┌─────┴─────┐
   ▼           ▼
┌───────┐  ┌───────┐
│ W_UK  │  │ W_UV  │  reconstruct c → heads
└───┬───┘  └───┬───┘
    ▼          ▼
   K_t        V_t  ──▶ exact attention over t tokens
───────────────────────────────────────────────────────
CACHE   = t × layers × c × 2 B     ← STILL LINEAR IN t
STATE S = layers × d_k × d_v × 2 B ← constant, never sees t

The problem — Standard KV cache

Your service targets 131,072 tokens of context. Before any optimization, you need the raw number: at 32 layers, 8 KV heads, d_head=128 and BF16, each token costs 128 KiB of cache — 16 GiB per request at full length. And that bill returns with every request.

The idea — Standard KV cache

The cache keeps every past token’s keys and values, per layer: bytes ≈ tokens × layers × 2 × heads × d_head × bytes. Reads are faithful — exact attention over the whole past — and the arithmetic can be redone factor by factor, no calculator.

bytes≈tokens×layers×2×heads×d_head×bytes/value

Why / at what price — Standard KV cache

Fidelity is total: exact recall of any token. The price is structural: strictly linear growth — ×32 tokens = ×32 memory — with re-read bandwidth to match. No setting changes the slope, only the coefficient.

Check: With layers=32, KV heads=8, d_head=128 and BF16, the cache is 128 KiB per token. Recompute that 131,072 factor by factor, then say which factor changes under INT8.

The problem — Fixed recurrent state

16 GiB per request rules out most deployments. Sessions 13 to 16 built the radical alternative: what if the past fit in a fixed-size matrix, whatever the token count?

The idea — Fixed recurrent state

The state S — 32 layers × 128 × 128 × BF16 = 1 MiB — summarizes the whole past: 4,096 or 524,288 tokens, still 1 MiB. Growth disappears; this is where the previous sessions’ memories were heading.

Why / at what price — Fixed recurrent state

A constant, negligible budget — 16,000× less than the cache at 131,072 tokens. The price, familiar from sessions 13-16: compression and interference — recall is no longer exact and degrades with length, even though the memory itself never moves.

Check: The 1 MiB fixed state does not move between 4,096 and 524,288 tokens. Which quantity does degrade over that range, and when would you notice it in an output?

The problem — MLA

Between an exact 16 GiB and an approximate 1 MiB, the gap is brutal. Is there a middle ground: keep one entry PER token — structured fidelity — but pay less per entry?

The idea — MLA

Multi-head Latent Attention compresses each token into a latent vector c_t (512 values) — the only thing cached — then reconstructs K and V through W_UK and W_UV at read time. Width ÷ 4 ⇒ 32 KiB/token, 4 GiB at 131,072 tokens.

Why / at what price — MLA

The cache keeps its per-token structure and the bill is divided by 4. The price: a reconstruction on every read — memory traded for compute, mostly at decode — and growth stays O(n): the gain is a coefficient, not a change of asymptote.

Check: MLA caches c_t (512 values) and reconstructs K_t and V_t via W_UK and W_UV. Which cost moves from memory to compute, and at which step — prefill or decode?

The problem — Low-rank factorization

MLA’s compression rests on a pure algebra question: when does replacing a large matrix W by a product of two small ones actually save anything? Ill-chosen, the bottleneck r saves nothing at all.

The idea — Low-rank factorization

W (d×m) ≈ A(d×r)·B(r×m) costs r(d+m) parameters instead of d·m. For 4096×4096: r=512 divides by 4; r=2048 gives 16,777,216 — exactly the original cost. The break-even threshold is r = d·m/(d+m) = 2048 here.

W≈AB, A∈R^{d×r}, B∈R^{r×m}

Why / at what price — Low-rank factorization

Below the threshold the saving is real and the compute faster. The price: the projection’s capacity is capped at rank r — any transformation requiring a higher rank is structurally out of reach, no matter the training.

Check: For W ∈ R^{4096×4096}, r=512 divides parameters by 4 but r=2048 saves nothing. Derive the threshold r = d·m/(d+m) and explain why it equals exactly 2048 here.

Visual support — Low-rank factorization

W: 4096 × 4096 = 16,777,216 parameters

r =  512 : 4096·512 + 512·4096   =  4,194,304   ✅ ÷ 4
r = 1024 : 4096·1024 + 1024·4096 =  8,388,608   ✅ ÷ 2
r = 2048 : 4096·2048 + 2048·4096 = 16,777,216   ❌ = all of W

threshold: r* = d·m/(d+m) = 4096·4096/8192 = 2048
past the break-even bottleneck, the “compression” costs more

The problem — MLA is not fixed memory

Two announcements sound alike: “4× compressed cache” and “constant memory”. A team budgets 4 GiB “forever” with MLA — and discovers 16 GiB at 524,288 tokens. Where did the promise go?

The idea — MLA is not fixed memory

There never was one: MLA compresses each entry, it does not merge tokens. One entry per token ⇒ linear growth with a smaller coefficient. Only a recurrent state merges the past into a fixed-size object.

Why / at what price — MLA is not fixed memory

Separating the two avoids the production capacity mistake: MLA bounds the coefficient, the state bounds the growth. The price of confusing them is written in the trace: MLA’s ×4 gain is eaten by ×4 tokens — length still rules.

Check: Going from 131,072 to 524,288 tokens brings MLA back to 16 GiB, the original standard-cache budget. What kind of gain does per-token compression provide — constant or asymptotic?

Visual support — MLA is not fixed memory

                 4,096 t     131,072 t     524,288 t
standard cache   512 MiB     16 GiB        64 GiB      line, 128 KiB/t
MLA (c ÷ 4)      128 MiB      4 GiB        16 GiB      line, 32 KiB/t
fixed state        1 MiB      1 MiB         1 MiB      flat
                              ▲
     MLA at 524,288 t = standard cache at 131,072 t:
     a coefficient can be caught up, a slope cannot

The problem — Low rank is not LoRA

The same formula W ≈ AB in two contexts: MLA’s architectural factorization and LoRA adaptation. A hurried reader concludes “MLA is built-in LoRA” — and proposes “removing the adapter” from a model that has none.

The idea — Low rank is not LoRA

Opposite roles: in MLA, A and B ARE the normal path, trained from scratch, immovable. LoRA adds a low-rank delta BESIDE frozen weights, to adapt after the fact — mergeable or removable at will.

Why / at what price — Low rank is not LoRA

The distinction prevents absurd decisions — freezing MLA’s “adapter”, or believing LoRA is required at inference. The price: permanent vigilance; the same algebra serves architectures and adaptation procedures, and only context decides the meaning.

Check: W_DKV and W_UK sit in the model’s forward path and are trained with it; a LoRA adapter adds A and B beside frozen weights. Of the two, which can be removed afterwards without breaking the model?

Visual support — Low rank is not LoRA

same algebra W ≈ A·B      MLA (architecture)     LoRA (adaptation)

trained…                  from scratch           after the fact
beside…                   nothing: it IS W       frozen weights
removable?                never                  merge or detach
goal                      narrower cache         cheap specialization

Worked case — full trace

With 32 layers, 8 KV heads, d_head=128, BF16, and 4,096 tokens, simplified raw cache is 4,096×32×2×8×128×2 bytes = 536,870,912 bytes = 512 MiB (0.5 GiB). Quartering latent width reduces the per-token term, not linear growth.

CONFIG: layers=32, KV heads=8, d_head=128, BF16 (2 B/value)

bytes ≈ tokens × layers × 2 × heads × d_head × 2

per token = 32 × 2 × 8 × 128 × 2 = 131,072 B = 128 KiB/token

t = 4,096    → 4,096 × 131,072 = 536,870,912 B = 512 MiB  ✅
t = 131,072  → 131,072 × 131,072 = 17,179,869,184 B = 16 GiB ✅
               (×32 tokens ⇒ ×32 memory: strictly linear)

MLA, latent width ÷ 4 → 32 KiB/token
t = 131,072  → 4 GiB        ✅ 4× less… but still linear
t = 524,288  → 16 GiB       ❌ the constant gain is eaten by ×4 tokens

FIXED STATE (d_k=d_v=128): 32 × 128 × 128 × 2 = 1 MiB
t = 4,096 → 1 MiB ; t = 524,288 → 1 MiB   ← independent of t

LOW RANK: W (4096×4096) = 16,777,216 params.
  r = 512  → 2 × 4096 × 512  =  4,194,304   ✅ ÷4
  r = 2048 → 2 × 4096 × 2048 = 16,777,216   ❌ zero gain (threshold r = d·m/(d+m))

UNIT CHECK: 512 MiB ≠ 512 MB. 1 GiB = 1,024 MiB = 2³⁰ B.

Three memory budgets at 131,072 tokens (32 layers, BF16)

Mechanism Memory at 131,072 tokens What you pay
Standard KV cache 16 GiB, O(n) growth nothing in fidelity: exact recall
MLA (c ÷ 4) 4 GiB, O(n) growth K/V reconstruction on every read
Fixed recurrent state 1 MiB, constant compression and inter-token interference
Low rank r=512 on W does not affect the cache projection capacity capped at r

Causal lab

Predict → change one variable → run → explain the delta

/interactives/curriculum/cache-mla-comparison.html?lang=en

Common errors

“MLA turns the cache into fixed-size memory.”

MLA compresses each cache entry; it does not merge tokens. The cache stays one entry per token: 4 GiB at 131,072 tokens, 16 GiB at 524,288. Only a recurrent state (1 MiB here) is genuinely length-independent.

“The model’s low-rank factorization is just built-in LoRA.”

Same algebra (W ≈ AB), opposite roles. Here A and B are the model’s normal path, trained from scratch and never removable. LoRA adds a trainable low-rank delta beside frozen weights, to adapt after the fact and be merged or switched off at will.

Boundary, evidence, and sources

Simplified formulas omit alignment, quantization, buffers, and sharing details. They compare trends, not promise real footprint.

Evidence status: Mixed: established mechanisms + source-reported Kimi K3-style choices.

  • Owner-supplied bilingual course packet, Chapter 13.
  • DeepSeek-AI, “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model” (introduces Multi-head Latent Attention), arXiv:2405.04434 (2024).
  • Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, ICLR (2022).
  • Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Transfer challenge

Pick ONE assumption from the trace — d_head, latent width c, or served length — and change it by a factor of 2.

  1. Pre-register the expected effect on per-token cost AND on memory at 131,072 tokens.
  2. Recompute both figures and compare with your prediction.
  3. Which of the three changes would also affect quality, and why not the other two directly?

Synthesis and exit ticket

  • Standard KV cache
  • Fixed recurrent state
  • MLA
  • Low-rank factorization
  • MLA is not fixed memory
  • Low rank is not LoRA

Ticket: mechanism · trace · observation · boundary · evidence · next experiment

Instructor notes: Frame the problem before naming the mechanism. Collect an initial prediction and retain it for the exit ticket.

Instructor notes: Connect every step to the next with a causal verb. Flag any merely decorative arrow.

Instructor notes: Open with the bill: “your PM wants 131,072 tokens of context — how much memory per request?”. Collect three estimates before any computation; the gap between the answers and 16 GiB structures the whole session.

Instructor notes: Have the computation written on the board factor by factor, no calculator, until it lands on 128 KiB/token. Then have them double d_head aloud: the reflex “which factor moves” is the point of this slide.

Instructor notes: Answer: 32 × 2 × 8 × 128 × 2 = 131,072 B — layers × (K and V) × heads × d_head × bytes/value. Under INT8 the last factor goes from 2 to 1: 64 KiB/token. Expected wrong answer: “INT8 removes a factor” — no, it halves one; no factor disappears.

Instructor notes: Flash recall of sessions 13-16 in one question: “how big is S after a million tokens?”. “The same” must shoot back from the room — otherwise redo sixty seconds of session 13 before moving on.

Instructor notes: Ask before commenting: “at 524,288 tokens, how big is the state?” The answer “still 1 MiB” must come from the room, not the instructor — that is what makes the trade-off credible afterwards.

Instructor notes: Answer: recall quality degrades (accumulated interference), not memory; you notice when an old read comes back blended — hence a controlled recall test, not a RAM monitor. Expected wrong answer: “nothing degrades since the size is constant”.

Instructor notes: Hook question: “can you pay less per token without giving up per-token?”. Let the room articulate the trade-off space — structured fidelity versus coefficient — before saying “MLA”.

Instructor notes: Have someone trace the diagram arrow with a marker: what is stored, what is recomputed? Do not move on until the room has pointed at c_t alone.

Instructor notes: Answer: you move memory (a 4× narrower cache) into compute (reconstruction through W_UK/W_UV on every read); the overhead lands mostly at decode, where each new token re-reads the whole cache. Expected wrong answer: “at prefill” — prefill is already compute-bound.

Instructor notes: Write 16,777,216 on the board and ask: “does splitting W into two matrices always save?”. Take a yes/no vote; the vote sets up the beat’s discovery of the r = 2048 threshold.

Instructor notes: Hand out r = 2048 without comment and let the group discover the zero saving. A computed failure beats a stated rule; only then derive the threshold.

Instructor notes: Answer: factored cost r(d+m) < d·m ⇔ r < d·m/(d+m); with d = m = 4096, r* = 4096/2 = 2048, and at r = 2048 the equality is exact: zero gain. Expected wrong answer: assuming the threshold is d/2 “by luck” — redo it with d = 4096, m = 1024 (r* ≈ 819).

Instructor notes: Have the room compute the r = 1024 row before revealing it, then ask for r* at d = 4096, m = 1024 (≈ 819): the threshold depends on BOTH dimensions, not on a “half” rule.

Instructor notes: Reread the two announcements side by side: “4× compressed cache” and “constant memory”. Ask who budgets what at 524,288 tokens — the harvested budgeting errors are the beat’s content.

Instructor notes: Provoke the error on purpose: take a show of hands on “is MLA fixed-memory?” before revealing 4 GiB → 16 GiB. The vote makes the correction stick.

Instructor notes: Answer: a constant gain — a ÷4 factor on growth that stays linear; the O(n) order is unchanged. Expected wrong answer: “asymptotic, since it is 4× less at every length” — which is precisely the definition of a constant gain. Point to the visual: two straight lines, different coefficients.

Instructor notes: Read the table diagonally: the MLA cell at 524,288 equals the standard cell at 131,072. Then ask: “which row changes if we go to c ÷ 8?” — only the two lines move, never the flat one.

Instructor notes: Show W ≈ AB alone, with no context, and ask: “architecture or adaptation?”. The impossibility of answering without context IS the beat’s lesson — state it explicitly at the end.

Instructor notes: Write W ≈ AB once on the board and have two columns annotated: “trained from scratch / grafted onto frozen”, “immovable / mergeable”. Same algebra, two columns: that is the whole message.

Instructor notes: Answer: the LoRA adapter is removable — a delta beside frozen weights; W_DKV and W_UK are the model’s forward path, removing them breaks it. Expected wrong answer: confusing “LoRA can be merged” (a choice) with “LoRA is required” (false), and the reverse for MLA.

Instructor notes: Hide the column headers and have each cell attributed row by row. Hesitations on “removable?” are the real comprehension diagnostic.

Instructor notes: Walk line by line. Locate an inconsistency at the first faulty step, not only on the final line.

Instructor notes: Have learners fill the final row before revealing it: that trade-off is what decides in production.

Instructor notes: Retain initial and final values. Do not allow simultaneous changes that make the delta impossible to attribute.

Instructor notes: For each claim, have the group produce the smallest counterexample before giving the correction.

Instructor notes: Separate verifiable mechanism, reported implementation choice, and experimental result. Evidence precision must match claim precision.

Instructor notes: Answers: d_head ×2 → 256 KiB/token and 32 GiB (quality possibly affected: wider heads); c ÷2 → 16 KiB/token and 2 GiB (quality affected: a tighter reconstruction); length ÷2 → per-token cost UNCHANGED, memory 8 GiB (quality affected only if useful content exceeds the window). Error to harvest: “halving the length halves the per-token cost”. Check that every pair pre-registered before computing. Ten minutes.

Instructor notes: Rebuild the chain without looking at the slides, then fill the ticket in at most six lines. Compare with the opening prediction and name what actually changed.