1. Standard KV cache

The problem: Your service targets 131,072 tokens of context. Before any optimization, you need the raw number: at 32 layers, 8 KV heads, d_head=128 and BF16, each token costs 128 KiB of cache — 16 GiB per request at full length. And that bill returns with every request.

The idea: The cache keeps every past token’s keys and values, per layer: bytes ≈ tokens × layers × 2 × heads × d_head × bytes. Reads are faithful — exact attention over the whole past — and the arithmetic can be redone factor by factor, no calculator.

bytes≈tokens×layers×2×heads×d_head×bytes/value

Why / at what price: Fidelity is total: exact recall of any token. The price is structural: strictly linear growth — ×32 tokens = ×32 memory — with re-read bandwidth to match. No setting changes the slope, only the coefficient.

2. Fixed recurrent state

The problem: 16 GiB per request rules out most deployments. Sessions 13 to 16 built the radical alternative: what if the past fit in a fixed-size matrix, whatever the token count?

The idea: The state S — 32 layers × 128 × 128 × BF16 = 1 MiB — summarizes the whole past: 4,096 or 524,288 tokens, still 1 MiB. Growth disappears; this is where the previous sessions’ memories were heading.

Why / at what price: A constant, negligible budget — 16,000× less than the cache at 131,072 tokens. The price, familiar from sessions 13-16: compression and interference — recall is no longer exact and degrades with length, even though the memory itself never moves.

3. MLA

The problem: Between an exact 16 GiB and an approximate 1 MiB, the gap is brutal. Is there a middle ground: keep one entry PER token — structured fidelity — but pay less per entry?

The idea: Multi-head Latent Attention compresses each token into a latent vector c_t (512 values) — the only thing cached — then reconstructs K and V through W_UK and W_UV at read time. Width ÷ 4 ⇒ 32 KiB/token, 4 GiB at 131,072 tokens.

Why / at what price: The cache keeps its per-token structure and the bill is divided by 4. The price: a reconstruction on every read — memory traded for compute, mostly at decode — and growth stays O(n): the gain is a coefficient, not a change of asymptote.

4. Low-rank factorization

The problem: MLA’s compression rests on a pure algebra question: when does replacing a large matrix W by a product of two small ones actually save anything? Ill-chosen, the bottleneck r saves nothing at all.

The idea: W (d×m) ≈ A(d×r)·B(r×m) costs r(d+m) parameters instead of d·m. For 4096×4096: r=512 divides by 4; r=2048 gives 16,777,216 — exactly the original cost. The break-even threshold is r = d·m/(d+m) = 2048 here.

W≈AB, A∈R^{d×r}, B∈R^{r×m}

Why / at what price: Below the threshold the saving is real and the compute faster. The price: the projection’s capacity is capped at rank r — any transformation requiring a higher rank is structurally out of reach, no matter the training.

5. MLA is not fixed memory

The problem: Two announcements sound alike: “4× compressed cache” and “constant memory”. A team budgets 4 GiB “forever” with MLA — and discovers 16 GiB at 524,288 tokens. Where did the promise go?

The idea: There never was one: MLA compresses each entry, it does not merge tokens. One entry per token ⇒ linear growth with a smaller coefficient. Only a recurrent state merges the past into a fixed-size object.

Why / at what price: Separating the two avoids the production capacity mistake: MLA bounds the coefficient, the state bounds the growth. The price of confusing them is written in the trace: MLA’s ×4 gain is eaten by ×4 tokens — length still rules.

6. Low rank is not LoRA

The problem: The same formula W ≈ AB in two contexts: MLA’s architectural factorization and LoRA adaptation. A hurried reader concludes “MLA is built-in LoRA” — and proposes “removing the adapter” from a model that has none.

The idea: Opposite roles: in MLA, A and B ARE the normal path, trained from scratch, immovable. LoRA adds a low-rank delta BESIDE frozen weights, to adapt after the fact — mergeable or removable at will.

Why / at what price: The distinction prevents absurd decisions — freezing MLA’s “adapter”, or believing LoRA is required at inference. The price: permanent vigilance; the same algebra serves architectures and adaptation procedures, and only context decides the meaning.