Teacher guide — KV cache, recurrent memory, MLA, and low rank

Duration: 120 minutes
Positioning: Compare three memory budgets and distinguish per-token compression, fixed state, and low-rank factorization.
Expected evidence: Mixed: established mechanisms + source-reported Kimi K3-style choices.

Observable outcomes and preparation

Compute KV-cache growth.
Compare MLA and fixed state.
Distinguish architectural low rank from LoRA.

Before class, the instructor runs the worked case and lab, prints the exercise packet, prepares a four-column board—assumption, prediction, observation, delta—and checks that every mathematical notation is paired with dimensions. The demonstration must not become slide reading.

Diagnostic

Explain in one sentence: Standard KV cache. What observation would falsify your explanation?
Explain in one sentence: Fixed recurrent state. What observation would falsify your explanation?
Explain in one sentence: MLA. What observation would falsify your explanation?

Teaching decision: if two of three answers remain nominal or lack a validity condition, rebuild the vocabulary with a numerical example before any formula. A fluent but unfalsifiable answer does not count as mastery.

Timed plan

Time	Activity	Observable evidence
0–10 min	Individual diagnostic, then pair comparison	Three answers and one named uncertainty
10–25 min	Situation and vocabulary	Annotated input → state → output diagram
25–55 min	Develop the mechanism on the board	Shapes, assumptions, and intermediate calculation visible
55–75 min	Worked case with deliberate errors	Reasoned correction, not only the right number
75–95 min	Causal lab: predict, change one variable, run	Prediction / observation / delta table
95–112 min	Exercises 1 and 2 with peer correction	Retained artifact and applied rubric
112–120 min	Exit ticket and transfer	Mechanism, boundary, next experiment

Teaching notes

Chapter 13 — Full records versus compressed memory

13.1 Two memory strategies

Goal: compare exact token records with summaries. Standard key-value caching stores per-token records. Recurrent linear memory compresses many tokens into a fixed-size state.

Intuition: A video archive preserves every frame; meeting notes preserve a compact summary. Each is useful for different questions.

Step by step

Per-token cache: grows with context; preserves more direct access to individual tokens.
Fixed state: does not grow with token count; risks interference and information loss.
Hybrid designs can use both mechanisms in different layers or roles.

Worked example: For 1,000 tokens, a per-token method keeps 1,000 records per relevant layer. A fixed-state method keeps one state of predetermined shape. This does not prove it uses less total memory in every implementation, but it explains the scaling difference.

Why it matters: Architecture is a trade-off among fidelity, speed, memory traffic and trainability.

Quick check: Which strategy is more likely to retrieve one exact old token? Answer: the per-token record, although retrieval quality still depends on learned attention.

13.2 Latent compression and MLA

Goal: reduce cache size without collapsing all history into one recurrent matrix. A latent vector is a smaller learned representation. Multi-head Latent Attention, abbreviated MLA, stores compressed latent information and reconstructs head-specific keys or values when needed.

Intuition: Store a zipped folder instead of several expanded copies, then unpack the view needed by each worker.

Step by step

Compress hidden representation x into c = xW_down.
Cache the smaller latent c.
Use learned up-projections to create the key/value forms needed by attention heads.
A head is one parallel attention subspace; multiple heads can learn different relations.

Worked example: Worked shape example: x has 8 coordinates. Compressing to 2 gives c with 2 coordinates. Expanding c back to an 8-coordinate key does not magically preserve every possible 8-D vector; it restricts the key to patterns learnable through the 2-D bottleneck.

Why it matters: MLA is compressed per-token memory, not the same as one fixed recurrent state for the whole past.

Quick check: Does MLA’s cache normally remain constant as token count grows? Answer: No. It can be smaller per token, but still grows with the number of cached tokens.

13.3 Low-rank factorization and LoRA

Goal: understand narrow intermediate spaces. A full 8×8 matrix has 64 entries. Replacing it with an 8×2 matrix followed by a 2×8 matrix uses 16+16=32 entries in this simplified count.

Intuition: A narrow hallway limits how many independent flows can pass at once.

Step by step

Down-project: h = xA.
Up-project: y = hB.
The combined map AB has rank at most the narrow width.
Low-Rank Adaptation, LoRA, usually adds a trainable low-rank update to a frozen base weight; it is related mathematics but not automatically the same use as architectural compression.

Worked example: This distinction prevents a common confusion: every low-rank factorization is not necessarily a LoRA fine-tuning adapter.

Why it matters: Low rank trades flexibility for fewer parameters, less storage or cheaper computation, depending on where it is used.

Quick check: How many entries are in 10×3 plus 3×10 matrices? Answer: 30+30=60.

Running the worked case

With 32 layers, 8 KV heads, d_head=128, BF16, and 4,096 tokens, simplified raw cache is 4,096×32×2×8×128×2 bytes = 536,870,912 bytes = 512 MiB (0.5 GiB). Quartering latent width reduces the per-token term, not linear growth.

Do not reveal the result at once. Ask learners to predict the next operation, its shape, and the expected sign. After each line ask: “What changed? What stayed fixed? Which assumption did we use?” A calculation error repaired with a causal chain is worth more than a guessed result.

Lab protocol

Write a qualitative and, where possible, numerical prediction before touching a control.
Change one variable only; retain a capture or record initial and final values.
Explain the delta through the mechanism, not “the tool did that.”
Test one boundary value and state where the model stops representing a real system.

Misconceptions

#	Observable misconception	Grounded correction	Probe
1	“Standard KV cache guarantees the outcome without assumptions or measurement.”	The cache keeps every past token’s keys and values, per layer: bytes ≈ tokens × layers × 2 × heads × d_head × bytes. Reads are faithful — exact attention over the whole past — and the arithmetic can be redone factor by factor, no calculator.	Ask for a counterexample, then restate the mechanism with its validity condition.
2	“Fixed recurrent state guarantees the outcome without assumptions or measurement.”	The state S — 32 layers × 128 × 128 × BF16 = 1 MiB — summarizes the whole past: 4,096 or 524,288 tokens, still 1 MiB. Growth disappears; this is where the previous sessions’ memories were heading.	Ask for a counterexample, then restate the mechanism with its validity condition.
3	“MLA guarantees the outcome without assumptions or measurement.”	Multi-head Latent Attention compresses each token into a latent vector c_t (512 values) — the only thing cached — then reconstructs K and V through W_UK and W_UV at read time. Width ÷ 4 ⇒ 32 KiB/token, 4 GiB at 131,072 tokens.	Ask for a counterexample, then restate the mechanism with its validity condition.

Boundary to maintain: Simplified formulas omit alignment, quantization, buffers, and sharing details. They compare trends, not promise real footprint.

Probing questions

If we remove or reverse Standard KV cache, which output changes first, and what observation would show it?
If we remove or reverse Fixed recurrent state, which output changes first, and what observation would show it?
If we remove or reverse MLA, which output changes first, and what observation would show it?
If we remove or reverse Low-rank factorization, which output changes first, and what observation would show it?
If we remove or reverse MLA is not fixed memory, which output changes first, and what observation would show it?
If we remove or reverse Low rank is not LoRA, which output changes first, and what observation would show it?

Assessment

Level	Criterion
0	Repeats terms without connecting input, transformation, and output.
1	Describes the chain but checks neither shape nor assumption.
2	Executes the case, explains the result, and names one limitation.
3	Transfers to a new case, compares an alternative, and proposes a measurement that could invalidate the choice.

Exit threshold: level 2 on the worked case and at least one exercise; a memorized formula without interpretation remains level 1.

Observation and remediation protocol

During discussion, the instructor records evidence rather than impressions. Evidence of understanding contains a named object, a justified transformation, and a checkable consequence. If a learner gives the right result without a chain, ask for the preceding line. If the chain is coherent but the result is wrong, preserve the chain and isolate the arithmetic error. If vocabulary from another concept is used, compare both mechanisms in an input, state, output, cost, and boundary table. Remediation targets the first break only: vocabulary, shapes, operation, interpretation, or claim scope. After correction, use a neighboring case with a changed value; success on the same example does not prove transfer. For pair work, assign operator and verifier roles, then swap. The verifier does not supply the answer: they request an assumption, check the shape, and ask what observation could contradict the reasoning. The instructor retains the exit ticket and classifies the dominant break. The next session opens with a three-minute problem aimed at that break instead of repeating the whole lesson.

Differentiation

Support: provide shapes and the first transformation; let the learner complete interpretation and boundary.
Core path: worked case without result, lab with one assigned variable, diagnostic exercise.
Extension: change one assumption, compare two mechanisms, and define the metric that would decide between them.

Post-session follow-up

Within twenty-four hours, return each annotated exit ticket with one priority, the exact resource to reopen, and a mini-case different from the worked case. Revision requires three items: a written prediction, a retained trace, and one sentence explaining the delta. At the next session, sample two submissions: one that repaired the break and one that remains ambiguous. Discuss them anonymously, then state the criterion that separates them. Do not use completion rate as evidence of mastery. Evidence of remediation is a correct chain on a new case with a stated boundary. If the same break appears in more than one third of the group, repair the support or demonstration before blaming learners.

Follow-up closes only when the new artifact shows the causal chain, check, and boundary—not merely when a file has been submitted.

Sources and evidence boundary

Owner-supplied bilingual course packet, Chapter 13.
DeepSeek-AI, “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model” (introduces Multi-head Latent Attention), arXiv:2405.04434 (2024).
Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, ICLR (2022).
Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Scope: Mixed: established mechanisms + source-reported Kimi K3-style choices. These references support the session frame; they do not turn a reported product choice into an independently verified result.

Exit ticket

In no more than six lines: mechanism; calculation or trace; observation; boundary; evidence level; next experiment. The instructor marks one priority causal break for revision.