Teacher guide — Linear attention and fixed-size matrix memory

Duration: 120 minutes
Positioning: Move from a growing token notebook to a fixed state matrix, then measure reads and interference.
Expected evidence: Established mechanisms; numerical simplifications are pedagogical.

Observable outcomes and preparation

Use dot and outer products.
Write and read a state matrix.
Diagnose interference.

Before class, the instructor runs the worked case and lab, prints the exercise packet, prepares a four-column board—assumption, prediction, observation, delta—and checks that every mathematical notation is paired with dimensions. The demonstration must not become slide reading.

Diagnostic

Explain in one sentence: Matrix foundations. What observation would falsify your explanation?
Explain in one sentence: Dot product. What observation would falsify your explanation?
Explain in one sentence: Outer product. What observation would falsify your explanation?

Teaching decision: if two of three answers remain nominal or lack a validity condition, rebuild the vocabulary with a numerical example before any formula. A fluent but unfalsifiable answer does not count as mastery.

Timed plan

Time	Activity	Observable evidence
0–10 min	Individual diagnostic, then pair comparison	Three answers and one named uncertainty
10–25 min	Situation and vocabulary	Annotated input → state → output diagram
25–55 min	Develop the mechanism on the board	Shapes, assumptions, and intermediate calculation visible
55–75 min	Worked case with deliberate errors	Reasoned correction, not only the right number
75–95 min	Causal lab: predict, change one variable, run	Prediction / observation / delta table
95–112 min	Exercises 1 and 2 with peer correction	Retained artifact and applied rubric
112–120 min	Exit ticket and transfer	Mechanism, boundary, next experiment

Teaching notes

Chapter 8 — Linear attention and fixed-size matrix memory

8.0 Matrix foundations from zero

Goal: understand the small pieces of mathematics used from this chapter onward. A scalar is one number, such as 3. A vector is an ordered list, such as [2, 5]. A matrix is a rectangular grid of numbers. Its shape is written rows × columns.

Intuition: Think of a vector as one student’s report card and a matrix as the whole class register. Rows can represent students; columns can represent subjects.

Step by step

A = [[1, 2, 3], [4, 5, 6]] has 2 rows and 3 columns, so its shape is 2 × 3.
The transpose swaps rows and columns: transpose([2, 5]) turns a row into a column.
A dot product multiplies matching entries and adds them: [2, 3] · [4, 5] = 2×4 + 3×5 = 23.
An outer product makes a grid: column [2, 3] × row [4, 5] = [[8, 10], [12, 15]].
Matrix multiplication is repeated dot products. Shapes must connect: (2 × 3)(3 × 4) gives (2 × 4).

Worked example: x = [2, 1] and W = [[3, 0], [4, 5]]. Then xW = [2×3 + 1×4, 2×0 + 1×5] = [10, 5]. The matrix mixed the two input coordinates into two new coordinates.

Why it matters: These operations are the grammar of neural networks. We will always state what a matrix stores and check its shape.

Quick check: What is the shape of a grid with 4 rows and 7 columns? Answer: 4 × 7.

8.1 From a growing notebook to a fixed-size summary

Goal: see why ordinary attention becomes expensive. Exact causal attention keeps a key and value for every earlier token. During generation, the key-value cache therefore grows with the conversation.

Intuition: Exact attention is like keeping every receipt. Linear attention tries to maintain one running accounting table instead.

Step by step

Transform each key k with a feature map φ(k). A feature map simply changes coordinates before comparison.
Write the key-value association into a state matrix S using an outer product: S ← S + φ(k) vᵀ.
Read with a query q: output ≈ φ(q)ᵀS. A normalization term may also be used.
The dimensions of S depend on representation width, not on the number of tokens.

Worked example: start with S = [[0,0],[0,0]]. Let k = [1,0] and v = [3,4]. The outer product is [[3,4],[0,0]], so the new S is [[3,4],[0,0]]. Query q = [1,0] reads qᵀS = [3,4]. Query [0,1] reads [0,0].

Why it matters: The memory remains the same size even after many tokens. This can reduce memory growth and make recurrent decoding efficient.

Quick check: Does fixed-size memory mean unlimited perfect memory? Answer: No. Many associations must share the same limited grid.

8.2 Interference

Goal: understand the main weakness of simple additive memory. If two keys point in similar directions, their writes overlap inside S. A later query may retrieve a mixture.

Intuition: Imagine writing several answers in the same small square of a whiteboard. The ink overlaps.

Step by step

Write k₁ = [1,0], v₁ = [1,0].
Write k₂ = [1,1], v₂ = [0,1].
The state becomes [[1,1],[0,1]].
Reading with q = [1,0] returns [1,1], not the original [1,0]. The second write leaked into the first read.

Worked example: This toy example shows cross-talk. Real models use learned projections, gates, normalization and correction rules, but finite memory still creates trade-offs.

Why it matters: We need a write rule that can correct what a key currently remembers instead of only adding forever.

Quick check: Why can two memories interfere? Answer: Their key directions are not perfectly separate, so their matrix writes overlap.

Running the worked case

With zero S, k=[1,0] and v=[2,3], the write gives [[2,3],[0,0]]. Query q=[1,0] reads [2,3]. Trying to add a length-3 key [1,0,2] also reveals why shape checks are mandatory.

Do not reveal the result at once. Ask learners to predict the next operation, its shape, and the expected sign. After each line ask: “What changed? What stayed fixed? Which assumption did we use?” A calculation error repaired with a causal chain is worth more than a guessed result.

Lab protocol

Write a qualitative and, where possible, numerical prediction before touching a control.
Change one variable only; retain a capture or record initial and final values.
Explain the delta through the mechanism, not “the tool did that.”
Test one boundary value and state where the model stops representing a real system.

Misconceptions

#	Observable misconception	Grounded correction	Probe
1	“Matrix foundations guarantees the outcome without assumptions or measurement.”	A vector is an ordered list of d numbers; a matrix organizes rows and columns (d_k × d_v). Dimensions dictate which multiplications are valid: here k(2) and v(2) build S(2×2), and a length-3 key [1;0;2] is rejected before any computation.	Ask for a counterexample, then restate the mechanism with its validity condition.
2	“Dot product guarantees the outcome without assumptions or measurement.”	The dot product compresses alignment into one number: q·k = Σqᵢkᵢ. Here q=[1,0] against k=[1,0] gives 1 (aligned); against k=[0,1] it gives 0 (orthogonal). That is the addressing mechanism: strong = relevant, zero = ignored.	Ask for a counterexample, then restate the mechanism with its validity condition.
3	“Outer product guarantees the outcome without assumptions or measurement.”	The outer product k vᵀ builds a matrix: the row is selected by k, the content is carried by v. For k=[1,0] and v=[2,3]: k vᵀ = [[2,3],[0,0]] — the value is filed on row 1, row 2 stays blank. Writes accumulate: S ← S + k vᵀ.	Ask for a counterexample, then restate the mechanism with its validity condition.

Boundary to maintain: Fixed memory means neither perfect memory nor infinite context: capacity and interference remain bounded.

Probing questions

If we remove or reverse Matrix foundations, which output changes first, and what observation would show it?
If we remove or reverse Dot product, which output changes first, and what observation would show it?
If we remove or reverse Outer product, which output changes first, and what observation would show it?
If we remove or reverse Reading state, which output changes first, and what observation would show it?
If we remove or reverse Fixed memory, which output changes first, and what observation would show it?
If we remove or reverse Interference, which output changes first, and what observation would show it?

Assessment

Level	Criterion
0	Repeats terms without connecting input, transformation, and output.
1	Describes the chain but checks neither shape nor assumption.
2	Executes the case, explains the result, and names one limitation.
3	Transfers to a new case, compares an alternative, and proposes a measurement that could invalidate the choice.

Exit threshold: level 2 on the worked case and at least one exercise; a memorized formula without interpretation remains level 1.

Observation and remediation protocol

During discussion, the instructor records evidence rather than impressions. Evidence of understanding contains a named object, a justified transformation, and a checkable consequence. If a learner gives the right result without a chain, ask for the preceding line. If the chain is coherent but the result is wrong, preserve the chain and isolate the arithmetic error. If vocabulary from another concept is used, compare both mechanisms in an input, state, output, cost, and boundary table. Remediation targets the first break only: vocabulary, shapes, operation, interpretation, or claim scope. After correction, use a neighboring case with a changed value; success on the same example does not prove transfer. For pair work, assign operator and verifier roles, then swap. The verifier does not supply the answer: they request an assumption, check the shape, and ask what observation could contradict the reasoning. The instructor retains the exit ticket and classifies the dominant break. The next session opens with a three-minute problem aimed at that break instead of repeating the whole lesson.

Differentiation

Support: provide shapes and the first transformation; let the learner complete interpretation and boundary.
Core path: worked case without result, lab with one assigned variable, diagnostic exercise.
Extension: change one assumption, compare two mechanisms, and define the metric that would decide between them.

Post-session follow-up

Within twenty-four hours, return each annotated exit ticket with one priority, the exact resource to reopen, and a mini-case different from the worked case. Revision requires three items: a written prediction, a retained trace, and one sentence explaining the delta. At the next session, sample two submissions: one that repaired the break and one that remains ambiguous. Discuss them anonymously, then state the criterion that separates them. Do not use completion rate as evidence of mastery. Evidence of remediation is a correct chain on a new case with a stated boundary. If the same break appears in more than one third of the group, repair the support or demonstration before blaming learners.

Follow-up closes only when the new artifact shows the causal chain, check, and boundary—not merely when a file has been submitted.

Sources and evidence boundary

Owner-supplied bilingual course packet, Chapter 8.
Katharopoulos et al., “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention”, ICML (2020).
Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Scope: Established mechanisms; numerical simplifications are pedagogical. These references support the session frame; they do not turn a reported product choice into an independently verified result.

Exit ticket

In no more than six lines: mechanism; calculation or trace; observation; boundary; evidence level; next experiment. The instructor marks one priority causal break for revision.