# Teacher guide — DeltaNet: correcting memory

**Duration:** 120 minutes<br>
**Positioning:** Replace blind additive writing with read, compare, correct.<br>
**Expected evidence:** Established mechanisms; numerical simplifications are pedagogical.

## Observable outcomes and preparation

- Compute memory error.
- Apply a delta rule.
- Separate fast weights and trained parameters.

Before class, the instructor runs the worked case and lab, prints the exercise packet, prepares a four-column board—assumption, prediction, observation, delta—and checks that every mathematical notation is paired with dimensions. The demonstration must not become slide reading.

## Diagnostic

1. Explain in one sentence: **Limit of addition**. What observation would falsify your explanation?
2. Explain in one sentence: **Read before writing**. What observation would falsify your explanation?
3. Explain in one sentence: **Local error**. What observation would falsify your explanation?

**Teaching decision:** if two of three answers remain nominal or lack a validity condition, rebuild the vocabulary with a numerical example before any formula. A fluent but unfalsifiable answer does not count as mastery.

## Timed plan

| Time | Activity | Observable evidence |
|---|---|---|
| 0–10 min | Individual diagnostic, then pair comparison | Three answers and one named uncertainty |
| 10–25 min | Situation and vocabulary | Annotated input → state → output diagram |
| 25–55 min | Develop the mechanism on the board | Shapes, assumptions, and intermediate calculation visible |
| 55–75 min | Worked case with deliberate errors | Reasoned correction, not only the right number |
| 75–95 min | Causal lab: predict, change one variable, run | Prediction / observation / delta table |
| 95–112 min | Exercises 1 and 2 with peer correction | Retained artifact and applied rubric |
| 112–120 min | Exit ticket and transfer | Mechanism, boundary, next experiment |

## Teaching notes

# Chapter 9 — DeltaNet: correct the memory

### 9.1 The delta rule

**Goal:** replace blind addition with error correction. Before writing the desired value v, the model asks what the current memory returns for key k. It writes only the difference.

**Intuition:** A teacher does not rewrite an entire answer when only one line is wrong; the teacher marks the difference to correct.

**Step by step**

- Read old value: v_old = kᵀS.

- Compute error: e = v − v_old.

- Choose write strength β between 0 and 1.

- Update: S ← S + β k eᵀ.

**Worked example:** S is zero, k=[1,0], desired v=[2,3], β=1. Old value=[0,0], error=[2,3], so S becomes [[2,3],[0,0]]. Now change the desired value to [5,1]. Old value=[2,3], error=[3,−2], and the update changes the first row to [5,1] rather than adding the whole new value.

**Why it matters:** DeltaNet treats S as fast weights: temporary numbers that change while reading the current sequence. They differ from ordinary model parameters, which are learned during training and then reused.

**Quick check:** If memory already returns the desired value, what is the error? Answer: zero, so no correction is needed.

### 9.2 Orientation and shapes

**Goal:** avoid confusion about transposes. In this course S is a key-by-value matrix. A column key k has shape dₖ × 1, a value row has shape 1 × dᵥ, and k vᵀ has shape dₖ × dᵥ.

**Intuition:** Shapes are like connector sizes. If connectors do not fit, the multiplication is not defined.

**Step by step**

- Write: (dₖ×1)(1×dᵥ) → dₖ×dᵥ.

- Read: (1×dₖ)(dₖ×dᵥ) → 1×dᵥ.

- Some papers transpose every object and use a value-by-key convention. The mechanism can still be equivalent.

**Worked example:** Worked shape example: dₖ=2 and dᵥ=3. Then S has shape 2×3. A 1×2 query multiplied by S produces a 1×3 value.

**Why it matters:** Shape checking lets you read unfamiliar equations safely without memorizing every symbol.

**Quick check:** What shape results from (1×4)(4×6)? Answer: 1×6.

### Running the worked case

If S reads [0.6,0.2] for target [1,0], error [0.4,−0.2] drives only the missing correction, unlike fully adding the target again.

Do not reveal the result at once. Ask learners to predict the next operation, its shape, and the expected sign. After each line ask: “What changed? What stayed fixed? Which assumption did we use?” A calculation error repaired with a causal chain is worth more than a guessed result.

### Lab protocol

1. Write a qualitative and, where possible, numerical prediction before touching a control.
2. Change one variable only; retain a capture or record initial and final values.
3. Explain the delta through the mechanism, not “the tool did that.”
4. Test one boundary value and state where the model stops representing a real system.

## Misconceptions

| # | Observable misconception | Grounded correction | Probe |
|---|---|---|---|
| 1 | “Limit of addition guarantees the outcome without assumptions or measurement.” | The diagnosis is structural: S ← S + k vᵀ never consults S. The write is decided without knowing what memory already holds — repeating information amplifies it, correcting it entangles it. | Ask for a counterexample, then restate the mechanism with its validity condition. |
| 2 | “Read before writing guarantees the outcome without assumptions or measurement.” | Memory predicts first: v̂ = Sᵀk. On S = [[2,3],[5,1]] with k=[1,0], v̂ = [2,3] — exactly what an aligned query would read. The write becomes conditional on the state, no longer only on the input. | Ask for a counterexample, then restate the mechanism with its validity condition. |
| 3 | “Local error guarantees the outcome without assumptions or measurement.” | Only the difference: e = v − v̂ = [7,6]. It carries both what is missing (positive components) and what must be erased (negative ones). And if v̂ = v, then e = [0,0]: the redundant token writes nothing. | Ask for a counterexample, then restate the mechanism with its validity condition. |

> **Boundary to maintain:** The delta rule reduces some interference; it does not create unlimited capacity, and stability depends on keys, gates, and normalization.

## Probing questions

1. If we remove or reverse **Limit of addition**, which output changes first, and what observation would show it?
2. If we remove or reverse **Read before writing**, which output changes first, and what observation would show it?
3. If we remove or reverse **Local error**, which output changes first, and what observation would show it?
4. If we remove or reverse **Delta update**, which output changes first, and what observation would show it?
5. If we remove or reverse **Fast weights**, which output changes first, and what observation would show it?
6. If we remove or reverse **Orientations**, which output changes first, and what observation would show it?

## Assessment

| Level | Criterion |
|---|---|
| 0 | Repeats terms without connecting input, transformation, and output. |
| 1 | Describes the chain but checks neither shape nor assumption. |
| 2 | Executes the case, explains the result, and names one limitation. |
| 3 | Transfers to a new case, compares an alternative, and proposes a measurement that could invalidate the choice. |

**Exit threshold:** level 2 on the worked case and at least one exercise; a memorized formula without interpretation remains level 1.

## Observation and remediation protocol

During discussion, the instructor records evidence rather than impressions. Evidence of understanding contains a named object, a justified transformation, and a checkable consequence. If a learner gives the right result without a chain, ask for the preceding line. If the chain is coherent but the result is wrong, preserve the chain and isolate the arithmetic error. If vocabulary from another concept is used, compare both mechanisms in an input, state, output, cost, and boundary table. Remediation targets the first break only: vocabulary, shapes, operation, interpretation, or claim scope. After correction, use a neighboring case with a changed value; success on the same example does not prove transfer. For pair work, assign operator and verifier roles, then swap. The verifier does not supply the answer: they request an assumption, check the shape, and ask what observation could contradict the reasoning. The instructor retains the exit ticket and classifies the dominant break. The next session opens with a three-minute problem aimed at that break instead of repeating the whole lesson.

## Differentiation

- **Support:** provide shapes and the first transformation; let the learner complete interpretation and boundary.
- **Core path:** worked case without result, lab with one assigned variable, diagnostic exercise.
- **Extension:** change one assumption, compare two mechanisms, and define the metric that would decide between them.

## Post-session follow-up

Within twenty-four hours, return each annotated exit ticket with one priority, the exact resource to reopen, and a mini-case different from the worked case. Revision requires three items: a written prediction, a retained trace, and one sentence explaining the delta. At the next session, sample two submissions: one that repaired the break and one that remains ambiguous. Discuss them anonymously, then state the criterion that separates them. Do not use completion rate as evidence of mastery. Evidence of remediation is a correct chain on a new case with a stated boundary. If the same break appears in more than one third of the group, repair the support or demonstration before blaming learners.

Follow-up closes only when the new artifact shows the causal chain, check, and boundary—not merely when a file has been submitted.

## Sources and evidence boundary

- Owner-supplied bilingual course packet, Chapter 9.
- Schlag, Irie & Schmidhuber, “Linear Transformers Are Secretly Fast Weight Programmers”, ICML (2021).
- Yang et al., “Parallelizing Linear Transformers with the Delta Rule over Sequence Length” (DeltaNet), NeurIPS (2024).
- Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

> **Scope:** Established mechanisms; numerical simplifications are pedagogical. These references support the session frame; they do not turn a reported product choice into an independently verified result.

## Exit ticket

In no more than six lines: mechanism; calculation or trace; observation; boundary; evidence level; next experiment. The instructor marks one priority causal break for revision.