# Exercises with solutions — DeltaNet: correcting memory **General instruction:** every answer must show data, transformation, result, one check, and one limitation. A bare number or copied definition is insufficient. > **Starting data:** If S reads [0.6,0.2] for target [1,0], error [0.4,−0.2] drives only the missing correction, unlike fully adding the target again. > > **Boundary to retain:** The delta rule reduces some interference; it does not create unlimited capacity, and stability depends on keys, gates, and normalization. ## Exercise 1 — Calculated trace — Limit of addition Reproduce and annotate the chain `v̂=Sᵀk`. Replace the first target coordinate in [1,0] with 1.2 while keeping readout [0.6,0.2]. Recompute correction error. **Deliverable:** a data → operation → result → interpretation table, plus two sentences about the changed value.
Worked solution If S reads [0.6,0.2] for target [1,0], error [0.4,−0.2] drives only the missing correction, unlike fully adding the target again. **Solved variant:** Target-minus-read error moves from [0.4,−0.2] to [0.6,−0.2]. The first correction rises by 50% while the second is unchanged; the rule corrects observed error rather than rewriting all memory indiscriminately. The diagnosis is structural: S ← S + k vᵀ never consults S. The write is decided without knowing what memory already holds — repeating information amplifies it, correcting it entangles it. Memory predicts first: v̂ = Sᵀk. On S = [[2,3],[5,1]] with k=[1,0], v̂ = [2,3] — exactly what an aligned query would read. The write becomes conditional on the state, no longer only on the input. The minimum check covers dimensions, sign, and order of magnitude. If the observed change contradicts the prediction, locate the first operation whose direction changes instead of fixing only the final line.
### Rubric Exercise 1 — /10 | Criterion | Points | |---|---:| | Explicit data and shapes | 2 | | Traceable calculation | 3 | | Prediction before variation | 2 | | Interpretation and check | 2 | | Named limitation | 1 | ## Exercise 2 — Diagnose a seductive explanation — Local error A colleague claims: « Local error proves the system will be accurate, fast, and stable in every context. » 1. Separate mechanism, assumption, observation, and conclusion. 2. Name two correct lesson elements and two unsupported extrapolations. 3. Propose a bounded experiment with controlled variable, metric, and stop threshold. 4. Rewrite the claim as one defensible sentence.
Reasoned solution Only the difference: e = v − v̂ = [7,6]. It carries both what is missing (positive components) and what must be erased (negative ones). And if v̂ = v, then e = [0,0]: the redundant token writes nothing. S ← S + β k eᵀ: k localizes (row 1 only), e carries the content, β sets the dose. With β=1 and ‖k‖=1: exact replacement, [2,3] → [9,9], row 2 untouched. With β=0.5: halfway, [5.5, 6]. The delta rule reduces some interference; it does not create unlimited capacity, and stability depends on keys, gates, and normalization. The claim mixes a local relation with a global guarantee. A defensible version states only the observed mechanism, test conditions, and measured metric. Stop the test if shapes become invalid, the metric crosses the declared degradation threshold, or another variable changes.
### Rubric Exercise 2 — /10 2 points per element: separation, lesson grounding, extrapolations, protocol, and rewrite. ## Exercise 3 — Architecture decision and transfer — Fast weights You must reproduce the worked case “If S reads [0.6,0.2] for target [1,0], error [0.4,−0.2] drives only the missing correction, unlike fully adding the target again.” under two conditions. Option A uses the full chain through “Fast weights.” Option B is a transparent baseline that retains “Limit of addition,” calculates the expected output directly, and does not use the compression or adjustment mechanism studied. Build a decision record containing: - the workload and dominant constraint; - each option’s mechanism, without slogans; - one quality, memory, or latency prediction; - one case where your preferred procedure loses; - an A/B protocol, metrics, and rollback threshold; - a bounded verdict: choose, defer, or reject.
Elements of a strong solution Two speeds coexist: S is a FAST weight, rewritten token by token and discarded at sequence end; the matrices producing k, v, β are SLOW parameters, frozen at inference. Training learns how to drive the memory, not its contents. Neither: the whole presentation transposes (rows ↔ columns) without changing the mechanism. What is immutable: read, compare, correct — and the DECLARED shapes: k(d_k), e(d_v), S(d_k×d_v) in one convention, transposed in the other. Established mechanisms; numerical simplifications are pedagogical. A strong answer does not make the newer mechanism the default winner. It retains a measurable baseline, sets thresholds before testing, and separates component cost from whole-system behavior. The verdict names what remains uncertain and the next evidence that could change it.
### Rubric Exercise 3 — /15 | Criterion | Points | |---|---:| | Framing and baseline | 3 | | Compared causal chains | 4 | | Protocol and metrics | 4 | | Rollback threshold | 2 | | Bounded verdict | 2 | ## Extension Repeat Exercise 3 after reversing the dominant constraint. If you optimized memory, impose a strict quality floor; if you optimized fidelity, halve the memory budget. Identify the first part of the verdict that changes and the evidence required. ## Review before submission Review the packet as if another group had to reproduce it without speaking to you. Are all starting values or assumptions present? Are shapes or roles stated before operations? Does the prediction truly precede the observation? Is the result translated into behavior rather than left as an isolated number? Did you test a boundary value and identify a stop condition? Does the procedure or architecture choice retain a measurable baseline and a rollback threshold set before the test? Finally, highlight one sentence that states what is established, one that remains a hypothesis, and one measurement that could change your verdict. If any element is missing, the work is not reproducible. ## Reference appendix for correction # Chapter 9 — DeltaNet: correct the memory ### 9.1 The delta rule **Goal:** replace blind addition with error correction. Before writing the desired value v, the model asks what the current memory returns for key k. It writes only the difference. **Intuition:** A teacher does not rewrite an entire answer when only one line is wrong; the teacher marks the difference to correct. **Step by step** - Read old value: v_old = kᵀS. - Compute error: e = v − v_old. - Choose write strength β between 0 and 1. - Update: S ← S + β k eᵀ. **Worked example:** S is zero, k=[1,0], desired v=[2,3], β=1. Old value=[0,0], error=[2,3], so S becomes [[2,3],[0,0]]. Now change the desired value to [5,1]. Old value=[2,3], error=[3,−2], and the update changes the first row to [5,1] rather than adding the whole new value. **Why it matters:** DeltaNet treats S as fast weights: temporary numbers that change while reading the current sequence. They differ from ordinary model parameters, which are learned during training and then reused. **Quick check:** If memory already returns the desired value, what is the error? Answer: zero, so no correction is needed. ### 9.2 Orientation and shapes **Goal:** avoid confusion about transposes. In this course S is a key-by-value matrix. A column key k has shape dₖ × 1, a value row has shape 1 × dᵥ, and k vᵀ has shape dₖ × dᵥ. **Intuition:** Shapes are like connector sizes. If connectors do not fit, the multiplication is not defined. **Step by step** - Write: (dₖ×1)(1×dᵥ) → dₖ×dᵥ. - Read: (1×dₖ)(dₖ×dᵥ) → 1×dᵥ. - Some papers transpose every object and use a value-by-key convention. The mechanism can still be equivalent. **Worked example:** Worked shape example: dₖ=2 and dᵥ=3. Then S has shape 2×3. A 1×2 query multiplied by S produces a 1×3 value. **Why it matters:** Shape checking lets you read unfamiliar equations safely without memorizing every symbol. **Quick check:** What shape results from (1×4)(4×6)? Answer: 1×6. ## Sources and evidence boundary - Owner-supplied bilingual course packet, Chapter 9. - Schlag, Irie & Schmidhuber, “Linear Transformers Are Secretly Fast Weight Programmers”, ICML (2021). - Yang et al., “Parallelizing Linear Transformers with the Delta Rule over Sequence Length” (DeltaNet), NeurIPS (2024). - Course source packet supplied by the owner; named-product details remain source-reported until primary verification. > **Scope:** Established mechanisms; numerical simplifications are pedagogical. These references support the session frame; they do not turn a reported product choice into an independently verified result.