Applied AI · advanced · Session 16
Exercises with solutions — Gated DeltaNet and selective forgetting
← Back to courseFrançaisMarkdown source

Exercises with solutions — Gated DeltaNet and selective forgetting

General instruction: every answer must show data, transformation, result, one check, and one limitation. A bare number or copied definition is insufficient.

Starting data: With α=0.8, a trace of value 1 becomes 0.8 then 0.64 without new writes. Correction β=0.5 then adds only half the observed error.

Boundary to retain: The Mamba parallel is conceptual. It does not imply Gated DeltaNet and Mamba are interchangeable.

Exercise 1 — Calculated trace — Why forget

Reproduce and annotate the chain S_t=α_t S_{t−1}+update_t. Increase α from 0.8 to 0.96 for an initial trace of 1 with no new write. Calculate the next two states and compare retention.

Deliverable: a data → operation → result → interpretation table, plus two sentences about the changed value.

Worked solution

With α=0.8, a trace of value 1 becomes 0.8 then 0.64 without new writes. Correction β=0.5 then adds only half the observed error.

Solved variant: With α=0.8 the trace becomes 0.8 then 0.64. With α=0.96 it becomes 0.96 then 0.9216. Retention rises sharply, but α near 1 also slows forgetting of stale information.

Reverse the reflex: forgetting is a function, not a failure. A fixed-capacity memory that never frees anything eventually distinguishes nothing; erasing the stale is exactly what keeps the present readable. A gate α_t ∈ [0,1], produced at every token, multiplies the state before the write: S′ = α_t·S_{t−1}. At 0.8 a trace follows 1 → 0.8 → 0.64, and its half-life is ln(0.5)/ln(0.8) ≈ 3.1 tokens. The model learns when to tighten or open the gate. The minimum check covers dimensions, sign, and order of magnitude. If the observed change contradicts the prediction, locate the first operation whose direction changes instead of fixing only the final line.

Rubric Exercise 1 — /10

Criterion Points
Explicit data and shapes 2
Traceable calculation 3
Prediction before variation 2
Interpretation and check 2
Named limitation 1

Exercise 2 — Diagnose a seductive explanation — Write gate

A colleague claims: « Write gate proves the system will be accurate, fast, and stable in every context. »

  1. Separate mechanism, assumption, observation, and conclusion.
  2. Name two correct lesson elements and two unsupported extrapolations.
  3. Propose a bounded experiment with controlled variable, metric, and stop threshold.
  4. Rewrite the claim as one defensible sentence.
Reasoned solution

β_t doses the delta correction: S_t = S′ + β_t k eᵀ. In the trace, β = 0.5 applies only half the error — exactly (1−β)·e remains. β ≈ 0 ignores, β = 1 fully corrects, decided token by token. Stability rests on binding details: α bounded in [0,1] — not merely |α| ≤ 1, a negative α would flip the sign each step —, normalized keys, and a declared operation order: decay, read, correct. The Mamba parallel is conceptual. It does not imply Gated DeltaNet and Mamba are interchangeable.

The claim mixes a local relation with a global guarantee. A defensible version states only the observed mechanism, test conditions, and measured metric. Stop the test if shapes become invalid, the metric crosses the declared degradation threshold, or another variable changes.

Rubric Exercise 2 — /10

2 points per element: separation, lesson grounding, extrapolations, protocol, and rewrite.

Exercise 3 — Architecture decision and transfer — Effective capacity

You must reproduce the worked case “With α=0.8, a trace of value 1 becomes 0.8 then 0.64 without new writes. Correction β=0.5 then adds only half the observed error.” under two conditions. Option A uses the full chain through “Effective capacity.” Option B is a transparent baseline that retains “Why forget,” calculates the expected output directly, and does not use the compression or adjustment mechanism studied. Build a decision record containing:

Elements of a strong solution

The right quantity is useful lifetime: half-life ≈ 69 tokens at α = 0.99, ≈ 3.1 at α = 0.80, 1.0 at α = 0.5. Effective capacity combines S’s size, forgetting, and key separation — not the advertised maximum context length. The overlap is real: recurrence plus learned controls. So is the difference: the delta memory READS its state (v̂ = S′ᵀk) and writes the error; a classical SSM filters the signal with no associative read. Same family of motivations, distinct equations. Established: the delta rule, exponential decay, and the half-lives computed here. Pedagogical: the α, β values and the 2×2 matrices of the trace. Reported: the exact gate choices of named models.

A strong answer does not make the newer mechanism the default winner. It retains a measurable baseline, sets thresholds before testing, and separates component cost from whole-system behavior. The verdict names what remains uncertain and the next evidence that could change it.

Rubric Exercise 3 — /15

Criterion Points
Framing and baseline 3
Compared causal chains 4
Protocol and metrics 4
Rollback threshold 2
Bounded verdict 2

Extension

Repeat Exercise 3 after reversing the dominant constraint. If you optimized memory, impose a strict quality floor; if you optimized fidelity, halve the memory budget. Identify the first part of the verdict that changes and the evidence required.

Review before submission

Review the packet as if another group had to reproduce it without speaking to you. Are all starting values or assumptions present? Are shapes or roles stated before operations? Does the prediction truly precede the observation? Is the result translated into behavior rather than left as an isolated number? Did you test a boundary value and identify a stop condition? Does the procedure or architecture choice retain a measurable baseline and a rollback threshold set before the test? Finally, highlight one sentence that states what is established, one that remains a hypothesis, and one measurement that could change your verdict. If any element is missing, the work is not reproducible.

Reference appendix for correction

Chapter 11 — Forgetting: Gated DeltaNet and the Mamba connection

11.1 Why forgetting helps

Goal: prevent old information from occupying memory forever. A gate is a learned number, often between 0 and 1, that controls how much information passes.

Intuition: A whiteboard is useful because we can partly erase outdated notes before writing new ones.

Step by step

Worked example: if S contains 10 and α=0.8, decay leaves 8 before the new write. Repeating 0.8 decay gives 10, 8, 6.4, 5.12… Old influence fades smoothly.

Why it matters: Forgetting is selective resource management. It can improve effective capacity when old details are no longer useful.

Quick check: Does α=1 erase memory? Answer: No; it keeps the old state unchanged before the new write.

11.2 Connection to Mamba

Goal: place gated recurrent memory in a broader family. Mamba is a selective state-space model: it carries a fixed-size state through sequence time and lets input-dependent controls decide what to retain and inject.

Intuition: Both systems resemble a running scientific experiment: keep a state, decay or transform it, add new evidence, and produce an output.

Step by step

Worked example: The connection is conceptual, not an assertion that the two architectures are identical.

Why it matters: Recognizing families helps us compare design choices without collapsing distinct mechanisms into one name.

Quick check: What is the shared central idea? Answer: a learned fixed-size state updated as the sequence advances.

Sources and evidence boundary

Scope: Established: the delta rule, exponential decay, and the half-lives computed here. Pedagogical: the α, β values and the 2×2 matrices of the trace. Reported: the exact gate choices of named models. These references support the session frame; they do not turn a reported product choice into an independently verified result.