Gated DeltaNet and selective forgetting

Add decay and write gates to correctable memory.

Applied AI · advanced · Session 16

Mechanism map

inherited state   S_{t−1} = [[2,3],
                                [5,1]]
      │
      ▼
┌──────────────────────────┐   α_t = 0.8   ① DECAY
│  S′ = α_t · S_{t−1}      │   (gate, token-dependent)
└──────────────────────────┘
      │   S′ = [[1.6, 2.4],
      ▼         [4.0, 0.8]]
┌──────────────────────────┐   v̂ = S′ᵀk = [1.6, 2.4]
│  ② READ / COMPARE        │   e  = v − v̂ = [7.4, 6.6]
└──────────────────────────┘   (k=[1,0], v=[9,9])
      │
      ▼
┌──────────────────────────┐   β_t = 0.5   ③ WRITE GATE
│  S_t = S′ + β_t k eᵀ     │   0 → ignore, 1 → correct fully
└──────────────────────────┘
      │
      ▼   S_t = [[5.3, 5.7],
              [4.0, 0.8]]      half-life at α=0.8 ≈ 3.1 tokens
TWO INDEPENDENT GATES: α = what I keep, β = what I write

The problem — Why forget

A document changes topic at token 500; the delta memory keeps everything. The old topic’s associations keep answering the new topic’s queries — session 13’s superposition returns, not through nearby keys but through sheer accumulation over time.

The idea — Why forget

Reverse the reflex: forgetting is a function, not a failure. A fixed-capacity memory that never frees anything eventually distinguishes nothing; erasing the stale is exactly what keeps the present readable.

Why / at what price — Why forget

Forgetting frees capacity and bounds the past’s influence. The price, obvious but real: what has decayed is gone — a fact stated at token 10 and needed at token 10,000 must have been rewritten in between, or it no longer exists.

Check: A document changes topic at token 500 but memory keeps the associations of the first 500. Describe one concrete generation error this produces, then name the mechanism from this session that fixes it.

The problem — Decay gate

Fine, forget — but how fast? A fixed rate would be wrong everywhere: a dialogue wants to hold 50 tokens, a contract 5,000. Forgetting must be driven by content, not by a global constant.

The idea — Decay gate

A gate α_t ∈ [0,1], produced at every token, multiplies the state before the write: S′ = α_t·S_{t−1}. At 0.8 a trace follows 1 → 0.8 → 0.64, and its half-life is ln(0.5)/ln(0.8) ≈ 3.1 tokens. The model learns when to tighten or open the gate.

S_t=α_t S_{t−1}+update_t

Why / at what price — Decay gate

Forgetting becomes measurable — a half-life — and contextual. The price: decay is global. Row 2 of the trace, never rewritten, decays anyway; whatever is not regularly rewritten evaporates, relevant or not.

Check: With α = 0.8, a trace is 0.64 after two steps. How many tokens until it falls below 0.1, and how do you recover the 3.1 half-life from α alone?

Visual support — Decay gate

trace of a value-1 write, never rewritten (α = 0.8)

1.00 ●
0.80 │  ●
0.64 │     ●
0.51 │        ●
     │           ●
     │              ●  …
0.10 ┤─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ● ─   crossed around 11 tokens
     └────────────────────────▶ tokens

half-life = ln(0.5)/ln(α) ≈ 3.1 tokens

The problem — Write gate

Symmetrically: does every token deserve to write? A redundant “the” writing at full strength wastes capacity and pushes the state toward interference; a critical correction written timidly corrects nothing.

The idea — Write gate

β_t doses the delta correction: S_t = S′ + β_t k eᵀ. In the trace, β = 0.5 applies only half the error — exactly (1−β)·e remains. β ≈ 0 ignores, β = 1 fully corrects, decided token by token.

Why / at what price — Write gate

Two independent gates: α decides what stays, β what enters — fine-grained, and learned. The price: two settings that can compensate each other; diagnosing “memory too short” requires separating α’s effect from β’s.

Check: On the decayed state S′ = [[1.6, 2.4],[4, 0.8]] with e = [7.4, 6.6], give S for β=0 then β=1. Which kind of token in the text would justify each of those two settings?

The problem — Stability

These recurrences run 100,000 times in a row. A gate at 1.05 — 5% too much — multiplies the state by 7 in 40 tokens, by 131 in 100. And the same code with correct-then-decay ordering produces different numbers: two models for one formula.

The idea — Stability

Stability rests on binding details: α bounded in [0,1] — not merely |α| ≤ 1, a negative α would flip the sign each step —, normalized keys, and a declared operation order: decay, read, correct.

Why / at what price — Stability

These constraints make the recurrence safe at arbitrary length. The price: nothing is cosmetic — an implementation writing S = αS + βk eᵀ without stating the order is irreproducible. The details are the architecture.

Check: Two implementations start from S₀ = [[2,3],[5,1]] with α=0.8, β=0.5, k=[1,0], v=[9,9]. One decays then corrects: [[5.30, 5.70],[4.00, 0.80]]. The other corrects then decays: [[4.40, 4.80],[4.00, 0.80]]. Only one detail changed — which? Why is a report writing “S = αS + βk eᵀ” without stating the order irreproducible?

Visual support — Stability

same formula, two orders, two models

① decay → correct : S₁ = α·S₀ + β k e′ᵀ → [[5.30, 5.70],…]
② correct → decay : S₁ = α·(S₀ + β k eᵀ) → [[4.40, 4.80],…]
                                  ↑
                in ②, the fresh correction suffers
                a decay it never lived through

The problem — Effective capacity

S is still 2×2 — yet α = 0.99 and α = 0.80 yield radically different memories. If the physical size never moves, what exactly does this model’s “memory” measure?

The idea — Effective capacity

The right quantity is useful lifetime: half-life ≈ 69 tokens at α = 0.99, ≈ 3.1 at α = 0.80, 1.0 at α = 0.5. Effective capacity combines S’s size, forgetting, and key separation — not the advertised maximum context length.

Why / at what price — Effective capacity

Half-life gives an honest language for comparing memories — and for reading announcements: “1 M token context” says nothing about what survives 1,000 steps. The price: it is an average; actual retention depends on the learned gates, hence on content.

Check: The size of S stays 2×2 throughout the session. At α=0.99 a trace’s half-life is ≈ 69 tokens; at α=0.80, ≈ 3.1. What changed if not physical capacity? Restate “1 M token context” as one sentence containing a half-life.

Visual support — Effective capacity

α           0.50           0.80           0.99
            │              │              │
half-life   1 token        3.1 tokens     69 tokens
            working        local          near-retention,
            memory         context        accumulated interference

“1 M token context” without a half-life:
the length of the paper, not of the memory

The problem — Connection to Mamba

Two families — SSMs (Mamba) and gated delta memories — advertise the same pitch: fixed state, input-dependent gates, long context. Should you conclude they are interchangeable, and benchmark one to choose the other?

The idea — Connection to Mamba

The overlap is real: recurrence plus learned controls. So is the difference: the delta memory READS its state (v̂ = S′ᵀk) and writes the error; a classical SSM filters the signal with no associative read. Same family of motivations, distinct equations.

Why / at what price — Connection to Mamba

The parallel guides intuition and tool transfer (chunking, gates). The price of the easy analogy: non-transferable conclusions — a Mamba result proves nothing about Gated DeltaNet, and vice versa. Comparing requires both implementations, same task, same budget.

Check: Mamba and Gated DeltaNet both keep recurrent state with input-dependent controls. Name one thing the read-and-compare step does (v̂ = S′ᵀk, e = v − v̂) that “recurrent state + gate” alone does not describe.

Worked case — full trace

With α=0.8, a trace of value 1 becomes 0.8 then 0.64 without new writes. Correction β=0.5 then adds only half the observed error.

S₀ = [[2,3],[5,1]]   (state inherited from sessions 13-14)

── A. Decay only, α = 0.8, no write ───────────────────────────────
  t+1 : S = 0.8·S₀        = [[1.60, 2.40],[4.00, 0.80]]
  t+2 : S = 0.8·S         = [[1.28, 1.92],[3.20, 0.64]]
  a trace of value 1 follows 1 → 0.80 → 0.64 → 0.512 …
  half-life: ln(0.5)/ln(0.8) = 3.106 → ≈ 3.1 tokens   ✅ measurable bound

── B. Decay THEN correction, α=0.8 β=0.5, k=[1,0] v=[9,9] ─────────
  S′  = [[1.60, 2.40],[4.00, 0.80]]
  v̂   = S′ᵀk = [1.60, 2.40]
  e   = [9−1.60, 9−2.40] = [7.40, 6.60]
  β k eᵀ = [[3.70, 3.30],[0, 0]]
  S₁  = [[5.30, 5.70],[4.00, 0.80]]
  read q=[1,0] → [5.30, 5.70]   ❌ not [9,9]
  residual error = [3.70, 3.30] = exactly (1−β)·e   ✅ predictable
  row 2: never written, yet decayed by 0.8 → forgetting is GLOBAL

── C. ❌ Unbounded gate, α = 1.05 ──────────────────────────────────
  1.05⁴⁰ ≈ 7.0   ;   1.05¹⁰⁰ ≈ 131.5
  with no write at all the state diverges: bounding α to [0,1] is not
  cosmetic, it is what keeps the recurrence stable over 100,000 tokens

── D. ❌ Reversed order: correct THEN decay ────────────────────────
  S = 0.8·([[2,3],[5,1]] + 0.5·k(v−S₀ᵀk)ᵀ) = 0.8·[[5.5, 6.0],[5,1]]
    = [[4.40, 4.80],[4.00, 0.80]]  ≠  [[5.30, 5.70],[4.00, 0.80]]
  the fresh correction suffers a decay it never lived through: same
  formula, different model. Ordering is an architectural decision.

UNIT CHECK: α and β are UNITLESS scalars; e carries the units of v;
so α·S and βk eᵀ are both homogeneous with S. A negative α would flip
the sign of memory each step — hence the bound, not merely |α|≤1.

Decay-gate regimes and the useful lifetime of memory

α_t Half-life (tokens) Behaviour and risk
0.99 ≈ 69 near-retention: accumulated interference, saturation
0.80 ≈ 3.1 fast forgetting: good for local context, drops the opening
0.50 1.0 working memory: barely holds more than the previous token
1.05 none (divergence) ×7 over 40 tokens, ×131 over 100: NaN nearly certain

Causal lab

Predict → change one variable → run → explain the delta

/interactives/curriculum/chunk-gate-memory.html?lang=en

Common errors

“Forgetting loses information, so α should be kept as close to 1 as possible.”

At α=0.99 the half-life is ≈ 69 tokens: old associations stay and superpose on new ones — precisely the session-13 interference. Decay frees capacity; it does not destroy an otherwise infinite memory.

“Gated DeltaNet is just Mamba under another name.”

The parallel is conceptual: recurrent state plus input-dependent controls. But the v̂ = S′ᵀk then e = v − v̂ step assumes an associative read of the state, which “state-space model” does not entail. Same family of motivations, distinct equations.

Boundary, evidence, and sources

The Mamba parallel is conceptual. It does not imply Gated DeltaNet and Mamba are interchangeable.

Evidence status: Established: the delta rule, exponential decay, and the half-lives computed here. Pedagogical: the α, β values and the 2×2 matrices of the trace. Reported: the exact gate choices of named models.

  • Owner-supplied bilingual course packet, Chapter 11.
  • Yang, Kautz & Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule”, ICLR (2025), arXiv:2412.06464.
  • Gu & Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”, arXiv:2312.00752 (2023).
  • Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Transfer challenge

A support assistant must retain the customer’s name — stated once around token 20 — across roughly 2,000 tokens of conversation.

  1. With a constant α, what value keeps ≥ 50% of the trace at 2,000 tokens? (solve α²⁰⁰⁰ = 0.5)
  2. Is that α livable for the rest of the memory? Cite the regime from the table.
  3. Propose the learned-gate solution: what must α_t and β_t do at the key moments?

Synthesis and exit ticket

  • Why forget
  • Decay gate
  • Write gate
  • Stability
  • Effective capacity
  • Connection to Mamba

Ticket: mechanism · trace · observation · boundary · evidence · next experiment

Instructor notes: Frame the problem before naming the mechanism. Collect an initial prediction and retain it for the exit ticket.

Instructor notes: Connect every step to the next with a causal verb. Flag any merely decorative arrow.

Instructor notes: Open with the lived example before any mechanism: “your contact moved house — what do you do with the old address?”. The room answers “forget it” without hesitating; note that nobody counts that as a loss.

Instructor notes: Do not present forgetting as a technical constraint. First ask for a lived example where remembering everything would hurt (a change of address, a correction in an email thread). The mechanism then justifies itself.

Instructor notes: Expected answer: the new topic inherits associations from the old one — e.g. a departed character resurfacing in generation; the corrective mechanism is the decay gate α. Expected wrong answer: invoking the delta rule alone — it corrects one key at a time, not global accumulation.

Instructor notes: Ask: “should forgetting run at the same speed in a text message and in a contract?”. The unanimous no justifies the LEARNED gate before its formula is even written.

Instructor notes: Have them punch 0.8ⁿ into a calculator until it drops below 0.1, then introduce half-life as the shortcut. The reverse order — formula first — leaves no memory trace.

Instructor notes: Answer: 0.8ⁿ < 0.1 → n = 11 tokens since the write; half-life = ln(0.5)/ln(0.8) = 3.106. Expected wrong answer: linear reasoning (“two more double-steps”) on a geometric process — point back to the visual-support curve.

Instructor notes: Have the 0.10 point checked on a calculator (0.8¹¹ ≈ 0.086), then re-sketch the curve mentally for α = 0.99: it goes nearly flat — the regime table’s “near-retention” row.

Instructor notes: Have two real tokens found: one that deserves no write (“the”) and one that demands it (“actually, the price moved to 9”). The two examples serve as the β≈0 and β=1 anchors for the whole beat.

Instructor notes: Set β=0 and β=1 on the same example and have them find a real corpus token for each. As long as β stays an abstract number, “gate” is just a word.

Instructor notes: Answer: β=0 → S = S′, no write (redundant token, “the”); β=1 → row 1 = [1.6+7.4, 2.4+6.6] = [9, 9] exactly (critical correction). Expected wrong answer: applying the correction to row 2 as well — k = [1,0] does not select it.

Instructor notes: Announce the number before the explanation: “5% off on α, and the state is multiplied by 131 within 100 tokens”. Let the enormity land, then ask where it comes from — exponential geometry does the work.

Instructor notes: Write both operation orders on the board without saying which is right, have both computed, then reveal the 5.30 versus 4.40 gap. That is when “implementation detail” stops being an excuse.

Instructor notes: Answer: only the ORDER changes — decay-then-correct versus correct-then-decay; in the second, the fresh correction suffers a decay it never lived through. Without the order, “S = αS + βk eᵀ” says neither what the read sees (S or S′) nor who undergoes α: the numbers are irreproducible. Expected wrong answer: “both eventually converge”.

Instructor notes: Point out that e′ is the error measured on the decayed state S′, e the one measured on S₀: the two orders also change what the read sees. Have the difference located in a real pseudo-code listing.

Instructor notes: Write “S: 2×2 yesterday, 2×2 today” on the board and ask what could possibly have changed. The apparent paradox (same size, different memory) is the beat’s hook — do not resolve it too fast.

Instructor notes: Have them translate a long-context marketing claim into an estimated half-life. Short exercise, lasting effect: they will never read a “context length” announcement the same way.

Instructor notes: Answer: the useful lifetime changed, not the physical capacity — half-life 69 versus 3.1 tokens. Expected restatement: “the context advertises 1 M tokens, but an unrewritten trace survives ~69 tokens at α=0.99”. Expected wrong answer: answering in gigabytes.

Instructor notes: Have a real use case placed on the ruler (short dialogue, report summary, codebase) before giving the reading. Each placement forces choosing a half-life — that is the exercise.

Instructor notes: Show both marketing pitches (SSM / delta) side by side with identical words highlighted. Ask: “can you choose on this basis?”. The collective no motivates hunting for the equation-level difference.

Instructor notes: On Mamba, resist the urge to unify. Ask explicitly “what does one do that the other does not?” and accept a collective “we don’t know” rather than a false analogy they will go on repeating.

Instructor notes: Answer: the associative read of the state followed by writing the error — a bare “recurrent state + gate” also describes a filter that never reads its memory. Expected wrong answer: answering with a reported performance gap instead of a mechanism difference.

Instructor notes: Walk line by line. Locate an inconsistency at the first faulty step, not only on the final line.

Instructor notes: Have learners fill the final row before revealing it: that trade-off is what decides in production.

Instructor notes: Retain initial and final values. Do not allow simultaneous changes that make the delta impossible to attribute.

Instructor notes: For each claim, have the group produce the smallest counterexample before giving the correction.

Instructor notes: Separate verifiable mechanism, reported implementation choice, and experimental result. Evidence precision must match claim precision.

Instructor notes: Answers: (1) α = 0.5^(1/2000) ≈ 0.99965 — practically 1; (2) no: that is the table’s “near-retention” regime — everything else accumulates too, interference and saturation guaranteed; (3) learned gates: α_t near 1 by default, lowered at topic changes, and β_t high whenever the name reappears — the real answer is “rewrite what must last”, not “never forget”. Misconception to harvest: hunting for THE perfect α. Ten minutes, pairs.

Instructor notes: Rebuild the chain without looking at the slides, then fill the ticket in at most six lines. Compare with the opening prediction and name what actually changed.