1. Why forget

The problem: A document changes topic at token 500; the delta memory keeps everything. The old topic’s associations keep answering the new topic’s queries — session 13’s superposition returns, not through nearby keys but through sheer accumulation over time.

The idea: Reverse the reflex: forgetting is a function, not a failure. A fixed-capacity memory that never frees anything eventually distinguishes nothing; erasing the stale is exactly what keeps the present readable.

Why / at what price: Forgetting frees capacity and bounds the past’s influence. The price, obvious but real: what has decayed is gone — a fact stated at token 10 and needed at token 10,000 must have been rewritten in between, or it no longer exists.

2. Decay gate

The problem: Fine, forget — but how fast? A fixed rate would be wrong everywhere: a dialogue wants to hold 50 tokens, a contract 5,000. Forgetting must be driven by content, not by a global constant.

The idea: A gate α_t ∈ [0,1], produced at every token, multiplies the state before the write: S′ = α_t·S_{t−1}. At 0.8 a trace follows 1 → 0.8 → 0.64, and its half-life is ln(0.5)/ln(0.8) ≈ 3.1 tokens. The model learns when to tighten or open the gate.

S_t=α_t S_{t−1}+update_t

Why / at what price: Forgetting becomes measurable — a half-life — and contextual. The price: decay is global. Row 2 of the trace, never rewritten, decays anyway; whatever is not regularly rewritten evaporates, relevant or not.

3. Write gate

The problem: Symmetrically: does every token deserve to write? A redundant “the” writing at full strength wastes capacity and pushes the state toward interference; a critical correction written timidly corrects nothing.

The idea: β_t doses the delta correction: S_t = S′ + β_t k eᵀ. In the trace, β = 0.5 applies only half the error — exactly (1−β)·e remains. β ≈ 0 ignores, β = 1 fully corrects, decided token by token.

Why / at what price: Two independent gates: α decides what stays, β what enters — fine-grained, and learned. The price: two settings that can compensate each other; diagnosing “memory too short” requires separating α’s effect from β’s.

4. Stability

The problem: These recurrences run 100,000 times in a row. A gate at 1.05 — 5% too much — multiplies the state by 7 in 40 tokens, by 131 in 100. And the same code with correct-then-decay ordering produces different numbers: two models for one formula.

The idea: Stability rests on binding details: α bounded in [0,1] — not merely |α| ≤ 1, a negative α would flip the sign each step —, normalized keys, and a declared operation order: decay, read, correct.

Why / at what price: These constraints make the recurrence safe at arbitrary length. The price: nothing is cosmetic — an implementation writing S = αS + βk eᵀ without stating the order is irreproducible. The details are the architecture.

5. Effective capacity

The problem: S is still 2×2 — yet α = 0.99 and α = 0.80 yield radically different memories. If the physical size never moves, what exactly does this model’s “memory” measure?

The idea: The right quantity is useful lifetime: half-life ≈ 69 tokens at α = 0.99, ≈ 3.1 at α = 0.80, 1.0 at α = 0.5. Effective capacity combines S’s size, forgetting, and key separation — not the advertised maximum context length.

Why / at what price: Half-life gives an honest language for comparing memories — and for reading announcements: “1 M token context” says nothing about what survives 1,000 steps. The price: it is an average; actual retention depends on the learned gates, hence on content.

6. Connection to Mamba

The problem: Two families — SSMs (Mamba) and gated delta memories — advertise the same pitch: fixed state, input-dependent gates, long context. Should you conclude they are interchangeable, and benchmark one to choose the other?

The idea: The overlap is real: recurrence plus learned controls. So is the difference: the delta memory READS its state (v̂ = S′ᵀk) and writes the error; a classical SSM filters the signal with no associative read. Same family of motivations, distinct equations.

Why / at what price: The parallel guides intuition and tool transfer (chunking, gates). The price of the easy analogy: non-transferable conclusions — a Mamba result proves nothing about Gated DeltaNet, and vice versa. Comparing requires both implementations, same task, same budget.