Synthesis: a hybrid long-context architecture

Assemble training, Q/K/V, memories, caches, experts, and residual paths into complete prefill and decode journeys.

Applied AI · advanced · Session 21

Mechanism map

BEFORE: pre-training freezes embeddings, W_Q/K/V, routers, gates
════════════════════════════════════════════════════════════════
 text ─▶ tokens ─▶ embeddings ─▶ RESIDUAL STREAM x₀
                                        │
 ┌────────────────────────────────────────▼──────────────────────┐
 │ BLOCK ×48   (pattern: 3 delta layers, then 1 exact)           │
 │  ┌───────────────┐         ┌───────────────┐                  │
 │  │ SEQUENCE MIX  │  or     │ MLA ATTENTION │ ← c_t cache      │
 │  │ delta state S │         │ exact, Q/K/V  │   (1 KiB/token)  │
 │  │ (32 KiB fixed)│         └───────────────┘                  │
 │  └───────┬───────┘                 │                          │
 │          └────────────┬────────────┘                          │
 │                       ▼  ⊕ residual                           │
 │              ┌────────────────┐                               │
 │              │ top-2 ROUTER   │──▶ E_i, E_j (+ shared)        │
 │              └────────┬───────┘    capacity C, all-to-all     │
 │                       ▼  ⊕ residual                           │
 └───────────────────────┬───────────────────────────────────────┘
        checkpoints x₀,x₁₂,x₂₄,x₃₆ ──▶ depth mixture (softmax)
                         ▼
               norm ─▶ W_vocab ─▶ softmax ─▶ next token
 PREFILL: parallel chunks fill caches + states
 DECODE: 1 token, reads the past, updates S and the cache, loops

The problem — Before inference

Twelve sessions of mechanisms, and one trap question in a meeting: “so the model learns while we talk to it?”. Without a clean sort between what is learned and what is written, everything that follows will be misread.

The idea — Before inference

The sort: pre-training learns then FREEZES embeddings, projections, routers, and gates. At inference those parameters are applied; only the context, the MLA cache, the delta state S, and the checkpoints evolve — working state, zeroed at the next request.

Why / at what price — Before inference

This split makes the system analyzable: every behavior traces either to weights (offline) or to state (this request). The price: no memory across conversations without external machinery — a design choice, not an oversight.

Check: The router, W_DKV and the gates are frozen at inference; only the MLA cache, the delta state S and the checkpoints evolve. Sort those six objects into “learned offline” and “written during the request”, then say which one resets between two conversations.

The problem — Input and representation

48 heterogeneous blocks — delta, MLA, MoE, retrieval — must cooperate without knowing each other. You need a shared medium each one reads and enriches, or the assembly is just a pile of incompatible modules.

The idea — Input and representation

That medium is the residual stream: text becomes tokens then embeddings, and one d_model = 4096 vector per position crosses the 48 blocks, each ADDING its contribution (session 20). The stream is the whole architecture’s data bus.

Why / at what price — Input and representation

A single interface contract — this is what makes the hybrid composable. The familiar price: roughly 96 additions (two per block) dilute early contributions; session 20’s question — what remains of x₀? — returns here at system scale.

Check: The residual stream carries one 4096-vector per position through 48 blocks. How many additions does that vector undergo before W_vocab in this design, and which quantity from session 20 should that make you worry about?

The problem — Sequence mixing

The dominant memory line item comes from sequence mixing: all-exact = 24 GiB at 131,072 tokens (session 18); all-delta = 1 MiB but approximate recall (sessions 13-16). Neither pure option is livable. How do you dose?

The idea — Sequence mixing

The hybrid pattern: three delta layers (fixed state, length nearly free) then one exact layer under MLA (faithful recall, reduced cache). The trace’s result: 1.5 GiB of cache and 1.1 MiB of state — ÷16 on the dominant line item.

Why / at what price — Sequence mixing

Each layer gets the mechanism suited to its role. The price: the RATIO becomes an architecture hyperparameter — 1 in 4? 1 in 2? — paid in GiB and justified by recall quality: a choice to test, not to declare.

Check: The pattern is “3 delta layers then 1 exact”, i.e. 12 exact out of 48. Recompute the cache for a “1 in 2” pattern: which line item moves from 1.5 GiB to what, and what quality gain would justify that doubling?

Visual support — Sequence mixing

pattern “3 delta, 1 exact” repeated 12 times over 48 layers

D D D E   D D D E   D D D E   …        E = exact under MLA
                                       D = delta, fixed state
cache: 12 E × 1 KiB/token = 12 KiB/token → 1.5 GiB at 131,072 t
state: 36 D × 32 KiB      ≈ 1.1 MiB      → independent of n

all E (48): 24 GiB   ‖   all D: ≈ 1.5 MiB, approximate recall
the E/D ratio is the fidelity ↔ memory slider

The problem — Experts

FFN capacity must grow without every token paying for the whole block — session 19 posed the problem, and its trap: confusing active parameters with speed.

The idea — Experts

Each block routes its tokens: top-2 of 64 experts plus one shared — 26B resident parameters, 1.2B active per token. In the global budget, MoE does not appear in context memory: it lives in the weight budget and in all-to-all latency.

Why / at what price — Experts

Massive conditional capacity. The price: three separate budgets to hold — weights (26B), compute (1.2B), network (all-to-all, capacity C) — and the 21× ratio governs only one of them.

Check: The table announces 26B in weights for 1.2B active per token. Does that 21× ratio appear anywhere in the trace’s memory budget, and if not, in which separate budget must it be entered?

The problem — Depth and output

After 48 blocks, early representations are diluted (session 20) — and the output, W_vocab, sometimes needs them. How do you give access to the depth past without re-exploding the budget MLA just compressed?

The idea — Depth and output

Spaced checkpoints — x₀, x₁₂, x₂₄, x₃₆: four here, session 20’s spacing of 12 extended over 48 layers — re-selected by a softmax mixture, then norm → W_vocab → next-token softmax.

Why / at what price — Depth and output

Retrieval becomes a choice. The price, discovered in the trace: unbounded, the checkpoints cost 4 GiB — 73% of the total budget — and reintroduce the eliminated O(n); windowed to 8,192 tokens, 256 MiB. The last component added can dominate the whole bill.

Check: The 4 depth checkpoints cost an unbounded 4 GiB, i.e. 73% of the budget, versus 256 MiB windowed to 8,192 tokens. What does the window make unretrievable, and for which task would that concession be unacceptable?

Visual support — Depth and output

memory budget at 131,072 tokens — the trace’s hybrid

MLA cache (12 E)      ██████ 1.5 GiB
delta state (36 D)    ▏ ≈ 1.1 MiB
raw checkpoints       ████████████████ 4 GiB   ← 73% of total
  → windowed 8,192 t  █ 256 MiB

revised total ≈ 1.76 GiB   (all-exact baseline: 24 GiB, ÷13.6)
the last component added was dominating the bill

The problem — Prefill then decode

The same assembly must swallow a whole prompt then generate token by token. Measure only one regime and the design verdict is wrong for the other — prefill and decode do not saturate the same resources.

The idea — Prefill then decode

Prefill: parallel chunks (session 15) fill caches and states — a compute-bound regime. Decode: one token re-reads all available past, updates S and the cache — a memory-bandwidth-bound regime. Two profiles, one code.

Why / at what price — Prefill then decode

Reading both regimes separates first-token latency from generation throughput. The methodological price: any verdict demands measurements — quality, per-regime latency, memory — on the real task; a diagram, however coherent, remains a hypothesis.

Check: Prefill handles 2048-position chunks, decode a single one. The same code goes from FLOP-bound to bandwidth-bound: which of the table’s four line items dominates each regime, and which would you measure first to deliver the design verdict?

Visual support — Prefill then decode

PREFILL (once per request)         DECODE (at every token)
──────────────────────────────     ─────────────────────────────
2048-position chunks in parallel   1 position
fills cache + states               re-reads cache + states, writes 1
bound by: FLOPs                    bound by: memory bandwidth
metric: first-token latency        metric: tokens/second

   a design verdict must cite BOTH columns

Worked case — full trace

Bounded design: periodic exact-attention layers for faithful retrieval, delta layers between them for fixed state, MLA to reduce per-token cache, MoE for conditional capacity, and spaced depth checkpoints. Verdict depends on quality/latency/memory measurements on the real task.

BOUNDED DESIGN: 48 layers, 1 exact in 4 (12 exact, 36 delta),
MLA (c divides the per-token term by 4), MoE top-2 + 1 shared,
depth checkpoints every 12. Context n = 131,072 tokens.

BASELINE — 48 full-attention layers, 8 KV heads, d_head=128, BF16
  per token/layer = 2 × 8 × 128 × 2 = 4,096 B = 4 KiB
  48 layers → 192 KiB/token
  × 131,072 tokens = 24 GiB                                   ❌ untenable

HYBRID — line by line
  12 exact layers under MLA: 1 KiB/token/layer → 12 KiB/token
    × 131,072 = 1,572,864 KiB = 1,536 MiB = 1.5 GiB           ✅ ÷16
  36 delta layers, fixed state 128×128×2 = 32 KiB/layer
    36 × 32 KiB = 1,152 KiB ≈ 1.1 MiB, independent of n       ✅
  4 checkpoints, d_model = 4096, BF16, over the whole context
    4 × 4096 × 2 B × 131,072 = 4,294,967,296 B = 4 GiB        ❌
  TOTAL = 1.5 + 0.001 + 4 = 5.5 GiB
  → the "depth" line item, added last, is 73% of the budget and
    reintroduces exactly the O(n) growth MLA had just divided away.
    Bounded fix: checkpoints over the recent window (8,192 tokens)
    → 4 × 4096 × 2 × 8,192 = 256 MiB                           ✅
  REVISED TOTAL = 1.5 GiB + 1.1 MiB + 256 MiB ≈ 1.76 GiB (÷13.6 vs baseline)

COMPUTE PER DECODE STEP
  FFN: 64 experts × 0.4B, active (2 + 1) × 0.4 = 1.2B out of 26B
  but latency is paced by the saturated expert and the all-to-all,
  not by the 21× ratio.                                        ❌ trap

UNIT AND SHAPE CHECK
  1 GiB = 1,024 MiB; 4 KiB ÷ 4 = 1 KiB (MLA), not 4 MiB.
  Prefill: [chunk=2048, d_model] in parallel → writes 2048 cache entries.
  Decode: [1, d_model] → writes 1 entry. Same code, batch 2048 versus 1:
  decode is memory-bandwidth bound, not FLOP bound.

Memory budget by line item at 131,072 tokens, and what it buys

Line item Cost at n = 131,072 What it buys / its risk
12 exact layers + MLA 1.5 GiB, O(n) faithful recall of distant tokens; still linear
36 delta layers ≈ 1.1 MiB, constant length nearly free; interference between writes
MoE top-2 + shared 26B in weights (≈ 52 GiB BF16), 1.2B active conditional capacity; all-to-all latency and drops
4 depth checkpoints 4 GiB → 256 MiB if windowed chosen retrieval; reintroduces O(n) if unbounded

Causal lab

Predict → change one variable → run → explain the delta

/interactives/curriculum/cache-mla-comparison.html?lang=en

Common errors

“This hybrid architecture is the right one: it combines the best of every mechanism.”

The trace shows the opposite of stacked benefits: MLA divides the cache by 16, then the depth checkpoints add 4 GiB and reintroduce the very O(n) just eliminated — 73% of the budget for the last component added. A diagram is a hypothesis; only controlled quality/latency/memory tests on the real task decide. No assembly is universally best.

“The model learns during the conversation: the cache and the delta state are memory.”

Two different things share one word. Weights learned offline — embeddings, W_Q/K/V, W_DKV, router, experts — do not move by a single byte during inference. What changes is working state: MLA cache, state S, checkpoints, all zeroed at the next request. Prefill fills that state, decode reads and updates it; none of it is learning.

Boundary, evidence, and sources

There is no universally best assembly. A diagram is a hypothesis; only controlled tests, hardware profiles, and user evaluations establish value.

Evidence status: Mixed: established mechanisms + source-reported Kimi K3-style choices.

  • Owner-supplied bilingual course packet, Chapter 16.
  • Vaswani et al., “Attention Is All You Need”, NeurIPS (2017).
  • DeepSeek-AI, “DeepSeek-V2” (Multi-head Latent Attention), arXiv:2405.04434 (2024).
  • Yang, Kautz & Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule”, ICLR (2025).
  • Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Transfer challenge

Your team must serve a 1 M-token context on the same memory budget (~2 GiB per request).

  1. Revisit the table’s four line items: which ones explode at 1 M, which ones hold?
  2. Propose ONE change per exploding item, and name what it sacrifices.
  3. Deliver a bounded verdict: the three measurements that would decide, and your rollback threshold.

Synthesis and exit ticket

  • Before inference
  • Input and representation
  • Sequence mixing
  • Experts
  • Depth and output
  • Prefill then decode

Ticket: mechanism · trace · observation · boundary · evidence · next experiment

Instructor notes: Frame the problem before naming the mechanism. Collect an initial prediction and retain it for the exit ticket.

Instructor notes: Connect every step to the next with a causal verb. Flag any merely decorative arrow.

Instructor notes: Open with the trap question heard in meetings: “so it learns while we talk to it?”. Get hot-take answers from the room and keep them posted — the beat re-files every one of them.

Instructor notes: Open the synthesis with a two-column sort on the board, “frozen” and “written now”, and have the room place the six objects. Any leftover confusion here makes the rest of the session unintelligible.

Instructor notes: Answer: learned offline — router, W_DKV, gates; written during the request — MLA cache, state S, checkpoints; and all three reset between conversations. Expected wrong answer: filing the delta state under “learned” because it looks like weights — session 14 callback: fast weights.

Instructor notes: Ask: “which object crosses ALL the blocks?”, then have its size guessed (4096 per position) and its bus role named. If the word “residual” does not surface, thirty seconds of session-20 recall suffice.

Instructor notes: Have one token traced with a marker from text to the softmax output, one person per block of the diagram. The relay forces each learner to name what enters and what leaves their stage.

Instructor notes: Answer: roughly 96 additions — two per block, mixing then FFN/MoE. The quantity to fear: session 20’s dilution — early contributions’ share collapses, which is precisely what the depth checkpoints will correct. Expected wrong answer: 48, one addition per block.

Instructor notes: Vote on the pattern before any computation: all exact, all delta, or a mix? Each camp states its reason — the beat’s numbers then arbitrate (24 GiB / 1 MiB / 1.5 GiB).

Instructor notes: Vote on the exact/delta pattern before revealing the number: “1 in 4, 1 in 2, or all exact?” Then unveil 1.5 GiB versus 24 GiB. The vote turns a hyperparameter into an owned decision.

Instructor notes: Answer: 24 exact layers × 1 KiB = 24 KiB/token → 3 GiB at 131,072 tokens: the cache line item doubles (1.5 → 3 GiB), the other items do not move. Required justification: a MEASURED long-range recall gain on the real task — not “more exact must be better”. Expected wrong answer: recomputing the whole budget instead of the single affected item.

Instructor notes: Have the cache row recomputed for a 1-in-2 pattern (24 E → 3 GiB): that is exactly the check that follows. Stress that “all E” does not use MLA in the trace’s baseline — hence 24 GiB, not 6.

Instructor notes: Session-19 recall question: “the 21× ratio — what does it govern again?”. Expected: FLOPs, and only FLOPs. The beat then re-files MoE into its three budgets.

Instructor notes: Recall session 19 with a single question: “where is the MoE in this memory budget?” The absence of the 21× ratio from the table is the point; let the silence do the work.

Instructor notes: Answer: no — the 21× does not sit in context memory: the 26B live in the WEIGHT budget (≈ 52 GiB in BF16, the table row) and the 1.2B in the COMPUTE budget. Three budgets, three separate lines. Expected wrong answer: hunting for the 21× in the cache column.

Instructor notes: First present the checkpoints as session 20’s victory, then have their cost at 131,072 tokens computed BEFORE showing the 4 GiB. The punchline — 73% of the budget — must be discovered by the room.

Instructor notes: Present the 4 GiB of checkpoints as a win before having them compute its share of the total. Discovering the 73% yourself beats any warning about optimizations that cancel each other out.

Instructor notes: Answer: the window makes any depth state more than 8,192 tokens back unretrievable — unacceptable for a task that cites the start of a very long document (contract, codebase). Useful bridge: four checkpoints here (x₃₆ added) versus three in session 20 — same spacing of 12, deeper network. Expected wrong answer: confusing the token window with the layer selection.

Instructor notes: Have the 73% (4 / 5.5) and the ÷13.6 (24 / 1.76) verified on a calculator. Then ask: “which line item re-explodes if context goes to 1 M?” — the MLA cache, the only line still O(n).

Instructor notes: Ask: “your benchmark measures generation throughput — what does it miss?”. Expected: first-token latency, hence prefill. The visual support’s two columns structure the whole beat.

Instructor notes: Close by demanding a bounded verdict, phrased as three named measurements and one numeric threshold. Refuse “it depends” as a final answer: the session deliverable is a testable hypothesis, not an architecture.

Instructor notes: Answer: prefill — compute-dominated (chunked attention, MoE); decode — dominated by re-reading the MLA cache, hence bandwidth. First measurement for the verdict: decode throughput under growing length, where cache and all-to-all compound. Accept any argued answer that names ONE measurement per regime.

Instructor notes: Have three optimizations filed in the right column: chunking (prefill), MLA (mostly decode), cache quantization (decode). A misfiled optimization is a misread benchmark.

Instructor notes: Walk line by line. Locate an inconsistency at the first faulty step, not only on the final line.

Instructor notes: Have learners fill the final row before revealing it: that trade-off is what decides in production.

Instructor notes: Retain initial and final values. Do not allow simultaneous changes that make the delta impossible to attribute.

Instructor notes: For each claim, have the group produce the smallest counterexample before giving the correction.

Instructor notes: Separate verifiable mechanism, reported implementation choice, and experimental result. Evidence precision must match claim precision.

Instructor notes: Expected: (1) the MLA cache explodes — 12 KiB/token × 1,048,576 ≈ 12 GiB; windowed checkpoints hold (256 MiB, independent of n), the delta state holds (1.1 MiB), MoE weights do not move. (2) e.g. a 1-in-8 pattern (cache ÷2, sparser exact recall), a narrower c (tighter reconstruction), a window on exact attention (loses faithful distant recall). (3) measurements: needle-style long-context recall, decode throughput at 1 M, peak memory; threshold e.g. “recall < 95% of baseline → roll back”. Misconception to harvest: “keep everything and quantize” — quantization divides by 2 to 4, not by 8. Twelve minutes, groups of three.

Instructor notes: Rebuild the chain without looking at the slides, then fill the ticket in at most six lines. Compare with the opening prediction and name what actually changed.