Exercises with solutions — Synthesis: a hybrid long-context architecture

General instruction: every answer must show data, transformation, result, one check, and one limitation. A bare number or copied definition is insufficient.

Starting data: Bounded design: periodic exact-attention layers for faithful retrieval, delta layers between them for fixed state, MLA to reduce per-token cache, MoE for conditional capacity, and spaced depth checkpoints. Verdict depends on quality/latency/memory measurements on the real task.

Boundary to retain: There is no universally best assembly. A diagram is a hypothesis; only controlled tests, hardware profiles, and user evaluations establish value.

Exercise 1 — Causal trace — Before inference

Reproduce and annotate the chain input → state → output. Remove MLA from the hybrid and replace it with a full KV cache. Trace the first effect on memory, retrieval fidelity, and latency without assuming a universal winner.

Deliverable: a component → retained information → transformation → output table, followed by the before/after removal trace.

Worked solution

Bounded design: periodic exact-attention layers for faithful retrieval, delta layers between them for fixed state, MLA to reduce per-token cache, MoE for conditional capacity, and spaced depth checkpoints. Verdict depends on quality/latency/memory measurements on the real task.

Solved variant: Per-token cache grows because keys and values are no longer compressed. Retrieval may retain more detail, but memory traffic and capacity rise. The verdict depends on quality, latency, and memory measurements on the actual workload.

The sort: pre-training learns then FREEZES embeddings, projections, routers, and gates. At inference those parameters are applied; only the context, the MLA cache, the delta state S, and the checkpoints evolve — working state, zeroed at the next request. That medium is the residual stream: text becomes tokens then embeddings, and one d_model = 4096 vector per position crosses the 48 blocks, each ADDING its contribution (session 20). The stream is the whole architecture’s data bus. The minimum check confirms that every arrow carries named information and that the claimed output actually depends on the removed component. If the prediction fails, locate the first missing dependency.

Rubric Exercise 1 — /10

Criterion	Points
Explicit components, roles, and assumptions	2
Traceable causal chain	3
Prediction before removal	2
Interpretation and check	2
Named limitation	1

Exercise 2 — Diagnose a seductive explanation — Sequence mixing

A colleague claims: « Sequence mixing proves the system will be accurate, fast, and stable in every context. »

Separate mechanism, assumption, observation, and conclusion.
Name two correct lesson elements and two unsupported extrapolations.
Propose a bounded experiment with controlled variable, metric, and stop threshold.
Rewrite the claim as one defensible sentence.

Reasoned solution

The hybrid pattern: three delta layers (fixed state, length nearly free) then one exact layer under MLA (faithful recall, reduced cache). The trace’s result: 1.5 GiB of cache and 1.1 MiB of state — ÷16 on the dominant line item. Each block routes its tokens: top-2 of 64 experts plus one shared — 26B resident parameters, 1.2B active per token. In the global budget, MoE does not appear in context memory: it lives in the weight budget and in all-to-all latency. There is no universally best assembly. A diagram is a hypothesis; only controlled tests, hardware profiles, and user evaluations establish value.

The claim mixes a local relation with a global guarantee. A defensible version states only the observed mechanism, test conditions, and measured metric. Stop the test if shapes become invalid, the metric crosses the declared degradation threshold, or another variable changes.

Rubric Exercise 2 — /10

2 points per element: separation, lesson grounding, extrapolations, protocol, and rewrite.

Exercise 3 — Architecture decision and transfer — Depth and output

Compare two architectures for the same long-context service. Option A is the worked hybrid: periodic exact attention, delta memory between layers, MLA, MoE, and depth-retrieval checkpoints. Option B is a dense baseline: exact attention and a dense FFN at every layer, an uncompressed KV cache, with no routing or depth retrieval. Build a decision record containing:

the workload and dominant constraint;
each option’s mechanism, without slogans;
one quality, memory, or latency prediction;
one case where your preferred procedure loses;
an A/B protocol, metrics, and rollback threshold;
a bounded verdict: choose, defer, or reject.

Elements of a strong solution

Spaced checkpoints — x₀, x₁₂, x₂₄, x₃₆: four here, session 20’s spacing of 12 extended over 48 layers — re-selected by a softmax mixture, then norm → W_vocab → next-token softmax. Prefill: parallel chunks (session 15) fill caches and states — a compute-bound regime. Decode: one token re-reads all available past, updates S and the cache — a memory-bandwidth-bound regime. Two profiles, one code. Mixed: established mechanisms + source-reported Kimi K3-style choices.

A strong answer does not make the newer mechanism the default winner. It retains a measurable baseline, sets thresholds before testing, and separates component cost from whole-system behavior. The verdict names what remains uncertain and the next evidence that could change it.

Rubric Exercise 3 — /15

Criterion	Points
Framing and baseline	3
Compared causal chains	4
Protocol and metrics	4
Rollback threshold	2
Bounded verdict	2

Extension

Repeat Exercise 3 after reversing the dominant constraint. If you optimized memory, impose a strict quality floor; if you optimized fidelity, halve the memory budget. Identify the first part of the verdict that changes and the evidence required.

Review before submission

Review the packet as if another group had to reproduce it without speaking to you. Are all starting values or assumptions present? Are shapes or roles stated before operations? Does the prediction truly precede the observation? Is the result translated into behavior rather than left as an isolated number? Did you test a boundary value and identify a stop condition? Does the procedure or architecture choice retain a measurable baseline and a rollback threshold set before the test? Finally, highlight one sentence that states what is established, one that remains a hypothesis, and one measurement that could change your verdict. If any element is missing, the work is not reproducible.

Reference appendix for correction

Chapter 16 — Putting the Kimi K3-style system together

16.1 The backbone as a team

Goal: combine the pieces without pretending they all do the same job. The supplied architecture narrative describes a hybrid system using token representations, sequence-mixing mechanisms, feed-forward or expert computation, residual pathways, normalization and an output head.

Intuition: Think of a newsroom: archives remember, reporters retrieve context, specialists analyze, editors combine drafts, and the publisher chooses the next word.

Step by step

Tokenizer: converts text into token IDs.
Embedding: converts IDs into vectors.
KDA or another sequence mixer: brings information from earlier tokens into the current representation.
MLA or exact-attention components, if present: preserve more token-specific access in selected places.
MoE/feed-forward component: transforms each token through selected specialists.
Residual and normalization paths: stabilize and combine updates.
Output projection: creates one logit per vocabulary token; softmax converts logits to probabilities.

Worked example: No single component is ‘the intelligence.’ Capability emerges from trained interactions among representations, memory, routing, nonlinear transformations and a large dataset/objective.

Why it matters: The architectural lesson is division of labor under hardware constraints.

Quick check: Which component converts final hidden vectors to vocabulary scores? Answer: the output projection or language-model head.

16.2 A token journey during prefill

Goal: follow a prompt through the model. Prefill means processing the prompt tokens before generating the first new token.

Intuition: It is like reading all pages supplied with an exam before writing the first answer.

Step by step

Tokenize the prompt and look up embeddings.
Process many prompt positions in parallel where causality permits.
In chunkwise recurrent layers, use large matrix operations inside chunks and carry state between chunks.
In cached-attention layers, construct compressed or full per-token cache entries.
Route token representations through selected experts.
Produce logits at each training position or at the final prompt position for inference.

Worked example: Example: for 256 prompt tokens and chunk size 64, there are four recurrent chunks. Hardware can process much of each chunk as matrix batches instead of 256 tiny isolated loops.

Why it matters: Prefill performance often depends heavily on parallel compute and memory bandwidth.

Quick check: Does prefill generate the whole answer at once? Answer: No. It prepares states/caches; decoding then generates new tokens autoregressively.

16.3 A token journey during decode

Goal: follow one newly generated token. Decode means generating tokens one at a time after prefill.

Intuition: Write one word, reread the necessary notes, then choose the next word.

Step by step

Embed the most recent token.
At each recurrent layer, read the fixed-size state, compute the layer output and update the state.
At each cached-attention layer, compare the current query with cached token records.
Run routed experts and residual/depth pathways.
Project to logits, obtain a probability distribution, and select the next token.
Append the token and repeat until a stopping condition.

Worked example: A fixed recurrent state has constant shape per layer during decode. A per-token cache grows as more tokens are generated. A hybrid model inherits both behaviors in the layers where they occur.

Why it matters: Decode speed depends on active parameters, cache/state traffic, routing communication, kernels and hardware—not only on theoretical operation counts.

Quick check: Why is generation called autoregressive? Answer: each newly chosen token becomes part of the input used to choose the next token.

16.4 Final mental model

Goal: compress the whole course into one chain: text → tokens → vectors → layers → memory/context mixing → specialists → logits → next token.

Intuition: The model is a very large learned numerical machine. It does not store sentences as little files; it transforms vectors using learned matrices and temporary states.

Step by step

Exact attention asks individual past records.
Linear/KDA-style memory asks a compressed, correctable running state.
MLA compresses per-token records.
MoE activates selected specialists.
AttnRes-style paths can retrieve useful earlier depth representations.
The output head predicts the next token.

Worked example: The best design is not the one with the fanciest name. It is the one that reaches the required quality, latency, memory use, training stability and cost on real hardware.

Why it matters: You now have enough foundations to read architecture diagrams critically: define every object, check shapes, follow information flow and label evidence strength.

Quick check: What four questions should you ask about any new mechanism? Answer: What problem does it solve? What information does it store or transform? What are the tensor shapes? What evidence supports the claimed benefit?

Sources and evidence boundary

Owner-supplied bilingual course packet, Chapter 16.
Vaswani et al., “Attention Is All You Need”, NeurIPS (2017).
DeepSeek-AI, “DeepSeek-V2” (Multi-head Latent Attention), arXiv:2405.04434 (2024).
Yang, Kautz & Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule”, ICLR (2025).
Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Scope: Mixed: established mechanisms + source-reported Kimi K3-style choices. These references support the session frame; they do not turn a reported product choice into an independently verified result.