1. Before inference
The problem: Twelve sessions of mechanisms, and one trap question in a meeting: “so the model learns while we talk to it?”. Without a clean sort between what is learned and what is written, everything that follows will be misread.
The idea: The sort: pre-training learns then FREEZES embeddings, projections, routers, and gates. At inference those parameters are applied; only the context, the MLA cache, the delta state S, and the checkpoints evolve — working state, zeroed at the next request.
Why / at what price: This split makes the system analyzable: every behavior traces either to weights (offline) or to state (this request). The price: no memory across conversations without external machinery — a design choice, not an oversight.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
2. Input and representation
The problem: 48 heterogeneous blocks — delta, MLA, MoE, retrieval — must cooperate without knowing each other. You need a shared medium each one reads and enriches, or the assembly is just a pile of incompatible modules.
The idea: That medium is the residual stream: text becomes tokens then embeddings, and one d_model = 4096 vector per position crosses the 48 blocks, each ADDING its contribution (session 20). The stream is the whole architecture’s data bus.
Why / at what price: A single interface contract — this is what makes the hybrid composable. The familiar price: roughly 96 additions (two per block) dilute early contributions; session 20’s question — what remains of x₀? — returns here at system scale.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
3. Sequence mixing
The problem: The dominant memory line item comes from sequence mixing: all-exact = 24 GiB at 131,072 tokens (session 18); all-delta = 1 MiB but approximate recall (sessions 13-16). Neither pure option is livable. How do you dose?
The idea: The hybrid pattern: three delta layers (fixed state, length nearly free) then one exact layer under MLA (faithful recall, reduced cache). The trace’s result: 1.5 GiB of cache and 1.1 MiB of state — ÷16 on the dominant line item.
Why / at what price: Each layer gets the mechanism suited to its role. The price: the RATIO becomes an architecture hyperparameter — 1 in 4? 1 in 2? — paid in GiB and justified by recall quality: a choice to test, not to declare.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
4. Experts
The problem: FFN capacity must grow without every token paying for the whole block — session 19 posed the problem, and its trap: confusing active parameters with speed.
The idea: Each block routes its tokens: top-2 of 64 experts plus one shared — 26B resident parameters, 1.2B active per token. In the global budget, MoE does not appear in context memory: it lives in the weight budget and in all-to-all latency.
Why / at what price: Massive conditional capacity. The price: three separate budgets to hold — weights (26B), compute (1.2B), network (all-to-all, capacity C) — and the 21× ratio governs only one of them.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
5. Depth and output
The problem: After 48 blocks, early representations are diluted (session 20) — and the output, W_vocab, sometimes needs them. How do you give access to the depth past without re-exploding the budget MLA just compressed?
The idea: Spaced checkpoints — x₀, x₁₂, x₂₄, x₃₆: four here, session 20’s spacing of 12 extended over 48 layers — re-selected by a softmax mixture, then norm → W_vocab → next-token softmax.
Why / at what price: Retrieval becomes a choice. The price, discovered in the trace: unbounded, the checkpoints cost 4 GiB — 73% of the total budget — and reintroduce the eliminated O(n); windowed to 8,192 tokens, 256 MiB. The last component added can dominate the whole bill.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
6. Prefill then decode
The problem: The same assembly must swallow a whole prompt then generate token by token. Measure only one regime and the design verdict is wrong for the other — prefill and decode do not saturate the same resources.
The idea: Prefill: parallel chunks (session 15) fill caches and states — a compute-bound regime. Decode: one token re-reads all available past, updates S and the cache — a memory-bandwidth-bound regime. Two profiles, one code.
Why / at what price: Reading both regimes separates first-token latency from generation throughput. The methodological price: any verdict demands measurements — quality, per-regime latency, memory — on the real task; a diagram, however coherent, remains a hypothesis.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Development from the course source
Chapter 16 — Putting the Kimi K3-style system together
16.1 The backbone as a team
Goal: combine the pieces without pretending they all do the same job. The supplied architecture narrative describes a hybrid system using token representations, sequence-mixing mechanisms, feed-forward or expert computation, residual pathways, normalization and an output head.
Intuition: Think of a newsroom: archives remember, reporters retrieve context, specialists analyze, editors combine drafts, and the publisher chooses the next word.
Step by step
-
Tokenizer: converts text into token IDs.
-
Embedding: converts IDs into vectors.
-
KDA or another sequence mixer: brings information from earlier tokens into the current representation.
-
MLA or exact-attention components, if present: preserve more token-specific access in selected places.
-
MoE/feed-forward component: transforms each token through selected specialists.
-
Residual and normalization paths: stabilize and combine updates.
-
Output projection: creates one logit per vocabulary token; softmax converts logits to probabilities.
Worked example: No single component is ‘the intelligence.’ Capability emerges from trained interactions among representations, memory, routing, nonlinear transformations and a large dataset/objective.
Why it matters: The architectural lesson is division of labor under hardware constraints.
Quick check: Which component converts final hidden vectors to vocabulary scores? Answer: the output projection or language-model head.
16.2 A token journey during prefill
Goal: follow a prompt through the model. Prefill means processing the prompt tokens before generating the first new token.
Intuition: It is like reading all pages supplied with an exam before writing the first answer.
Step by step
-
Tokenize the prompt and look up embeddings.
-
Process many prompt positions in parallel where causality permits.
-
In chunkwise recurrent layers, use large matrix operations inside chunks and carry state between chunks.
-
In cached-attention layers, construct compressed or full per-token cache entries.
-
Route token representations through selected experts.
-
Produce logits at each training position or at the final prompt position for inference.
Worked example: Example: for 256 prompt tokens and chunk size 64, there are four recurrent chunks. Hardware can process much of each chunk as matrix batches instead of 256 tiny isolated loops.
Why it matters: Prefill performance often depends heavily on parallel compute and memory bandwidth.
Quick check: Does prefill generate the whole answer at once? Answer: No. It prepares states/caches; decoding then generates new tokens autoregressively.
16.3 A token journey during decode
Goal: follow one newly generated token. Decode means generating tokens one at a time after prefill.
Intuition: Write one word, reread the necessary notes, then choose the next word.
Step by step
-
Embed the most recent token.
-
At each recurrent layer, read the fixed-size state, compute the layer output and update the state.
-
At each cached-attention layer, compare the current query with cached token records.
-
Run routed experts and residual/depth pathways.
-
Project to logits, obtain a probability distribution, and select the next token.
-
Append the token and repeat until a stopping condition.
Worked example: A fixed recurrent state has constant shape per layer during decode. A per-token cache grows as more tokens are generated. A hybrid model inherits both behaviors in the layers where they occur.
Why it matters: Decode speed depends on active parameters, cache/state traffic, routing communication, kernels and hardware—not only on theoretical operation counts.
Quick check: Why is generation called autoregressive? Answer: each newly chosen token becomes part of the input used to choose the next token.
16.4 Final mental model
Goal: compress the whole course into one chain: text → tokens → vectors → layers → memory/context mixing → specialists → logits → next token.
Intuition: The model is a very large learned numerical machine. It does not store sentences as little files; it transforms vectors using learned matrices and temporary states.
Step by step
-
Exact attention asks individual past records.
-
Linear/KDA-style memory asks a compressed, correctable running state.
-
MLA compresses per-token records.
-
MoE activates selected specialists.
-
AttnRes-style paths can retrieve useful earlier depth representations.
-
The output head predicts the next token.
Worked example: The best design is not the one with the fanciest name. It is the one that reaches the required quality, latency, memory use, training stability and cost on real hardware.
Why it matters: You now have enough foundations to read architecture diagrams critically: define every object, check shapes, follow information flow and label evidence strength.
Quick check: What four questions should you ask about any new mechanism? Answer: What problem does it solve? What information does it store or transform? What are the tensor shapes? What evidence supports the claimed benefit?
Complete worked case
Bounded design: periodic exact-attention layers for faithful retrieval, delta layers between them for fixed state, MLA to reduce per-token cache, MoE for conditional capacity, and spaced depth checkpoints. Verdict depends on quality/latency/memory measurements on the real task.
Reading method: write the data, state every object shape, perform one transformation, and interpret the result before continuing.