Applied AI · advanced · Session 21
Teacher guide — Synthesis: a hybrid long-context architecture
← Back to courseFrançaisMarkdown source

Teacher guide — Synthesis: a hybrid long-context architecture

Duration: 120 minutes
Positioning: Assemble training, Q/K/V, memories, caches, experts, and residual paths into complete prefill and decode journeys.
Expected evidence: Mixed: established mechanisms + source-reported Kimi K3-style choices.

Observable outcomes and preparation

Before class, the instructor runs the worked case and lab, prints the exercise packet, prepares a four-column board—assumption, prediction, observation, delta—and checks that every mathematical notation is paired with dimensions. The demonstration must not become slide reading.

Diagnostic

  1. Explain in one sentence: Before inference. What observation would falsify your explanation?
  2. Explain in one sentence: Input and representation. What observation would falsify your explanation?
  3. Explain in one sentence: Sequence mixing. What observation would falsify your explanation?

Teaching decision: if two of three answers remain nominal or lack a validity condition, rebuild the vocabulary with a numerical example before any formula. A fluent but unfalsifiable answer does not count as mastery.

Timed plan

Time Activity Observable evidence
0–10 min Individual diagnostic, then pair comparison Three answers and one named uncertainty
10–25 min Situation and vocabulary Annotated input → state → output diagram
25–55 min Develop the mechanism on the board Shapes, assumptions, and intermediate calculation visible
55–75 min Worked case with deliberate errors Reasoned correction, not only the right number
75–95 min Causal lab: predict, change one variable, run Prediction / observation / delta table
95–112 min Exercises 1 and 2 with peer correction Retained artifact and applied rubric
112–120 min Exit ticket and transfer Mechanism, boundary, next experiment

Teaching notes

Chapter 16 — Putting the Kimi K3-style system together

16.1 The backbone as a team

Goal: combine the pieces without pretending they all do the same job. The supplied architecture narrative describes a hybrid system using token representations, sequence-mixing mechanisms, feed-forward or expert computation, residual pathways, normalization and an output head.

Intuition: Think of a newsroom: archives remember, reporters retrieve context, specialists analyze, editors combine drafts, and the publisher chooses the next word.

Step by step

Worked example: No single component is ‘the intelligence.’ Capability emerges from trained interactions among representations, memory, routing, nonlinear transformations and a large dataset/objective.

Why it matters: The architectural lesson is division of labor under hardware constraints.

Quick check: Which component converts final hidden vectors to vocabulary scores? Answer: the output projection or language-model head.

16.2 A token journey during prefill

Goal: follow a prompt through the model. Prefill means processing the prompt tokens before generating the first new token.

Intuition: It is like reading all pages supplied with an exam before writing the first answer.

Step by step

Worked example: Example: for 256 prompt tokens and chunk size 64, there are four recurrent chunks. Hardware can process much of each chunk as matrix batches instead of 256 tiny isolated loops.

Why it matters: Prefill performance often depends heavily on parallel compute and memory bandwidth.

Quick check: Does prefill generate the whole answer at once? Answer: No. It prepares states/caches; decoding then generates new tokens autoregressively.

16.3 A token journey during decode

Goal: follow one newly generated token. Decode means generating tokens one at a time after prefill.

Intuition: Write one word, reread the necessary notes, then choose the next word.

Step by step

Worked example: A fixed recurrent state has constant shape per layer during decode. A per-token cache grows as more tokens are generated. A hybrid model inherits both behaviors in the layers where they occur.

Why it matters: Decode speed depends on active parameters, cache/state traffic, routing communication, kernels and hardware—not only on theoretical operation counts.

Quick check: Why is generation called autoregressive? Answer: each newly chosen token becomes part of the input used to choose the next token.

16.4 Final mental model

Goal: compress the whole course into one chain: text → tokens → vectors → layers → memory/context mixing → specialists → logits → next token.

Intuition: The model is a very large learned numerical machine. It does not store sentences as little files; it transforms vectors using learned matrices and temporary states.

Step by step

Worked example: The best design is not the one with the fanciest name. It is the one that reaches the required quality, latency, memory use, training stability and cost on real hardware.

Why it matters: You now have enough foundations to read architecture diagrams critically: define every object, check shapes, follow information flow and label evidence strength.

Quick check: What four questions should you ask about any new mechanism? Answer: What problem does it solve? What information does it store or transform? What are the tensor shapes? What evidence supports the claimed benefit?

Running the worked case

Bounded design: periodic exact-attention layers for faithful retrieval, delta layers between them for fixed state, MLA to reduce per-token cache, MoE for conditional capacity, and spaced depth checkpoints. Verdict depends on quality/latency/memory measurements on the real task.

Do not reveal the result at once. Ask learners to predict the next operation, its shape, and the expected sign. After each line ask: “What changed? What stayed fixed? Which assumption did we use?” A calculation error repaired with a causal chain is worth more than a guessed result.

Lab protocol

  1. Write a qualitative and, where possible, numerical prediction before touching a control.
  2. Change one variable only; retain a capture or record initial and final values.
  3. Explain the delta through the mechanism, not “the tool did that.”
  4. Test one boundary value and state where the model stops representing a real system.

Misconceptions

# Observable misconception Grounded correction Probe
1 “Before inference guarantees the outcome without assumptions or measurement.” The sort: pre-training learns then FREEZES embeddings, projections, routers, and gates. At inference those parameters are applied; only the context, the MLA cache, the delta state S, and the checkpoints evolve — working state, zeroed at the next request. Ask for a counterexample, then restate the mechanism with its validity condition.
2 “Input and representation guarantees the outcome without assumptions or measurement.” That medium is the residual stream: text becomes tokens then embeddings, and one d_model = 4096 vector per position crosses the 48 blocks, each ADDING its contribution (session 20). The stream is the whole architecture’s data bus. Ask for a counterexample, then restate the mechanism with its validity condition.
3 “Sequence mixing guarantees the outcome without assumptions or measurement.” The hybrid pattern: three delta layers (fixed state, length nearly free) then one exact layer under MLA (faithful recall, reduced cache). The trace’s result: 1.5 GiB of cache and 1.1 MiB of state — ÷16 on the dominant line item. Ask for a counterexample, then restate the mechanism with its validity condition.

Boundary to maintain: There is no universally best assembly. A diagram is a hypothesis; only controlled tests, hardware profiles, and user evaluations establish value.

Probing questions

  1. If we remove or reverse Before inference, which output changes first, and what observation would show it?
  2. If we remove or reverse Input and representation, which output changes first, and what observation would show it?
  3. If we remove or reverse Sequence mixing, which output changes first, and what observation would show it?
  4. If we remove or reverse Experts, which output changes first, and what observation would show it?
  5. If we remove or reverse Depth and output, which output changes first, and what observation would show it?
  6. If we remove or reverse Prefill then decode, which output changes first, and what observation would show it?

Assessment

Level Criterion
0 Repeats terms without connecting input, transformation, and output.
1 Describes the chain but checks neither shape nor assumption.
2 Executes the case, explains the result, and names one limitation.
3 Transfers to a new case, compares an alternative, and proposes a measurement that could invalidate the choice.

Exit threshold: level 2 on the worked case and at least one exercise; a memorized formula without interpretation remains level 1.

Observation and remediation protocol

During discussion, the instructor records evidence rather than impressions. Evidence of understanding contains a named object, a justified transformation, and a checkable consequence. If a learner gives the right result without a chain, ask for the preceding line. If the chain is coherent but the result is wrong, preserve the chain and isolate the arithmetic error. If vocabulary from another concept is used, compare both mechanisms in an input, state, output, cost, and boundary table. Remediation targets the first break only: vocabulary, shapes, operation, interpretation, or claim scope. After correction, use a neighboring case with a changed value; success on the same example does not prove transfer. For pair work, assign operator and verifier roles, then swap. The verifier does not supply the answer: they request an assumption, check the shape, and ask what observation could contradict the reasoning. The instructor retains the exit ticket and classifies the dominant break. The next session opens with a three-minute problem aimed at that break instead of repeating the whole lesson.

Differentiation

Post-session follow-up

Within twenty-four hours, return each annotated exit ticket with one priority, the exact resource to reopen, and a mini-case different from the worked case. Revision requires three items: a written prediction, a retained trace, and one sentence explaining the delta. At the next session, sample two submissions: one that repaired the break and one that remains ambiguous. Discuss them anonymously, then state the criterion that separates them. Do not use completion rate as evidence of mastery. Evidence of remediation is a correct chain on a new case with a stated boundary. If the same break appears in more than one third of the group, repair the support or demonstration before blaming learners.

Follow-up closes only when the new artifact shows the causal chain, check, and boundary—not merely when a file has been submitted.

Sources and evidence boundary

Scope: Mixed: established mechanisms + source-reported Kimi K3-style choices. These references support the session frame; they do not turn a reported product choice into an independently verified result.

Exit ticket

In no more than six lines: mechanism; calculation or trace; observation; boundary; evidence level; next experiment. The instructor marks one priority causal break for revision.