Teacher guide — The complete pre-training pipeline
Duration: 120 minutes
Positioning: Connect raw data, filtering, tokenization, batches, targets, loss, updates, and the base model.
Expected evidence: Established mechanisms; numerical simplifications are pedagogical.
Observable outcomes and preparation
- Trace one example from raw data to a batch.
- Build one-token-shifted targets.
- Separate pre-training, post-training, and evaluation.
Before class, the instructor runs the worked case and lab, prints the exercise packet, prepares a four-column board—assumption, prediction, observation, delta—and checks that every mathematical notation is paired with dimensions. The demonstration must not become slide reading.
Diagnostic
- Explain in one sentence: Collect without accepting everything. What observation would falsify your explanation?
- Explain in one sentence: Tokenize. What observation would falsify your explanation?
- Explain in one sentence: Build inputs and targets. What observation would falsify your explanation?
Teaching decision: if two of three answers remain nominal or lack a validity condition, rebuild the vocabulary with a numerical example before any formula. A fluent but unfalsifiable answer does not count as mastery.
Timed plan
| Time | Activity | Observable evidence |
|---|---|---|
| 0–10 min | Individual diagnostic, then pair comparison | Three answers and one named uncertainty |
| 10–25 min | Situation and vocabulary | Annotated input → state → output diagram |
| 25–55 min | Develop the mechanism on the board | Shapes, assumptions, and intermediate calculation visible |
| 55–75 min | Worked case with deliberate errors | Reasoned correction, not only the right number |
| 75–95 min | Causal lab: predict, change one variable, run | Prediction / observation / delta table |
| 95–112 min | Exercises 1 and 2 with peer correction | Retained artifact and applied rubric |
| 112–120 min | Exit ticket and transfer | Mechanism, boundary, next experiment |
Teaching notes
1. Collect without accepting everything
Filtering is a design stage at the head of the pipeline: fingerprint deduplication, secret detection, licensing and provenance rules. The article repeated 3,000 times must count once — before a single gradient is spent on it.
2. Tokenize
The tokenizer learns a subword vocabulary. “Maya reads a book” becomes 5 tokens for 4 words: [Maya][ reads][ a][ bo][ok] — “book” is split in two; boundaries do not follow words.
3. Build inputs and targets
Shifting by one token manufactures supervision for free: for [A,B,C,D], input [A,B,C], target [B,C,D]. Each position predicts the next token without seeing the future; a sequence of L tokens yields L−1 training positions.
Working formula:
targets = tokens shifted left by one
4. Form batches
You align on the longest: T = 9; the short one gets 5 padded positions and a mask marks the emptiness. The mean loss divides by the 13 real positions, never by the 18 cells of the rectangle.
5. Loss and update
Softmax turns scores into probabilities, then the loss takes −log p(target): p = 0.25 → 1.386; p = 0.50 → 0.693; p = 0.01 → 4.605. Gradients then redistribute that surprise across every parameter that contributed to it.
Working formula:
loss = −log p(target token)
6. Base model and later stages
The pipeline ends in three branches: the base model; instruction tuning and alignment, which change behavior; and an evaluation set kept strictly separate from the corpus. That separation is decided at filtering time, not on test day.
Running the worked case
“Maya reads a book” becomes five tokens. The batch uses the first four as inputs and the last four as targets. If the model assigns 0.25 to the correct target, positional loss is −log(0.25) ≈ 1.386.
Do not reveal the result at once. Ask learners to predict the next operation, its shape, and the expected sign. After each line ask: “What changed? What stayed fixed? Which assumption did we use?” A calculation error repaired with a causal chain is worth more than a guessed result.
Lab protocol
- Write a qualitative and, where possible, numerical prediction before touching a control.
- Change one variable only; retain a capture or record initial and final values.
- Explain the delta through the mechanism, not “the tool did that.”
- Test one boundary value and state where the model stops representing a real system.
Misconceptions
| # | Observable misconception | Grounded correction | Probe |
|---|---|---|---|
| 1 | “Collect without accepting everything guarantees the outcome without assumptions or measurement.” | Filtering is a design stage at the head of the pipeline: fingerprint deduplication, secret detection, licensing and provenance rules. The article repeated 3,000 times must count once — before a single gradient is spent on it. | Ask for a counterexample, then restate the mechanism with its validity condition. |
| 2 | “Tokenize guarantees the outcome without assumptions or measurement.” | The tokenizer learns a subword vocabulary. “Maya reads a book” becomes 5 tokens for 4 words: [Maya][ reads][ a][ bo][ok] — “book” is split in two; boundaries do not follow words. | Ask for a counterexample, then restate the mechanism with its validity condition. |
| 3 | “Build inputs and targets guarantees the outcome without assumptions or measurement.” | Shifting by one token manufactures supervision for free: for [A,B,C,D], input [A,B,C], target [B,C,D]. Each position predicts the next token without seeing the future; a sequence of L tokens yields L−1 training positions. | Ask for a counterexample, then restate the mechanism with its validity condition. |
Boundary to maintain: A teaching pipeline omits distributed storage, security, data policies, and many production quality controls.
Probing questions
- If we remove or reverse Collect without accepting everything, which output changes first, and what observation would show it?
- If we remove or reverse Tokenize, which output changes first, and what observation would show it?
- If we remove or reverse Build inputs and targets, which output changes first, and what observation would show it?
- If we remove or reverse Form batches, which output changes first, and what observation would show it?
- If we remove or reverse Loss and update, which output changes first, and what observation would show it?
- If we remove or reverse Base model and later stages, which output changes first, and what observation would show it?
Assessment
| Level | Criterion |
|---|---|
| 0 | Repeats terms without connecting input, transformation, and output. |
| 1 | Describes the chain but checks neither shape nor assumption. |
| 2 | Executes the case, explains the result, and names one limitation. |
| 3 | Transfers to a new case, compares an alternative, and proposes a measurement that could invalidate the choice. |
Exit threshold: level 2 on the worked case and at least one exercise; a memorized formula without interpretation remains level 1.
Observation and remediation protocol
During discussion, the instructor records evidence rather than impressions. Evidence of understanding contains a named object, a justified transformation, and a checkable consequence. If a learner gives the right result without a chain, ask for the preceding line. If the chain is coherent but the result is wrong, preserve the chain and isolate the arithmetic error. If vocabulary from another concept is used, compare both mechanisms in an input, state, output, cost, and boundary table. Remediation targets the first break only: vocabulary, shapes, operation, interpretation, or claim scope. After correction, use a neighboring case with a changed value; success on the same example does not prove transfer. For pair work, assign operator and verifier roles, then swap. The verifier does not supply the answer: they request an assumption, check the shape, and ask what observation could contradict the reasoning. The instructor retains the exit ticket and classifies the dominant break. The next session opens with a three-minute problem aimed at that break instead of repeating the whole lesson.
Differentiation
- Support: provide shapes and the first transformation; let the learner complete interpretation and boundary.
- Core path: worked case without result, lab with one assigned variable, diagnostic exercise.
- Extension: change one assumption, compare two mechanisms, and define the metric that would decide between them.
Post-session follow-up
Within twenty-four hours, return each annotated exit ticket with one priority, the exact resource to reopen, and a mini-case different from the worked case. Revision requires three items: a written prediction, a retained trace, and one sentence explaining the delta. At the next session, sample two submissions: one that repaired the break and one that remains ambiguous. Discuss them anonymously, then state the criterion that separates them. Do not use completion rate as evidence of mastery. Evidence of remediation is a correct chain on a new case with a stated boundary. If the same break appears in more than one third of the group, repair the support or demonstration before blaming learners.
Follow-up closes only when the new artifact shows the causal chain, check, and boundary—not merely when a file has been submitted.
Sources and evidence boundary
- Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer” (T5/C4 corpus curation), JMLR (2020).
- Course source packet supplied by the owner; named-product details remain source-reported until primary verification.
Scope: Established mechanisms; numerical simplifications are pedagogical. These references support the session frame; they do not turn a reported product choice into an independently verified result.
Exit ticket
In no more than six lines: mechanism; calculation or trace; observation; boundary; evidence level; next experiment. The instructor marks one priority causal break for revision.