# Exercises with solutions — The complete pre-training pipeline **General instruction:** every answer must show data, transformation, result, one check, and one limitation. A bare number or copied definition is insufficient. > **Starting data:** “Maya reads a book” becomes five tokens. The batch uses the first four as inputs and the last four as targets. If the model assigns 0.25 to the correct target, positional loss is −log(0.25) ≈ 1.386. > > **Boundary to retain:** A teaching pipeline omits distributed storage, security, data policies, and many production quality controls. ## Exercise 1 — Calculated trace — Collect without accepting everything Reproduce and annotate the chain `targets = tokens shifted left by one`. Append a sixth token to the five-token sequence. Trace input→target pairs before and after, then state what changes in the batch without claiming that this one example improves the model. **Deliverable:** a data → operation → result → interpretation table, plus two sentences about the changed value.
Worked solution “Maya reads a book” becomes five tokens. The batch uses the first four as inputs and the last four as targets. If the model assigns 0.25 to the correct target, positional loss is −log(0.25) ≈ 1.386. **Solved variant:** Five tokens produce four next-token pairs; six tokens produce five. The batch gains one supervised position and one computation step, but quality still depends on filtering, diversity, and many updates. Filtering is a design stage at the head of the pipeline: fingerprint deduplication, secret detection, licensing and provenance rules. The article repeated 3,000 times must count once — before a single gradient is spent on it. The tokenizer learns a subword vocabulary. “Maya reads a book” becomes 5 tokens for 4 words: [Maya][ reads][ a][ bo][ok] — “book” is split in two; boundaries do not follow words. The minimum check covers dimensions, sign, and order of magnitude. If the observed change contradicts the prediction, locate the first operation whose direction changes instead of fixing only the final line.
### Rubric Exercise 1 — /10 | Criterion | Points | |---|---:| | Explicit data and shapes | 2 | | Traceable calculation | 3 | | Prediction before variation | 2 | | Interpretation and check | 2 | | Named limitation | 1 | ## Exercise 2 — Diagnose a seductive explanation — Build inputs and targets A colleague claims: « Build inputs and targets proves the system will be accurate, fast, and stable in every context. » 1. Separate mechanism, assumption, observation, and conclusion. 2. Name two correct lesson elements and two unsupported extrapolations. 3. Propose a bounded experiment with controlled variable, metric, and stop threshold. 4. Rewrite the claim as one defensible sentence.
Reasoned solution Shifting by one token manufactures supervision for free: for [A,B,C,D], input [A,B,C], target [B,C,D]. Each position predicts the next token without seeing the future; a sequence of L tokens yields L−1 training positions. You align on the longest: T = 9; the short one gets 5 padded positions and a mask marks the emptiness. The mean loss divides by the 13 real positions, never by the 18 cells of the rectangle. A teaching pipeline omits distributed storage, security, data policies, and many production quality controls. The claim mixes a local relation with a global guarantee. A defensible version states only the observed mechanism, test conditions, and measured metric. Stop the test if shapes become invalid, the metric crosses the declared degradation threshold, or another variable changes.
### Rubric Exercise 2 — /10 2 points per element: separation, lesson grounding, extrapolations, protocol, and rewrite. ## Exercise 3 — Transfer test between two procedures — Loss and update You must reproduce the worked case ““Maya reads a book” becomes five tokens. The batch uses the first four as inputs and the last four as targets. If the model assigns 0.25 to the correct target, positional loss is −log(0.25) ≈ 1.386.” under two conditions. Option A uses the full chain through “Loss and update.” Option B is a transparent baseline that retains “Collect without accepting everything,” calculates the expected output directly, and does not use the compression or adjustment mechanism studied. Build a decision record containing: - the starting values, changed value, and expected result; - each option’s mechanism, without slogans; - a prediction about error, output, or number of steps before execution; - one case where your preferred procedure loses; - an A/B protocol, metrics, and rollback threshold; - a bounded verdict: choose, defer, or reject.
Elements of a strong solution Softmax turns scores into probabilities, then the loss takes −log p(target): p = 0.25 → 1.386; p = 0.50 → 0.693; p = 0.01 → 4.605. Gradients then redistribute that surprise across every parameter that contributed to it. The pipeline ends in three branches: the base model; instruction tuning and alignment, which change behavior; and an evaluation set kept strictly separate from the corpus. That separation is decided at filtering time, not on test day. Established mechanisms; numerical simplifications are pedagogical. A strong answer does not make the newer mechanism the default winner. It retains a measurable baseline, sets thresholds before testing, and separates component cost from whole-system behavior. The verdict names what remains uncertain and the next evidence that could change it.
### Rubric Exercise 3 — /15 | Criterion | Points | |---|---:| | Framing and baseline | 3 | | Compared causal chains | 4 | | Protocol and metrics | 4 | | Rollback threshold | 2 | | Bounded verdict | 2 | ## Extension Repeat Exercise 3 with a new input and target. Double or halve the adjustable value, predict the direction of change, then state which observation would make you abandon the procedure. ## Review before submission Review the packet as if another group had to reproduce it without speaking to you. Are all starting values or assumptions present? Are shapes or roles stated before operations? Does the prediction truly precede the observation? Is the result translated into behavior rather than left as an isolated number? Did you test a boundary value and identify a stop condition? Does the procedure or architecture choice retain a measurable baseline and a rollback threshold set before the test? Finally, highlight one sentence that states what is established, one that remains a hypothesis, and one measurement that could change your verdict. If any element is missing, the work is not reproducible. ## Reference appendix for correction ### 1. Collect without accepting everything Filtering is a design stage at the head of the pipeline: fingerprint deduplication, secret detection, licensing and provenance rules. The article repeated 3,000 times must count once — before a single gradient is spent on it. ### 2. Tokenize The tokenizer learns a subword vocabulary. “Maya reads a book” becomes 5 tokens for 4 words: [Maya][ reads][ a][ bo][ok] — “book” is split in two; boundaries do not follow words. ### 3. Build inputs and targets Shifting by one token manufactures supervision for free: for [A,B,C,D], input [A,B,C], target [B,C,D]. Each position predicts the next token without seeing the future; a sequence of L tokens yields L−1 training positions. > **Working formula:** `targets = tokens shifted left by one` ### 4. Form batches You align on the longest: T = 9; the short one gets 5 padded positions and a mask marks the emptiness. The mean loss divides by the 13 real positions, never by the 18 cells of the rectangle. ### 5. Loss and update Softmax turns scores into probabilities, then the loss takes −log p(target): p = 0.25 → 1.386; p = 0.50 → 0.693; p = 0.01 → 4.605. Gradients then redistribute that surprise across every parameter that contributed to it. > **Working formula:** `loss = −log p(target token)` ### 6. Base model and later stages The pipeline ends in three branches: the base model; instruction tuning and alignment, which change behavior; and an evaluation set kept strictly separate from the corpus. That separation is decided at filtering time, not on test day. ## Sources and evidence boundary - Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer” (T5/C4 corpus curation), JMLR (2020). - Course source packet supplied by the owner; named-product details remain source-reported until primary verification. > **Scope:** Established mechanisms; numerical simplifications are pedagogical. These references support the session frame; they do not turn a reported product choice into an independently verified result.