Exercises with solutions — Pre-training systems and optimization
General instruction: every answer must show data, transformation, result, one check, and one limitation. A bare number or copied definition is insufficient.
Starting data: For logits [2,1,0] and target 0, softmax ≈ [0.665,0.245,0.090], so CE ≈ 0.408. The lab changes learning rate, gradient, and weight, then shows fields required for recovery.
Boundary to retain: Distributed performance depends on hardware, network, model size, and implementation; no lab estimate is a benchmark.
Exercise 1 — Calculated trace — Causal objective
Reproduce and annotate the chain L = −Σ log p(x_t | x_<t). Increase the target logit from 2 to 2.4 while keeping the other logits at 1 and 0. Recompute softmax and cross-entropy for target 0.
Deliverable: a data → operation → result → interpretation table, plus two sentences about the changed value.
Worked solution
For logits [2,1,0] and target 0, softmax ≈ [0.665,0.245,0.090], so CE ≈ 0.408. The lab changes learning rate, gradient, and weight, then shows fields required for recovery.
Solved variant: Softmax moves from about [0.665,0.245,0.090] to [0.748,0.184,0.068]. Loss −log p(target) falls from about 0.408 to 0.291: raising the target logit relative to the others reduces loss.
The model maximizes next-token likelihood at every position permitted by the causal mask. Mean loss aggregates valid positions and examples. Stable log-softmax subtracts log-sum-exp. Loss then selects the target log-probability. Larger logits matter only relative to others. The minimum check covers dimensions, sign, and order of magnitude. If the observed change contradicts the prediction, locate the first operation whose direction changes instead of fixing only the final line.
Rubric Exercise 1 — /10
| Criterion | Points |
|---|---|
| Explicit data and shapes | 2 |
| Traceable calculation | 3 |
| Prediction before variation | 2 |
| Interpretation and check | 2 |
| Named limitation | 1 |
Exercise 2 — Diagnose a seductive explanation — Backpropagation
A colleague claims: « Backpropagation proves the system will be accurate, fast, and stable in every context. »
- Separate mechanism, assumption, observation, and conclusion.
- Name two correct lesson elements and two unsupported extrapolations.
- Propose a bounded experiment with controlled variable, metric, and stop threshold.
- Rewrite the claim as one defensible sentence.
Reasoned solution
The chain rule computes how each parameter contributed to loss. Saved activations consume memory; activation checkpointing trades recomputation for memory. AdamW combines gradient moments, learning rate, and weight decay. Clipping can bound extreme gradients but does not repair faulty data or architecture. Distributed performance depends on hardware, network, model size, and implementation; no lab estimate is a benchmark.
The claim mixes a local relation with a global guarantee. A defensible version states only the observed mechanism, test conditions, and measured metric. Stop the test if shapes become invalid, the metric crosses the declared degradation threshold, or another variable changes.
Rubric Exercise 2 — /10
2 points per element: separation, lesson grounding, extrapolations, protocol, and rewrite.
Exercise 3 — Architecture decision and transfer — Precision and parallelism
You must reproduce the worked case “For logits [2,1,0] and target 0, softmax ≈ [0.665,0.245,0.090], so CE ≈ 0.408. The lab changes learning rate, gradient, and weight, then shows fields required for recovery.” under two conditions. Option A uses the full chain through “Precision and parallelism.” Option B is a transparent baseline that retains “Causal objective,” calculates the expected output directly, and does not use the compression or adjustment mechanism studied. Build a decision record containing:
- the workload and dominant constraint;
- each option’s mechanism, without slogans;
- one quality, memory, or latency prediction;
- one case where your preferred procedure loses;
- an A/B protocol, metrics, and rollback threshold;
- a bounded verdict: choose, defer, or reject.
Elements of a strong solution
BF16 reduces tensor memory without storing every state at full precision. Data parallelism replicates weights; tensor and pipeline parallelism split other dimensions with communication. Exact recovery requires weights, optimizer state, scheduler, optional scaler, data position, and random states. A weights-only file is not a complete training checkpoint. Established mechanisms; numerical simplifications are pedagogical.
A strong answer does not make the newer mechanism the default winner. It retains a measurable baseline, sets thresholds before testing, and separates component cost from whole-system behavior. The verdict names what remains uncertain and the next evidence that could change it.
Rubric Exercise 3 — /15
| Criterion | Points |
|---|---|
| Framing and baseline | 3 |
| Compared causal chains | 4 |
| Protocol and metrics | 4 |
| Rollback threshold | 2 |
| Bounded verdict | 2 |
Extension
Repeat Exercise 3 after reversing the dominant constraint. If you optimized memory, impose a strict quality floor; if you optimized fidelity, halve the memory budget. Identify the first part of the verdict that changes and the evidence required.
Review before submission
Review the packet as if another group had to reproduce it without speaking to you. Are all starting values or assumptions present? Are shapes or roles stated before operations? Does the prediction truly precede the observation? Is the result translated into behavior rather than left as an isolated number? Did you test a boundary value and identify a stop condition? Does the procedure or architecture choice retain a measurable baseline and a rollback threshold set before the test? Finally, highlight one sentence that states what is established, one that remains a hypothesis, and one measurement that could change your verdict. If any element is missing, the work is not reproducible.
Reference appendix for correction
1. Causal objective
The model maximizes next-token likelihood at every position permitted by the causal mask. Mean loss aggregates valid positions and examples.
Working formula:
L = −Σ log p(x_t | x_<t)
2. Cross-entropy from logits
Stable log-softmax subtracts log-sum-exp. Loss then selects the target log-probability. Larger logits matter only relative to others.
Working formula:
CE(z,y)=−z_y+log Σ exp(z_j)
3. Backpropagation
The chain rule computes how each parameter contributed to loss. Saved activations consume memory; activation checkpointing trades recomputation for memory.
4. Optimizer
AdamW combines gradient moments, learning rate, and weight decay. Clipping can bound extreme gradients but does not repair faulty data or architecture.
5. Precision and parallelism
BF16 reduces tensor memory without storing every state at full precision. Data parallelism replicates weights; tensor and pipeline parallelism split other dimensions with communication.
6. Complete checkpoint
Exact recovery requires weights, optimizer state, scheduler, optional scaler, data position, and random states. A weights-only file is not a complete training checkpoint.
Sources and evidence boundary
- Kingma & Ba, “Adam: A Method for Stochastic Optimization”, ICLR (2015).
- Loshchilov & Hutter, “Decoupled Weight Decay Regularization” (AdamW), ICLR (2019).
- Micikevicius et al., “Mixed Precision Training”, ICLR (2018).
- Course source packet supplied by the owner; named-product details remain source-reported until primary verification.
Scope: Established mechanisms; numerical simplifications are pedagogical. These references support the session frame; they do not turn a reported product choice into an independently verified result.