# Teacher guide — Pre-training systems and optimization

**Duration:** 120 minutes<br>
**Positioning:** Connect the causal objective, cross-entropy, backpropagation, optimizer, mixed precision, parallelism, and checkpoints.<br>
**Expected evidence:** Established mechanisms; numerical simplifications are pedagogical.

## Observable outcomes and preparation

- Compute cross-entropy loss.
- Trace gradient, optimizer, and update.
- Explain reliable checkpoint recovery.

Before class, the instructor runs the worked case and lab, prints the exercise packet, prepares a four-column board—assumption, prediction, observation, delta—and checks that every mathematical notation is paired with dimensions. The demonstration must not become slide reading.

## Diagnostic

1. Explain in one sentence: **Causal objective**. What observation would falsify your explanation?
2. Explain in one sentence: **Cross-entropy from logits**. What observation would falsify your explanation?
3. Explain in one sentence: **Backpropagation**. What observation would falsify your explanation?

**Teaching decision:** if two of three answers remain nominal or lack a validity condition, rebuild the vocabulary with a numerical example before any formula. A fluent but unfalsifiable answer does not count as mastery.

## Timed plan

| Time | Activity | Observable evidence |
|---|---|---|
| 0–10 min | Individual diagnostic, then pair comparison | Three answers and one named uncertainty |
| 10–25 min | Situation and vocabulary | Annotated input → state → output diagram |
| 25–55 min | Develop the mechanism on the board | Shapes, assumptions, and intermediate calculation visible |
| 55–75 min | Worked case with deliberate errors | Reasoned correction, not only the right number |
| 75–95 min | Causal lab: predict, change one variable, run | Prediction / observation / delta table |
| 95–112 min | Exercises 1 and 2 with peer correction | Retained artifact and applied rubric |
| 112–120 min | Exit ticket and transfer | Mechanism, boundary, next experiment |

## Teaching notes

### 1. Causal objective

The model maximizes next-token likelihood at every position permitted by the causal mask. Mean loss aggregates valid positions and examples.

> **Working formula:** `L = −Σ log p(x_t | x_<t)`

### 2. Cross-entropy from logits

Stable log-softmax subtracts log-sum-exp. Loss then selects the target log-probability. Larger logits matter only relative to others.

> **Working formula:** `CE(z,y)=−z_y+log Σ exp(z_j)`

### 3. Backpropagation

The chain rule computes how each parameter contributed to loss. Saved activations consume memory; activation checkpointing trades recomputation for memory.

### 4. Optimizer

AdamW combines gradient moments, learning rate, and weight decay. Clipping can bound extreme gradients but does not repair faulty data or architecture.

### 5. Precision and parallelism

BF16 reduces tensor memory without storing every state at full precision. Data parallelism replicates weights; tensor and pipeline parallelism split other dimensions with communication.

### 6. Complete checkpoint

Exact recovery requires weights, optimizer state, scheduler, optional scaler, data position, and random states. A weights-only file is not a complete training checkpoint.

### Running the worked case

For logits [2,1,0] and target 0, softmax ≈ [0.665,0.245,0.090], so CE ≈ 0.408. The lab changes learning rate, gradient, and weight, then shows fields required for recovery.

Do not reveal the result at once. Ask learners to predict the next operation, its shape, and the expected sign. After each line ask: “What changed? What stayed fixed? Which assumption did we use?” A calculation error repaired with a causal chain is worth more than a guessed result.

### Lab protocol

1. Write a qualitative and, where possible, numerical prediction before touching a control.
2. Change one variable only; retain a capture or record initial and final values.
3. Explain the delta through the mechanism, not “the tool did that.”
4. Test one boundary value and state where the model stops representing a real system.

## Misconceptions

| # | Observable misconception | Grounded correction | Probe |
|---|---|---|---|
| 1 | “Causal objective guarantees the outcome without assumptions or measurement.” | The model maximizes next-token likelihood at every position permitted by the causal mask. Mean loss aggregates valid positions and examples. | Ask for a counterexample, then restate the mechanism with its validity condition. |
| 2 | “Cross-entropy from logits guarantees the outcome without assumptions or measurement.” | Stable log-softmax subtracts log-sum-exp. Loss then selects the target log-probability. Larger logits matter only relative to others. | Ask for a counterexample, then restate the mechanism with its validity condition. |
| 3 | “Backpropagation guarantees the outcome without assumptions or measurement.” | The chain rule computes how each parameter contributed to loss. Saved activations consume memory; activation checkpointing trades recomputation for memory. | Ask for a counterexample, then restate the mechanism with its validity condition. |

> **Boundary to maintain:** Distributed performance depends on hardware, network, model size, and implementation; no lab estimate is a benchmark.

## Probing questions

1. If we remove or reverse **Causal objective**, which output changes first, and what observation would show it?
2. If we remove or reverse **Cross-entropy from logits**, which output changes first, and what observation would show it?
3. If we remove or reverse **Backpropagation**, which output changes first, and what observation would show it?
4. If we remove or reverse **Optimizer**, which output changes first, and what observation would show it?
5. If we remove or reverse **Precision and parallelism**, which output changes first, and what observation would show it?
6. If we remove or reverse **Complete checkpoint**, which output changes first, and what observation would show it?

## Assessment

| Level | Criterion |
|---|---|
| 0 | Repeats terms without connecting input, transformation, and output. |
| 1 | Describes the chain but checks neither shape nor assumption. |
| 2 | Executes the case, explains the result, and names one limitation. |
| 3 | Transfers to a new case, compares an alternative, and proposes a measurement that could invalidate the choice. |

**Exit threshold:** level 2 on the worked case and at least one exercise; a memorized formula without interpretation remains level 1.

## Observation and remediation protocol

During discussion, the instructor records evidence rather than impressions. Evidence of understanding contains a named object, a justified transformation, and a checkable consequence. If a learner gives the right result without a chain, ask for the preceding line. If the chain is coherent but the result is wrong, preserve the chain and isolate the arithmetic error. If vocabulary from another concept is used, compare both mechanisms in an input, state, output, cost, and boundary table. Remediation targets the first break only: vocabulary, shapes, operation, interpretation, or claim scope. After correction, use a neighboring case with a changed value; success on the same example does not prove transfer. For pair work, assign operator and verifier roles, then swap. The verifier does not supply the answer: they request an assumption, check the shape, and ask what observation could contradict the reasoning. The instructor retains the exit ticket and classifies the dominant break. The next session opens with a three-minute problem aimed at that break instead of repeating the whole lesson.

## Differentiation

- **Support:** provide shapes and the first transformation; let the learner complete interpretation and boundary.
- **Core path:** worked case without result, lab with one assigned variable, diagnostic exercise.
- **Extension:** change one assumption, compare two mechanisms, and define the metric that would decide between them.

## Post-session follow-up

Within twenty-four hours, return each annotated exit ticket with one priority, the exact resource to reopen, and a mini-case different from the worked case. Revision requires three items: a written prediction, a retained trace, and one sentence explaining the delta. At the next session, sample two submissions: one that repaired the break and one that remains ambiguous. Discuss them anonymously, then state the criterion that separates them. Do not use completion rate as evidence of mastery. Evidence of remediation is a correct chain on a new case with a stated boundary. If the same break appears in more than one third of the group, repair the support or demonstration before blaming learners.

Follow-up closes only when the new artifact shows the causal chain, check, and boundary—not merely when a file has been submitted.

## Sources and evidence boundary

- Kingma & Ba, “Adam: A Method for Stochastic Optimization”, ICLR (2015).
- Loshchilov & Hutter, “Decoupled Weight Decay Regularization” (AdamW), ICLR (2019).
- Micikevicius et al., “Mixed Precision Training”, ICLR (2018).
- Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

> **Scope:** Established mechanisms; numerical simplifications are pedagogical. These references support the session frame; they do not turn a reported product choice into an independently verified result.

## Exit ticket

In no more than six lines: mechanism; calculation or trace; observation; boundary; evidence level; next experiment. The instructor marks one priority causal break for revision.