advanced · Session 11
Connect the causal objective, cross-entropy, backpropagation, optimizer, mixed precision, parallelism, and checkpoints.
Study this session as a causal investigation. Before every formula or interaction, write what you expect to change and what must remain fixed. During the calculation, retain units, shapes, and intermediate values so an error can be located without restarting at random. After the result, translate the number or state into one sentence about system behavior. Always finish with a counterexample or boundary value. This discipline separates understanding a mechanism from merely recognizing its vocabulary and makes the lab reproducible by another learner. Record the evidence that changed your initial prediction.
The model maximizes next-token likelihood at every position permitted by the causal mask. Mean loss aggregates valid positions and examples.
L = −Σ log p(x_t | x_<t)
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Stable log-softmax subtracts log-sum-exp. Loss then selects the target log-probability. Larger logits matter only relative to others.
CE(z,y)=−z_y+log Σ exp(z_j)
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
The chain rule computes how each parameter contributed to loss. Saved activations consume memory; activation checkpointing trades recomputation for memory.
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
AdamW combines gradient moments, learning rate, and weight decay. Clipping can bound extreme gradients but does not repair faulty data or architecture.
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
BF16 reduces tensor memory without storing every state at full precision. Data parallelism replicates weights; tensor and pipeline parallelism split other dimensions with communication.
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Exact recovery requires weights, optimizer state, scheduler, optional scaler, data position, and random states. A weights-only file is not a complete training checkpoint.
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
For logits [2,1,0] and target 0, softmax ≈ [0.665,0.245,0.090], so CE ≈ 0.408. The lab changes learning rate, gradient, and weight, then shows fields required for recovery.
Reading method: write the data, state every object shape, perform one transformation, and interpret the result before continuing.
Distributed performance depends on hardware, network, model size, and implementation; no lab estimate is a benchmark.
Evidence status: Established mechanisms; numerical simplifications are pedagogical.
The model maximizes next-token likelihood at every position permitted by the causal mask. Mean loss aggregates valid positions and examples.
Stable log-softmax subtracts log-sum-exp. Loss then selects the target log-probability. Larger logits matter only relative to others.
The chain rule computes how each parameter contributed to loss. Saved activations consume memory; activation checkpointing trades recomputation for memory.
AdamW combines gradient moments, learning rate, and weight decay. Clipping can bound extreme gradients but does not repair faulty data or architecture.
Scope: Established mechanisms; numerical simplifications are pedagogical.