The complete pre-training pipeline

Connect raw data, filtering, tokenization, batches, targets, loss, updates, and the base model.

Applied AI · intermediate · Session 11

Mechanism map

RAW TEXT  “Maya reads a book”
      │
      ▼
┌─────────────────┐  duplicates, secrets, spam, licence, provenance
│  filtering      │
└────────┬────────┘
         ▼
┌─────────────────┐  → [7412, 611, 145, 2380, 91]   5 tokens / 4 words
│  tokenization   │
└────────┬────────┘
         ▼
┌─────────────────┐  input  [7412,  611, 145, 2380]
│  shift by one   │  target [ 611,  145, 2380,  91]
└────────┬────────┘
         ▼
┌─────────────────┐  B = 2, T = 9 (batch max), padding + mask
│  batching       │
└────────┬────────┘
         ▼
 logits (B,T,V) → softmax → loss = −log p(target) → gradients → update
         │
         ▼
 BASE MODEL ──▶ instruction tuning ──▶ alignment      │  SEPARATE eval

The problem — Collect without accepting everything

A raw crawl contains the same viral article 3,000 times, API keys, and SEO spam. Trained as is, the model memorizes the duplicates — and can regurgitate them, secrets included. Fix it afterwards? The copies are already spread across every weight.

The idea — Collect without accepting everything

Filtering is a design stage at the head of the pipeline: fingerprint deduplication, secret detection, licensing and provenance rules. The article repeated 3,000 times must count once — before a single gradient is spent on it.

Why / at what price — Collect without accepting everything

Why so early: removing a document costs one comparison; unlearning it would cost a retraining run. The price: every filter has false positives — too aggressive, and it strips out the very diversity the gradient average was meant to capture.

Check: The same article appears 3,000 times in the raw corpus. Describe the effect on parameters, then say why removing it at filtering costs less than fixing it after training.

Visual support — Collect without accepting everything

raw corpus: 100,000 documents        (illustrative proportions)
   │  deduplication       − 24,000   (the same article ×3,000…)
   ▼
 76,000
   │  secrets, licensing  −  1,500
   ▼
 74,500
   │  quality, spam       − 14,500
   ▼
training corpus: 60,000 documents

a document removed here = zero gradients ever spent on it

The problem — Tokenize

The model does not consume letters: it needs discrete units from a finite inventory. Splitting by words blows up the vocabulary (“book”, “books”, “bookish”…); splitting by characters stretches every sentence and its cost. Where do you cut?

The idea — Tokenize

The tokenizer learns a subword vocabulary. “Maya reads a book” becomes 5 tokens for 4 words: [Maya][ reads][ a][ bo][ok] — “book” is split in two; boundaries do not follow words.

Why / at what price — Tokenize

Direct consequence: cost is counted in tokens, not words — here 5/4, i.e. +25%. The price: the vocabulary is frozen before training; a poorly covered domain (code, rare language, jargon) fragments into short tokens and pays more for the same information.

Check: “Maya reads a book” yields 5 tokens for 4 words. Identify the word that got split and explain why a token count above the word count directly changes batch cost.

The problem — Build inputs and targets

You need millions of labeled examples, and nobody will hand-annotate the web. Where do targets come from? The text must supervise itself — without leakage: if a position can see its own answer, the loss drops to zero while learning nothing.

The idea — Build inputs and targets

Shifting by one token manufactures supervision for free: for [A,B,C,D], input [A,B,C], target [B,C,D]. Each position predicts the next token without seeing the future; a sequence of L tokens yields L−1 training positions.

targets = tokens shifted left by one

Why / at what price — Build inputs and targets

Free and unlimited — this is what makes pre-training possible at this scale. The price: supervision reduces to “the next token”. The model learns what the corpus makes follow, not what is true; target quality is exactly text quality.

Check: For [A,B,C,D], write input and target using the shift-left-by-one rule. Then apply a shift of 2: which position loses its target, and why does the shape check fail?

The problem — Form batches

A cost-effective GPU processes thousands of positions in parallel, but sequences differ in length: 4 positions here, 9 there. How do you stack them into one rectangular tensor without corrupting the loss?

The idea — Form batches

You align on the longest: T = 9; the short one gets 5 padded positions and a mask marks the emptiness. The mean loss divides by the 13 real positions, never by the 18 cells of the rectangle.

Why / at what price — Form batches

The price of the rectangle: 5 cells out of 18 — about 28% of this batch’s compute — heat up nothing. And a forgotten mask corrupts the curve silently: the mean drops (sum ÷ 18) while not a single prediction improves.

Check: A batch holds one 4-position and one 9-position sequence. Give T, the number of padded positions, and say what happens to the mean loss if the mask does not exclude them.

Visual support — Form batches

T = 9 (batch max length)

seq. A   [7412][ 611][ 145][2380][ PAD][ PAD][ PAD][ PAD][ PAD]
seq. B   [ ...][ ...][ ...][ ...][ ...][ ...][ ...][ ...][ ...]
mask A   [   1][   1][   1][   1][   0][   0][   0][   0][   0]

real positions = 4 + 9 = 13        rectangle cells = 18
mean loss = sum of losses ÷ 13     (never ÷ 18)

The problem — Loss and update

At every position the model emits one score per vocabulary token — often 100,000 of them. The target is ONE token. You must turn those scores and that target into a single number that measures surprise, and punishes confident error hard.

The idea — Loss and update

Softmax turns scores into probabilities, then the loss takes −log p(target): p = 0.25 → 1.386; p = 0.50 → 0.693; p = 0.01 → 4.605. Gradients then redistribute that surprise across every parameter that contributed to it.

loss = −log p(target token)

Why / at what price — Loss and update

The −log slope blows up near zero, deliberately: being confident and wrong is very expensive. The price: the mean loss becomes a spoofable indicator — denominator, padding, or corpus mix can push it down with no real progress.

Check: p(target) = 0.25 gives loss 1.386. Recompute for 0.50 and 0.01, and state how much the penalty grows between 0.25 and 0.01.

Visual support — Loss and update

loss = −log p(target)

4.6 ┤ ●  p = 0.01   “confident and wrong” is very expensive
    │
    │
1.4 ┤          ●  p = 0.25
0.7 ┤               ●  p = 0.50
0.0 ┤─────────────────────────● p = 1.00
    └─────────────────────────────▶ p(target)

The problem — Base model and later stages

The loop stops: you have a base model that completes text. It does not follow instructions and can be toxic — and you must prove its capabilities when it may already have read your test questions during pre-training.

The idea — Base model and later stages

The pipeline ends in three branches: the base model; instruction tuning and alignment, which change behavior; and an evaluation set kept strictly separate from the corpus. That separation is decided at filtering time, not on test day.

Why / at what price — Base model and later stages

Why the seal is vital: one test question seen in pre-training turns the score into a memory measurement. The operating price: keeping the seal across terabytes takes permanent tooling — fingerprints, n-gram overlap — an engineering cost, not a formality.

Check: A question from your evaluation set also appears in the pre-training corpus. Which score becomes uninterpretable, and at which stage of the diagram should the contamination have been blocked?

Worked case — full trace

“Maya reads a book” becomes five tokens. The batch uses the first four as inputs and the last four as targets. If the model assigns 0.25 to the correct target, positional loss is −log(0.25) ≈ 1.386.

“Maya reads a book” → 5 tokens: [Maya][ reads][ a][ bo][ok] = [7412,611,145,2380,91]

input   = [7412,  611,  145, 2380]     (4 positions)
target  = [ 611,  145, 2380,   91]     (shifted left by 1)

position 3: the model must predict token 2380 (“bo”)
  p(2380) = 0.25  →  loss = −log(0.25) = 1.386   ✅ finite, usable
  if p(2380) = 0.50  →  loss = 0.693             ✅ half the penalty
  if p(2380) = 0.01  →  loss = 4.605             ❌ the model is very surprised

BATCH: sequence A = 4 positions (our 5 tokens, shifted), sequence B = 9 positions
       → T = 9, 5 padded positions
  mean loss WITHOUT mask = sum / 18 positions    ❌ diluted by emptiness
  mean loss WITH mask    = sum / 13 positions    ✅ only real targets

SHAPE CHECK: input (B=2, T=9) → logits (2, 9, V) → targets (2, 9).
A shift of 2 would leave the last position targetless: (2, 8) ≠ (2, 9) → rejected.

Where a pipeline decision gets paid for later

Stage Design decision What breaks if you miss it
Filtering Deduplication + provenance A document repeated 3,000 times is memorized and leaks
Tokenization Vocabulary and boundaries “book” split into 2 tokens: cost and loss shift silently
Batching Length T, padding, mask Padding enters the mean loss and distorts the curve
Evaluation Strictly separate set Contamination: the score measures memory, not capability

Causal lab

Predict → change one variable → run → explain the delta

/interactives/curriculum/pretraining-pipeline.html?lang=en

Common errors

“More data is always better; we will filter at the end.”

Filtering sits at the head of the diagram for a reason: once the 3,000 copies are learned they are spread across every parameter. No later stage removes them — post-training only masks the behavior.

“A falling mean loss proves the model is improving.”

Not if padding is counted: dividing by 18 instead of 13 lowers the mean without a single prediction improving. Always check the denominator before reading the curve.

Boundary, evidence, and sources

A teaching pipeline omits distributed storage, security, data policies, and many production quality controls.

Evidence status: Established mechanisms; numerical simplifications are pedagogical.

  • Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer” (T5/C4 corpus curation), JMLR (2020).
  • Course source packet supplied by the owner; named-product details remain source-reported until primary verification.

Transfer challenge

Your team receives 10,000 documents from a new domain: medication leaflets.

  1. Pick ONE pipeline stage to harden first and justify it with a concrete risk.
  2. The current tokenizer splits “paracetamol” into 4 tokens: quantify the effect on the cost of a 200-word leaflet.
  3. State the test that would make you roll back before training.

Synthesis and exit ticket

  • Collect without accepting everything
  • Tokenize
  • Build inputs and targets
  • Form batches
  • Loss and update
  • Base model and later stages

Ticket: mechanism · trace · observation · boundary · evidence · next experiment

Instructor notes: Frame the problem before naming the mechanism. Collect an initial prediction and retain it for the exit ticket.

Instructor notes: Connect every step to the next with a causal verb. Flag any merely decorative arrow.

Instructor notes: Project the raw sample for sixty silent seconds before any instruction, then ask: “what worries you here?”. Expected: spam and duplicates; the API key usually goes unnoticed — point it out last, it is the real danger.

Instructor notes: Open with a real unfiltered sample on screen (spam, duplicate, a visible API key). Let the room propose rejection rules before you name “deduplication” and “provenance”.

Instructor notes: Answer: the 3,000 copies pull the weights toward verbatim recall and can leak at inference; at filtering, removal costs one fingerprint comparison, after training it would cost a retraining run. Expected wrong answer: “the model will average it out by itself”. Point back to common error 1.

Instructor notes: Have the room estimate the rejected share before revealing the numbers — it is almost always underestimated. State that proportions are illustrative; the stage order is real: cheapest-to-detect is removed first.

Instructor notes: Write “does the model read letters? words?” on the board and take a vote. The majority votes “words” — perfect: the next beat shows the real answer is “neither”.

Instructor notes: Tokenize the sentence live in an online tokenizer and have them count aloud. The 5-tokens/4-words mismatch must come from their screen, not the slide.

Instructor notes: Answer: “book” → [ bo][ok]. Cost — memory, compute, billing — is counted in tokens: +25% on this sentence. Expected wrong answer: picking “Maya” because a name “seems rare”; have them check the trace, Maya fits in one token.

Instructor notes: Ask: “who labeled the web?”. Let the silence stretch, then draw out that text labels itself by shifting. This is the session’s central click — give it three full minutes.

Instructor notes: Hand out four cards [A][B][C][D] and physically slide the target row by one. Then ask what falls off the edge: that is what explains the lost position.

Instructor notes: Answer: input [A,B,C], target [B,C,D]; with a shift of 2, D loses its target and shapes (2,8) ≠ (2,9) reject the batch. Expected wrong answer: “fill the target with PAD” — show that a PAD target would enter the loss and corrupt it.

Instructor notes: Ask for an estimate: “two sequences, 4 and 9 positions — how many GPU cells?”. The answers 13 and 18 always coexist; keep both on the board, the beat decides between them.

Instructor notes: Have them draw the 4+9 batch on grid paper with padded cells hatched. Count 18 then 13 cells together: the mask becomes visually obvious rather than a rule to memorize.

Instructor notes: Answer: T = 9 and 5 padded positions; without the mask the mean divides by 18 and drops artificially — that is common error 2. Expected wrong answer: T = 13, adding the lengths instead of taking the maximum.

Instructor notes: Have them highlight the five PAD cells and hand-write “0” on each. Then ask: “what does the mean become if we still divide by 18?” — lower, with zero progress. The mask becomes self-evident.

Instructor notes: Hook question: “the model puts 1% on the right answer — light fine or heavy fine?”. Have them justify before showing the −log curve; linear intuitions are about to be surprised.

Instructor notes: Have them sketch −log(p) by hand for p = 0.5 / 0.25 / 0.01. The slope blowing up near zero alone explains why confident errors are so costly.

Instructor notes: Answer: 0.693 for p = 0.50 and 4.605 for p = 0.01; from 0.25 to 0.01 the penalty climbs from 1.386 to 4.605, i.e. +3.22. Expected wrong answer: linear reasoning (“25× less probability = 25× more loss”) — the curve is logarithmic.

Instructor notes: Read the curve right to left: “halving the loss means going from 0.25 to 0.50 — but from 0.01 to 0.25 you gain 3.2”. A model’s first steps pay off enormously; the last points of loss are the most expensive.

Instructor notes: Announce a fake triumph: “our model jumps from 61 to 89%!”. Let the celebration deflate when someone asks where the test questions came from — if nobody asks, that is the room’s real diagnostic.

Instructor notes: Pose the contaminated benchmark as a judgment call, not a definition: “your model jumps from 61 to 89% — what do you check first?” Let the debate run.

Instructor notes: Answer: the benchmark score now measures memory, not capability; the block belongs at filtering — cross-fingerprinting corpus against evaluation — at the very top of the diagram. Expected wrong answer: “remove the question afterwards”; the rest of the set stays suspect.

Instructor notes: Walk line by line. Locate an inconsistency at the first faulty step, not only on the final line.

Instructor notes: Have learners fill the final row before revealing it: that trade-off is what decides in production.

Instructor notes: Retain initial and final values. Do not allow simultaneous changes that make the delta impossible to attribute.

Instructor notes: For each claim, have the group produce the smallest counterexample before giving the correction.

Instructor notes: Separate verifiable mechanism, reported implementation choice, and experimental result. Evidence precision must match claim precision.

Instructor notes: Expected: (1) filtering — personal data and medical confidentiality, late removal is impossible; (2) 200 words far exceed 250 tokens once jargon fragments: +25 to +100% depending on density — accept any argued figure; (3) a measurable recall: duplication rate, identifier leakage, vocabulary coverage on a sample. Misconception to harvest: “harden evaluation first” — without a clean corpus, evaluation measures noise. Ten minutes, groups of three.

Instructor notes: Rebuild the chain without looking at the slides, then fill the ticket in at most six lines. Compare with the opening prediction and name what actually changed.