1. Collect without accepting everything
The problem: A raw crawl contains the same viral article 3,000 times, API keys, and SEO spam. Trained as is, the model memorizes the duplicates — and can regurgitate them, secrets included. Fix it afterwards? The copies are already spread across every weight.
The idea: Filtering is a design stage at the head of the pipeline: fingerprint deduplication, secret detection, licensing and provenance rules. The article repeated 3,000 times must count once — before a single gradient is spent on it.
Why / at what price: Why so early: removing a document costs one comparison; unlearning it would cost a retraining run. The price: every filter has false positives — too aggressive, and it strips out the very diversity the gradient average was meant to capture.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
2. Tokenize
The problem: The model does not consume letters: it needs discrete units from a finite inventory. Splitting by words blows up the vocabulary (“book”, “books”, “bookish”…); splitting by characters stretches every sentence and its cost. Where do you cut?
The idea: The tokenizer learns a subword vocabulary. “Maya reads a book” becomes 5 tokens for 4 words: [Maya][ reads][ a][ bo][ok] — “book” is split in two; boundaries do not follow words.
Why / at what price: Direct consequence: cost is counted in tokens, not words — here 5/4, i.e. +25%. The price: the vocabulary is frozen before training; a poorly covered domain (code, rare language, jargon) fragments into short tokens and pays more for the same information.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
3. Build inputs and targets
The problem: You need millions of labeled examples, and nobody will hand-annotate the web. Where do targets come from? The text must supervise itself — without leakage: if a position can see its own answer, the loss drops to zero while learning nothing.
The idea: Shifting by one token manufactures supervision for free: for [A,B,C,D], input [A,B,C], target [B,C,D]. Each position predicts the next token without seeing the future; a sequence of L tokens yields L−1 training positions.
targets = tokens shifted left by one
Why / at what price: Free and unlimited — this is what makes pre-training possible at this scale. The price: supervision reduces to “the next token”. The model learns what the corpus makes follow, not what is true; target quality is exactly text quality.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
4. Form batches
The problem: A cost-effective GPU processes thousands of positions in parallel, but sequences differ in length: 4 positions here, 9 there. How do you stack them into one rectangular tensor without corrupting the loss?
The idea: You align on the longest: T = 9; the short one gets 5 padded positions and a mask marks the emptiness. The mean loss divides by the 13 real positions, never by the 18 cells of the rectangle.
Why / at what price: The price of the rectangle: 5 cells out of 18 — about 28% of this batch’s compute — heat up nothing. And a forgotten mask corrupts the curve silently: the mean drops (sum ÷ 18) while not a single prediction improves.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
5. Loss and update
The problem: At every position the model emits one score per vocabulary token — often 100,000 of them. The target is ONE token. You must turn those scores and that target into a single number that measures surprise, and punishes confident error hard.
The idea: Softmax turns scores into probabilities, then the loss takes −log p(target): p = 0.25 → 1.386; p = 0.50 → 0.693; p = 0.01 → 4.605. Gradients then redistribute that surprise across every parameter that contributed to it.
loss = −log p(target token)
Why / at what price: The −log slope blows up near zero, deliberately: being confident and wrong is very expensive. The price: the mean loss becomes a spoofable indicator — denominator, padding, or corpus mix can push it down with no real progress.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
6. Base model and later stages
The problem: The loop stops: you have a base model that completes text. It does not follow instructions and can be toxic — and you must prove its capabilities when it may already have read your test questions during pre-training.
The idea: The pipeline ends in three branches: the base model; instruction tuning and alignment, which change behavior; and an evaluation set kept strictly separate from the corpus. That separation is decided at filtering time, not on test day.
Why / at what price: Why the seal is vital: one test question seen in pre-training turns the score into a memory measurement. The operating price: keeping the seal across terabytes takes permanent tooling — fingerprints, n-gram overlap — an engineering cost, not a formality.
Understanding check
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.