Exercises — Session 1: Fundamentals: Transformers & Tokenization

Program : Applied AI — Intermediate Level Instructor: Yann Isola Session : 1 / Module 1

Four exercises of increasing difficulty. Detailed answers appear after each exercise – only consult them after searching!

Exercise 1 — Safari tokens 🔍 (practice)

Difficulty: ★☆☆ Duration : 10 minutes (8 min of exploration + 2 min of sharing) Material : the interactive web page of the session (“Tokenizer” tab) or an online tokenizer. Format: individually or in pairs.

Instructions

You'll explore how a model actually cuts text. Type each of the following entries into the tokenizer and record your observations in the table.

le chat
anticonstitutionnellement
ChatGPT
2024 Then 20240115
bonjour Then bonjuor (with typo)
A phrase from your profession (jargon included) — for example a technical term from your sector.

For each entry, complete:

Entrance	Number of tokens	Does the division surprise you? For what ?
…	…	…

Summary questions

Q1. What types of text produce FEW tokens per character? Which ones produce A LOT?

Q2. A colleague tells you: “I asked the model to write “unconstitutionally” backwards, he made a mistake. This AI sucks. » What do you answer, in one or two sentences, using what you have just observed?

Q3. (Bonus) Your company pays for the API (application programming interface) of an AI provider in token. You write your prompts in French. What budgetary conclusion do you draw from your observations?

Answer key — Exercise 1

Expected observations:

Entrance	Typical observation
`le chat`	Approximately 2 tokens — very frequent words = 1 token each (the space is often included in the following token).
`anticonstitutionnellement`	Rare and long word → broken into several fragments (e.g. `anti` / `constitution` / `nelle` / `ment` — the exact division varies depending on the tokenizer ⚠).
`ChatGPT`	Often 2–3 tokens (e.g. `Chat` / `G` / `PT` ): recent proper nouns or upper/lower case mixtures are fragmented.
`2024` vs. `20240115`	`2024` can be 1 token (common); `20240115` is cut into arbitrary fragments (`2024` / `01` / `15` or other). Explains errors when manipulating strings of numbers.
`bonjour` vs. `bonjuor`	The correct word = 1 token; the typo explodes into 2–4 fragments. The tokenizer only “knows” frequent sequences.
Business jargon	The rarer the term in the training corpus, the more fragmented it is.

Q1 — Model answer: Few tokens: current text, frequent words of the language, standard English. Lots of tokens: rare words, jargon, typos, sequences of numbers, code, languages poorly represented in the training data.

Q2 — Model answer: “The model does not see letters: it sees blocks of several characters (tokens). Reversing a word letter by letter requires access to individual characters that it doesn't natively have — it's a structural limitation of tokenization, not a sign of general "stupidity." On tasks where tokens are sufficient (summarizing, writing, translating), the same model excels. »

Q3 — Model answer (bonus): French is generally divided into more tokens than English for equivalent content (the tokenizers are optimized on predominantly English-speaking corpora ⚠). Consequences: higher API cost for equal content, and context window consumed faster. Avenues: concise prompts, possibly system components in English when acceptable, and monitoring of actual token consumption.

Success criteria: the exercise is successful if the participant has observed for himself (1) that frequency ⇒ fewer tokens, (2) that the same content can cost very different numbers of tokens, (3) and knows how to relate the failed letter count to the opacity of the tokens.

Exercise 2 — The vector analogies game 🧭 (conceptual)

Difficulty: ★★☆ Duration : 8 minutes (6 min in pairs + 2 min of collective correction) Material : paper/pen. No computer needed. Format: pairs.

Reminder of the principle

In the space of embeddings, the meaning relations are directions : roi − homme + femme ≈ reine means that the “arrow” which goes from man has women is the same as that which goes from king has queen .

Part A — Complete the analogies

For each line, find the missing word AND name the “direction of meaning” used (e.g.: masculine→feminine, country→capital, etc.).

Paris − France + Japon ≈ ?
marcher − marche + mange ≈ ? (hint: think about verb forms)
chaton − chat + chien ≈ ?
pilote − avion + navire ≈ ?
Berlin − Allemagne + Italie ≈ ?

Part B — Create your own analogies

Invent two analogies of form A − B + C ≈ D :

one from everyday language;
one from YOUR professional field (finance, law, health, marketing, etc.).

For each, name the direction of meaning.

Part C — The trick question

avocat − tribunal + salade ≈ ? … Does this “equation” make sense? What does this case teach us about the limits of embeddings? static (a unique vector per word) and on the interest of attention current view?

Answer key — Exercise 2

Part A:

Tokyo — direction: country → capital.
eat (or “eats” depending on the interpretation) — direction: conjugated form → infinitive (morphological relationship). Accepted answer: any form showing that the direction “conjugated→infinitive” applied to “eat” gives “eat”.
puppy — direction: adult → young of the animal.
captain (or “sailor/skipper”, accept if justified) — direction: vehicle → person who drives it.
Rome — direction: country → capital (same direction as in 1 and 5: this is the key point — THE SAME arrow works for all country/capital pairs).

Part B—examples of valid answers:

Common language: voir − vu + pris ≈ prendre (participle→infinitive); grand − plus grand + plus petit ≈ petit (comparative).
Professional : action − dividende + obligation ≈ coupon (instrument → associated income); diagnostic − médecin + avocat ≈ consultation juridique (professional → act); prospect − marketing + recrutement ≈ candidat (domain → target).
Correction criterion: the answer is correct if the SAME relationship links A→B and D→C. Have management verbalize: this is the targeted skill.

Part C — Model answer: The equation is shaky because “lawyer” has two meanings (job / fruit): an embedding static can only give ONE vector, which mixes the two meanings - the result is a blur between the legal field and the food field. This is precisely what the attention in transformers: the representation of “lawyer” becomes contextual — drawn towards the fruit if “salad” is in the context, towards the lawyer if “court” is. Morality: embeddings = starting position; attention = adjustment to context.

Success criteria: the participant knows how to (1) resolve an analogy by identifying the direction, (2) construct one, (3) explain why lexical ambiguity requires attention.

Exercise 3 — Triage: which stage of the pipeline? 🏗️ (app)

Difficulty: ★★★ Duration : 15 minutes (10 min in small groups + 5 min for correction) — can be given as homework if time is short. Material : paper/pen. Format: groups of 3–4.

Context

You are an AI consultant. Six clients explain their situation to you. For each, determine what concept of the course is in play and what answer you give it to them. The concepts that can be used: pre-training, post-training/RLHF (reinforcement learning from human feedback), fine-tuning, inference & memory bandwidth, tokenization, scaling laws.

Scenario 1. A legal firm: “The general model writes well, but it never uses our in-house contractual formulations or our standard plan. Should we train our own model from scratch? »

Scenario 2. A bank: “Our chatbot responds correctly but TOO SLOWLY during peak hours. Our service provider offers to double the computing power of the servers. Good idea? »

Scenario 3. A startup: “Our assistant sometimes gives factually good but curt answers, sometimes bordering on unpleasant with customers. The knowledge is there, the tone is wrong. »

Scenario 4. An e-retailer: “The model fails to validate our product references such as “REF-88472-XL-2024”: it reverses numbers, forgets them. Yet he writes impeccable product sheets! »

Scenario 5. An innovation director: “Suppliers are releasing ever larger models and announcing their performance in advance. How can they promise results on a model that has not yet been trained? Deceptive marketing? »

Scenario 6. International customer service: “Our API costs have increased by 40% since we switched our prompts from English to French and Polish, for the same volume of requests. Is the supplier sneakily billing us? »

For each scenario, produce:

There concept of the course concerned (one line).
THE diagnosis in 1–2 sentences.
There recommendation concrete in 1–2 sentences.

Answer key — Exercise 3

Scenario 1 — Fine-tuning. Diagnosis: the model already has the general skill (acquired in pre-training); it lacks in-house specialization – a textbook case of fine-tuning (“corporate onboarding”). Recommendation : especially NO training from scratch (cost of tens to hundreds of millions ⚠, massive data required). Options by increasing cost: (a) instructions + examples in the prompt, (b) fine-tuning of an existing model on a corpus of in-house contracts. Start with (a), move to (b) if insufficient.

Scenario 2 — Inference & memory bandwidth. Diagnosis: inference is governed by memory bandwidth (each generated token requires passing all model parameters), not by computing power. Doubling the calculation risks changing almost nothing. Recommendation : Instead, consider: a smaller model for simple requests (fewer parameters to move = faster), faster memory hardware (HBM — High Bandwidth Memory), shorter responses, or inference optimization techniques offered by the service provider. Require the service provider to diagnose the real bottleneck before paying.

Scenario 3 — Post-training / RLHF. Diagnosis: typical problem of behavioral alignment, not knowledge: this is the role of post-training (the “finishing school”). RLHF (reinforcement learning from human feedback) shapes tone, usefulness and politeness based on human preferences. Recommendation : on a client level: first correct via system instructions (expected tone, examples of good answers); if insufficient, choose a model whose post-training better corresponds to the use, or refine on preferred pairs of responses if the supplier allows it.

Scenario 4 — Tokenization. Diagnosis: “REF-88472-XL-2024” is split into arbitrary tokens; the model manipulates opaque blocks, not characters — hence inversions and omissions on reference type strings, while the writing (task at the token level) remains excellent. Recommendation : do not entrust character-by-character validation to the model alone: delegate it to classic code (regular expression, exact comparison) and reserve the model for editing. General rule: anything that requires precision down to the last character is a programmatic tool.

Scenario 5 — Scaling laws. Diagnosis: not misleading marketing: the laws of scaling show that performance (measured by prediction quality) evolves regularly and predictably with parameters, data and calculation. Laboratories extrapolate these curves before training. Recommendation : explain to the director that the prediction concerns aggregated statistical metrics; the appearance of specific capabilities is less predictable (emergence in stages) and advertising benchmarks always deserve verification on YOUR use cases.

Scenario 6 — Tokenization (multilingual efficiency). Diagnosis: no hidden billing: the tokenizers, optimized on predominantly English-speaking corpora, cut French and especially Polish into more tokens of equal content (⚠ variable ratio depending on the tokenizers). Billed by token, the same volume therefore costs more. Recommendation : measure the ratio of tokens/real word per language, compress system instructions (or even keep them in English if acceptable), compare suppliers — some recent tokenizers are more efficient on European languages (⚠ scalable).

Indicative scale (for group correction): 1 point per correctly identified concept, 1 point per exact diagnosis, 1 point per actionable recommendation — 18 points in total. From 13/18: very good mastery of the module.

Exercise 4 — Sorting the web 🍷 (application · Part E-bis)

Difficulty: ★★☆ Duration : 8 minutes (5 mins sorting + 3 mins pooling) — can be done as homework. Format: in pairs.

Context

You are a data curator at an AI laboratory. The FineWeb type pipeline seen in Part E-bis has just crawled the 5 documents below. For each, decide: KEEP , FILTER (specifying which filter rejects it), or DEDUPLICATE . Justify in one sentence.

Paper 1

Home | Products | About | Contact | Login | Cart (0) | Legal notices | Cookie policy | Accept | Refuse | Configure

Document 2

Photosynthesis is the process by which plants convert light energy into chemical energy. In chloroplasts, chlorophyll absorbs light, which sets off a chain of reactions transforming water and carbon dioxide into glucose and oxygen.

Paper 3

Photosynthesis is the process by which plants convert light energy into chemical energy. In chloroplasts, chlorophyll absorbs light, which triggers a chain of reactions transforming water and CO2 into glucose and oxygen. (published on an aggregator, different source from Document 2)

Paper 4

best vpn 2024 cheap vpn free vpn fast vpn streaming vpn netflix vpn france vpn comparison vpn promo vpn test vpn review vpn top 10 vpn

Document 5

Lorem ipsum dolor sit amet, consectetur adipiscing elit. { padding: 0; margin: 0; } function initCarousel() { return true; }

Answer key — Exercise 4

Document 1 — FILTER (quality filter / heuristics). Pure navigation boilerplate: very short lines, no terminal punctuation, no informative content. This is exactly what WARC + trafilatura extraction and “short line fraction” type filters eliminate — and what WET files let through (hence their 25% excess tokens).

Document 2 — KEEP. Informative, well-formed, autonomous text — and with high educational value: a FineWeb-Edu type classifier would rate it ≥ 3/5. This is the page profile that brings up MMLU and ARC.

Document 3 — DEDUPLICATE. Near copy of Document 2 (some accents and “CO2” changed): similarity well above the ~75% threshold targeted by MinHash, which compares fuzzy 5-gram fingerprints — precisely to catch copies imperfect that an exact comparison would miss. We keep ONE copy (the 2nd), we remove the other.

Document 4 — FILTER (quality/repetition filter). SEO spam: list of keywords without sentences, massive repetition of the same term, no punctuation. Rejected by repetition and terminal punctuation filters — the very type of content that visual inspection of low-quality data reveals (“ads, lists of keywords”).

Document 5 — FILTER (C4 type filters). “Lorem ipsum” and curly braces { are two explicit filters inherited from C4: filler text and code/CSS do not learn anything useful to a LANGUAGE model in a general corpus (the code is trained separately, on dedicated corpora).

Bonus question for sharing: “Why not just have humans sort?” » → At 15,000 billion tokens, this is physically impossible: hence the heuristics, then the next step — AI classifiers trained on a small annotated sample (500,000 pages for FineWeb-Edu) and applied to the entire corpus.