# Exercises — Session 1: Foundations: Transformers & Tokenization

**Program:** Applied AI — Intermediate Level
**Instructor:** Yann Isola
**Session:** 1 / Module 1

Four exercises of increasing difficulty. Detailed answers appear after each exercise — only consult them after searching!

---

## Exercise 1 — Safari tokens 🔍 (practice)

**Difficulty:** ★☆☆
**Duration:** 10 minutes (8 min of exploration + 2 min of sharing)
**Material:** the interactive web page of the session (“Tokenizer” tab) or an online tokenizer.
**Format:** individual or in pairs.

### Instructions

You'll explore how a model actually cuts text. Type each of the following entries into the tokenizer and record your observations in the table.

1. `le chat`
2. `anticonstitutionnellement`
3. `ChatGPT`
4. `2024` then `20240115`
5. `bonjour` then `bonjuor` (with the typo)
6. A phrase from your trade (jargon included) — for example a technical term from your industry.

For each entry, complete:

| Entrance | Number of tokens | Does the division surprise you? For what ? |
|---|---|---|
| … | … | … |

### Summary questions

**Q1.** What types of text produce FEW tokens per character? Which ones produce A LOT?

**Q2.** A colleague tells you: “I asked the model to write “unconstitutionally” backwards, he made a mistake. This AI sucks. » What do you answer, in one or two sentences, using what you have just observed?

**Q3.** (Bonus) Your company pays for an AI provider's API (application programming interface) by token. You write your prompts in French. What budgetary conclusion do you draw from your observations?

---

### ✅ Answer key — Exercise 1

**Expected comments:**

| Entrance | Typical observation |
|---|---|
| `le chat` | About 2 tokens — very frequent words = 1 token each (the space is often included in the following token). |
| `anticonstitutionnellement` | Rare and long word → split into several fragments (e.g. `anti` / `constitution` / `nelle` / `ment` — the exact split varies depending on the tokenizer ⚠). |
| `ChatGPT` | Often 2–3 tokens (e.g. `Chat` / `G` / `PT`): recent proper nouns or upper/lower case mixtures are fragmented. |
| `2024` vs `20240115` | `2024` can be 1 token (common); `20240115` is split into arbitrary fragments (`2024` / `01` / `15` or other). Explains errors when manipulating strings of numbers. |
| `bonjour` vs `bonjuor` | The correct word = 1 token; the typo explodes into 2–4 fragments. The tokenizer only “knows” frequent sequences. |
| Business jargon | The rarer the term is in the training corpus, the more fragmented it is. |

**Q1 — Model answer:** Few tokens: everyday text, frequent words of the language, standard English. Lots of tokens: rare words, jargon, typos, sequences of numbers, code, languages ​​poorly represented in the training data.

**Q2 — Model response:** “The model does not see letters: it sees blocks of several characters (tokens). Reversing a word letter by letter requires access to individual characters that it doesn't natively have — it's a structural limitation of tokenization, not a sign of general "stupidity." On tasks where tokens are sufficient (summarizing, writing, translating), the same model excels. »**Q3 — Model answer (bonus):** French is generally divided into more tokens than English for equivalent content (the tokenizers are optimized on predominantly English-speaking corpora ⚠). Consequences: higher API cost for equal content, and context window consumed faster. Avenues: concise prompts, possibly system components in English when acceptable, and monitoring of actual token consumption.

**Success criteria:** the exercise is successful if the participant has observed for himself (1) that frequency ⇒ fewer tokens, (2) that the same content can cost very different numbers of tokens, (3) and knows how to relate the missed letter counting to the opacity of the tokens.

---

## Exercise 2 — The vector analogies game 🧭 (conceptual)

**Difficulty:** ★★☆
**Duration:** 8 minutes (6 min in pairs + 2 min of collective correction)
**Material:** paper/pen. No computer needed.
**Format:** pairs.

### Reminder of the principle

In the space of embeddings, the meaning relations are **directions**: `roi − homme + femme ≈ reine` means that the “arrow” which goes from *man* to *woman* is the same as that which goes from *king* to *queen*.

### Part A — Complete the analogies

For each line, find the missing word AND name the “direction of meaning” used (e.g.: masculine→feminine, country→capital, etc.).

1. `Paris − France + Japon ≈ ?`
2. `marcher − marche + mange ≈ ?` *(hint: think about verb forms)*
3. `chaton − chat + chien ≈ ?`
4. `pilote − avion + navire ≈ ?`
5. `Berlin − Allemagne + Italie ≈ ?`

### Part B — Create your own analogies

Invent **two** analogies of the form `A − B + C ≈ D`:
- a source of everyday language;
- from YOUR professional field (finance, law, health, marketing, etc.).

For each, name the direction of meaning.

### Part C — The trick question

`avocat − tribunal + salade ≈ ?` … Does this “equation” make sense? What does this case teach us about the limits of **static** embeddings (a unique vector per word) and about the interest of **attention** seen in progress?

---

### ✅ Answer key — Exercise 2

**Part A:**

1. **Tokyo** — direction: country → capital.
2. **eat** *(or “eat” depending on the interpretation)* — direction: conjugated form → infinitive (morphological relationship). Accepted answer: any form showing that the direction “conjugated→infinitive” applied to “eat” gives “eat”.
3. **puppy** — direction: adult → small of the animal.
4. **captain** *(or “sailor/skipper”, accept if justified)* — direction: vehicle → person who drives it.
5. **Rome** — direction: country → capital (same direction as in 1 and 5: this is the key point — THE SAME arrow works for all country/capital pairs).

**Part B—examples of valid answers:**
- Common language: `voir − vu + pris ≈ prendre` (participle→infinitive); `grand − plus grand + plus petit ≈ petit` (comparison).
- Professional: `action − dividende + obligation ≈ coupon` (instrument → associated income); `diagnostic − médecin + avocat ≈ consultation juridique` (professional → act); `prospect − marketing + recrutement ≈ candidat` (domain → target).
- **Correction criterion:** the answer is correct if the SAME relationship links A→B and D→C. Have management verbalize: this is the targeted skill.**Part C - Model answer:** The equation is shaky because "lawyer" has two meanings (profession / fruit): a **static** embedding can only give ONE vector, which mixes the two meanings - the result is a blurry point between the legal field and the food field. This is precisely what **attention** in transformers corrects: the representation of “lawyer” becomes **contextual** — pulled toward the fruit if “salad” is in the context, toward the lawyer if “court” is. Morality: embeddings = starting position; attention = adjustment to context.

**Success criteria:** the participant knows how to (1) resolve an analogy by identifying the direction, (2) construct one, (3) explain why lexical ambiguity requires attention.

---

## Exercise 3 — Triage: which stage of the pipeline? 🏗️ (app)

**Difficulty:** ★★★
**Duration:** 15 minutes (10 min in small groups + 5 min for correction) — can be given as homework if there is not enough time.
**Material:** paper/pen.
**Format:** groups of 3–4.

### Context

You are an AI consultant. Six clients explain their situation to you. For each, determine **what concept of the course** is at stake and **what answer** you give them. The concepts that can be used: *pre-training, post-training/RLHF (reinforcement learning from human feedback), fine-tuning, inference & memory bandwidth, tokenization, scaling laws.*

**Scenario 1.** A legal firm: “The generalist model writes well, but it never uses our in-house contractual formulations or our standard plan. Should we train our own model from scratch? »

**Scenario 2.** A bank: “Our chatbot responds correctly but TOO SLOWLY during peak hours. Our service provider offers to double the computing power of the servers. Good idea? »

**Scenario 3.** A startup: “Our assistant sometimes gives factually good but curt answers, sometimes bordering on unpleasant with customers. The knowledge is there, the tone is wrong. »

**Scenario 4.** An e-retailer: “The model fails to validate our product references such as “REF-88472-XL-2024”: it reverses numbers, forgets some. Yet he writes impeccable product sheets! »

**Scenario 5.** An innovation director: “Suppliers are releasing ever larger models and announcing their performance in advance. How can they promise results on a model that has not yet been trained? Deceptive marketing? »

**Scenario 6.** International customer service: “Our API costs have increased by 40% since we switched our prompts from English to French and Polish, for the same volume of requests. Is the supplier sneakily billing us? »

### For each scenario, produce:

1. The **course concept** concerned (one line).
2. The **diagnosis** in 1–2 sentences.
3. The concrete **recommendation** in 1–2 sentences.

---

### ✅ Answer key — Exercise 3

**Scenario 1 — Fine-tuning.**
*Diagnosis:* the model already has general competence (acquired during pre-training); it lacks in-house specialization – a textbook case of fine-tuning (“corporate onboarding”).
*Recommendation:* above all NO training from scratch (cost of tens to hundreds of millions ⚠, massive data required). Options by increasing cost: (a) instructions + examples in the prompt, (b) fine-tuning of an existing model on a corpus of in-house contracts.Start with (a), move to (b) if insufficient.

**Scenario 2 — Inference & memory bandwidth.**
*Diagnostic:* inference is governed by memory bandwidth (each generated token requires passing all model parameters), not by computing power. Doubling the calculation risks changing almost nothing.
*Recommendation:* instead consider: a smaller model for simple requests (fewer parameters to move = faster), faster memory hardware (HBM — High Bandwidth Memory), shorter responses, or inference optimization techniques offered by the provider. Require the service provider to diagnose the real bottleneck before paying.

**Scenario 3 — Post-training / RLHF.**
*Diagnosis:* typical problem of behavioral alignment, not knowledge: this is the role of post-training (the “finishing school”). RLHF (reinforcement learning from human feedback) shapes tone, usefulness and politeness based on human preferences.
*Recommendation:* at a client level: first correct via system instructions (expected tone, examples of correct answers); if insufficient, choose a model whose post-training better matches the usage, or refine on preferred response pairs if the provider allows it.

**Scenario 4 — Tokenization.**
*Diagnostic:* “REF-88472-XL-2024” is split into arbitrary tokens; the model manipulates opaque blocks, not characters — hence inversions and omissions on reference type strings, while the writing (task at the token level) remains excellent.
*Recommendation:* do not entrust character-by-character validation to the model alone: ​​delegate it to classic code (regular expression, exact comparison) and reserve the model for editing. General rule: anything that requires precise precision comes under the programmatic tool.

**Scenario 5 — Scaling laws.**
*Diagnosis:* not misleading marketing: the laws of scaling show that performance (measured by prediction quality) evolves regularly and predictably with parameters, data and calculation. Laboratories extrapolate these curves before training.
*Recommendation:* explain to the director that the prediction concerns aggregated statistical metrics; the appearance of specific capabilities is less predictable (emergence in stages) and advertising benchmarks always deserve verification on YOUR use cases.

**Scenario 6 — Tokenization (multilingual efficiency).**
*Diagnosis:* no hidden billing: the tokenizers, optimized on predominantly English-speaking corpora, cut French and especially Polish into more tokens of equal content (⚠ variable ratio depending on the tokenizers). Billed by token, the same volume therefore costs more.
*Recommendation:* measure the token/real word ratio per language, compress system instructions (or even keep them in English if acceptable), compare suppliers — some recent tokenizers are more efficient on European languages ​​(⚠ scalable).

**Indicative scale (for group correction):** 1 point per correctly identified concept, 1 point per exact diagnosis, 1 point per actionable recommendation — 18 points in total. From 13/18: very good mastery of the module.

---

## Exercise 4 — Sorting the web 🍷 (application · Part E-bis)**Difficulty:** ★★☆
**Duration:** 8 minutes (5 min sorting + 3 min pooling) — can be done as homework.
**Format:** in pairs.

### Context

You are a data curator at an AI laboratory. The FineWeb type pipeline seen in Part E-bis has just crawled the 5 documents below. For each one, decide: **KEEP**, **FILTER** (specifying which filter rejects it), or **DEDUPLICATE**. Justify in one sentence.

**Document 1**
> Home | Products | About | Contact | Login | Cart (0) | Legal notices | Cookie policy | Accept | Refuse | Configure

**Document 2**
> Photosynthesis is the process by which plants convert light energy into chemical energy. In chloroplasts, chlorophyll absorbs light, which sets off a chain of reactions transforming water and carbon dioxide into glucose and oxygen.

**Document 3**
> Photosynthesis is the process by which plants convert light energy into chemical energy. In chloroplasts, chlorophyll absorbs light, which triggers a chain of reactions transforming water and CO2 into glucose and oxygen.
> *(published on an aggregator, different source from Document 2)*

**Document 4**
> best vpn 2024 cheap vpn free vpn fast vpn streaming vpn netflix vpn france vpn comparison vpn promo vpn test vpn review vpn top 10 vpn

**Document 5**
> Lorem ipsum dolor sit amet, consectetur adipiscing elit. { padding: 0; margin: 0; } function initCarousel() { return true; }

### ✅ Answer key — Exercise 4

**Document 1 — FILTER (quality filter / heuristics).** Pure navigation boilerplate: very short lines, no terminal punctuation, no informative content. This is exactly what WARC + trafilatura extraction and “short line fraction” type filters eliminate — and what WET files let through (hence their 25% excess tokens).

**Document 2 — KEEP.** Informative, well-formed, autonomous text — and with high educational value: a FineWeb-Edu type classifier would rate it ≥ 3/5. This is the page profile that brings up MMLU and ARC.

**Document 3 — DEDUPLICATE.** Near copy of Document 2 (some accents and "CO2" changed): similarity well above the ~75% threshold targeted by MinHash, which compares fuzzy 5-gram fingerprints — precisely to catch the *imperfect* copies that an exact comparison would miss. We keep ONE copy (the 2nd), we remove the other.

**Document 4 — FILTER (quality filter / repetition).** SEO spam: list of keywords without sentences, massive repetition of the same term, no punctuation. Rejected by repetition and terminal punctuation filters — the very type of content that visual inspection of low-quality data reveals (“ads, lists of keywords”).

**Document 5 — FILTER (C4 type filters).** “Lorem ipsum” and braces `{` are two explicit filters inherited from C4: filler text and code/CSS do not teach anything useful to a LANGUAGE model in a general corpus (the code is trained separately, on dedicated corpora).

**Bonus question for pooling:** “Why not just ask humans to sort?” » → At 15,000 billion tokens, this is physically impossible: hence the heuristics, then the next step — AI classifiers trained on a small annotated sample (500,000 pages for FineWeb-Edu) and applied to the entire corpus.

---*Exercises — Applied AI, Intermediate Level, Session 1. © Yann Isola. Version 1.0.*