Teacher Guide — Session 1: Fundamentals: Transformers & Tokenization

Program : Applied AI — Intermediate Level Instructor: Yann Isola Duration : 2 hours (120 minutes) Module covered: Module 1 — Foundations

1. Session overview

Educational objectives

At the end of this session, each participant should be able to:

Situating transformers in the history of AI : explain the transition from rule-based systems → classic machine learning → deep learning → transformers (2017).
Explain tokenization : understand that a model reads fragments of ~3-4 characters (“tokens”), and not words, and deduce why certain surprising failures occur (e.g.: counting the letters of a word).
Describe embeddings : each token becomes a vector; close meanings occupy close positions (“king − man + woman ≈ queen”).
Explain the attention mechanism : each token dynamically evaluates the relevance of all other tokens.
Describe the training pipeline : pre-training (university) → post-training / RLHF (finishing school) → fine-tuning (company integration).
Distinguish training and inference : understand that inference speed is limited by memory bandwidth, not computing power.
Understanding scaling laws : Performance gains are predictable with more data, calculation and parameters.

Prerequisites

No programming skills required.
Have already used an AI assistant (ChatGPT, Claude, Gemini or equivalent) at least a few times.
Curiosity and desire to understand “what’s going on under the hood”.

Necessary equipment

Video projector + session slides (see slides/slides.md ).
Interactive web page (webpage/index.html ) — works offline, open in a modern browser.
Ideally: each participant with a laptop or smartphone for practical exercises.
Worksheets printed or shared digitally (exercises/exercises.md ).
End of session quiz (quiz/quiz.md ).

Central message of the session

“A language model is not magic: it is a statistical machine that cuts text into fragments, transforms them into numbers, measures the relationships between these numbers, and predicts the next fragment. Understanding these four steps means understanding 80% of the strengths and weaknesses of modern AI. »

Repeat this idea in different forms throughout the session. This is the common thread.

2. Rolled out minute by minute

Hourly	Duration	Sequence	Support
0:00 – 0:05	5 mins	Welcome, objectives, session plan	Slides 1–2
0:05 – 0:18	13 mins	Part A — History: from rules to transformers	Slides 3–7
0:18 – 0:38	20 mins	Part B — Tokenization	Slides 8–12 + web demo
0:38 – 0:47	9 mins	Exercise 1: Exploring tokenization	Worksheet
0:47 – 1:00	13 mins	Part C — Embeddings	Slides 13–16 + web demo
1:00 – 1:05	5 mins	☕ Short break	—
1:05 – 1:19	14 mins	Part D — Caution	Slides 17–20 + web demo
1:19 – 1:25	6 mins	Exercise 2: The game of analogies	Worksheet
1:25 – 1:39	14 mins	Part E — Training pipeline & inference	Slides 21–25
1:39 – 1:49	10 mins	Part E-bis — Behind the scenes: the data pipeline (Karpathy)	Slides 25a–25c
1:49 – 1:53	4 mins	Part F — Scaling laws	Slides 26–27
1:53 – 2:00	7 mins	Quick quiz + Exit Tickets + announcement Session 2	Slides 28–30

Flexibility rating: If you fall behind, Exercise 2 can be shortened to 5 minutes (do only 2 analogies as a group) and Exercise 3 becomes homework. Part E-bis can be compressed to 6 minutes (points 1, 4, 7 and 8 only). NEVER sacrifice Part E (training pipeline): this is the part most reused in subsequent sessions.

3. Detailed teaching notes by sequence

0:00 – 0:05 | Reception and framing

What to say:

Welcome. Present the contract for the session: “In 2 hours, you will be able to explain to a colleague how a language model really works — without magic, without unnecessary jargon. »
Announce the golden rule of the course: each acronym will be explained . If a participant hears a term that is not explained, they have the right (and duty) to raise their hand.

Point of attention: quickly assess the level of the room. Ask by show of hands: “Who uses an AI assistant at least once a week?” » This allows you to calibrate the examples.

0:05 – 0:18 | Part A — History: from rules to transformers

Key concepts: rule-based systems → classic ML (Machine Learning) → deep learning → transformers (2017).

What to say - the narrative framework in 4 eras:

The period era (1950s–1990s). “We believed that it was enough to write all the rules by hand. Concrete example: a rules-based spam filter — “if the email contains the word FREE in capital letters, it’s spam”. Problem: Spammers write “GR4TUIT” and the rule fails. Handwritten rules don’t scale. »
Classical machine learning (1990s–2010s). “Change of philosophy: instead of writing the rules, we show examples and the machine discovers the rules itself. The spam filter learns from 100,000 emails labeled “spam/not spam”. But a human must still choose the characteristics to observe (word frequency, presence of links, etc.). »
Deep learning (2012–2017). “Deep neural networks learn themselves which features to observe. 2012: AlexNet crushes the ImageNet image recognition competition. But for language, a problem persists: the networks of the time (RNN — Recurrent Neural Networks) read the text word by word, sequentially, and “forget” the beginning of long sentences. And they are slow to train, because they are impossible to parallelize. »
The Transformers (2017). “A Google research article titled Attention Is All You Need (“Attention is all you need”) proposes an architecture that solves both problems at once: (a) the attention mechanism allows each word to “look” at all the other words in the sentence, even very distant ones — no more forgetting; (b) everything is calculated in parallel — thousands of graphics processors (GPUs — Graphics Processing Units) can be used at the same time. It is this parallelization that has allowed the explosion in the size of the models. »

To emphasize (write it on the board):

Transformers = attention (long memory) + parallelism (hardware scaling)

Frequently asked question #1: “Why is it called “transform”? » Answer: The name comes from the 2017 article — the architecture “transforms” an input sequence into an output sequence. No connection with the robots in the film. Anecdote that relaxes the room.

Frequently asked question #2: “ChatGPT, is it a transformer? » Answer: yes. GPT stands for Generative Pre-trained Transformer. The “T” is in the name. Claude, Gemini, Grok, Muse Spark (ex-Llama): all transformers, with variations. ⚠ Model names change quickly (in July 2026: GPT-5.6, Grok 4.5, Muse Spark 1.1, etc.) — what doesn't change is the transformative architecture under the hood.

Transition to Part B:

“Very well, now we know When And Why the transformers have arrived. But concretely, when you type a sentence in ChatGPT, what is the very first thing that happens? The model does not read your words. He doesn't even read your letters. He reads something else. Here we go for tokenization. »

0:18 – 0:38 | Part B — Tokenization

Key concept: the model cuts the text into tokens — fragments of around 3 to 4 characters on average (⚠ indicative average for English; in French, often a little more tokens per word). They are neither words nor letters.

What to say:

Definition by example. Show the web demo (“Tokenizer” tab). Type live: L'intelligence artificielle transforme nos métiers. Show the colored cutout. Point out that “intelligence” may be cut into several pieces, while “the” or “our” fits into a single token.
Why tokens and not words? “Two reasons. One: vocabulary. There are millions of words, proper nouns, typos, made-up words — it's impossible to store them all. With ~100,000 fragments (⚠ order of magnitude, varies depending on the model), we can reconstruct any text, even “Schtroumpfissime” or “COVID-19”. Two: efficiency. Frequent words become a single token, rare words are assembled from fragments. It's like Lego: few basic bricks, infinite constructions. »
Why not letters? “It would be possible, but the sequences would become very long (a sentence of 50 words = ~300 letters = ~300 calculation steps instead of ~70 tokens). Tokens are the compromise between manageable vocabulary and short sequences. »
The Crucial Point — Bizarre Failures Explained. This is THE “aha” moment of the game. “Ask a model: how many “r”s in the word “strawberry”? Models are regularly wrong. For what ? Because the model never sees s-t-r-a-w-b-e-r-r-y. Maybe he sees [str][aw][berry] — three opaque blocks. Asking it to count the letters is like asking you to count the “r”s in a word written in Chinese characters: you see the blocks, not the internal letters. » Same logic for: reversing a word, word games letter by letter, arithmetic on long numbers (cut into arbitrary tokens).

Additional example to give orally: the dates. “2024” can be a unique token, but “20241231” will be split arbitrarily — hence surprising errors when manipulating strings of numbers.

Frequently asked question #3: “So the model is bad at spelling? » Answer: no — he has seen so much text that he statistically knows the spelling of the words. But tasks that require view letters individually (count, reverse, spell backwards) are structurally difficult. Important nuance: recent models sometimes get around the problem by “reasoning” step by step, by first spelling the word. The structural limitation remains, circumvention strategies are improving. (⚠ state of models subject to change)

Frequently asked question #4: “Does French consume more tokens than English? » Answer: yes, generally — tokenizers are mostly optimized for English. The same text in French can cost ~1.2 to 2x more tokens (⚠ depends on the tokenizer and evolves). Practical consequence: API cost and context window consumed faster in French.

Transition to Exercise 1:

“Enough theory — now you're going to play with a tokenizer yourselves and see these oddities with your own eyes. »

0:38 – 0:47 | Exercise 1 — Exploring tokenization

See the worksheet (exercises/exercises.md , Exercise 1). Participants use the web demo (“Tokenizer” tab) or an online tokenizer.

Role of the instructor during the exercise:

Circulate around the room. Typical blocks: “I don’t see a difference” → suggest testing rare vs. frequent words, numbers, words with mistakes.
At 8 minutes, do a 2-minute pool session: ask 2–3 participants their most surprising discovery.

Expected restitution: frequent words = 1 token; rare words = several tokens; numbers and typos blow up the token count.

0:47 – 1:00 | Part C — Embeddings

Key concept: each token is converted into a vector (a list of numbers). Tokens of close meaning occupy close positions in this space. Meaning becomes geometry.

What to say:

The problem to be solved. “Computers don't calculate with words, only with numbers. It is therefore necessary to convert each token into numbers. But not just any way: we want the conversion preserve the meaning . »
The map analogy. “Imagine a geography map where we place words instead of cities. “Cat” and “dog” would be neighbors, like Lyon and Villeurbanne. “Cat” and “carburetor” would be poles apart. An embedding is exactly that: an address in a space of meaning. Except that space does not have 2 dimensions like a map, but hundreds or even thousands (⚠ order of magnitude depending on the models). Each dimension captures a nuance: animate/inanimate, masculine/feminine, concrete/abstract, technical/everyday…”
The famous demonstration: king − man + woman ≈ queen. “As they are numbers, we can do arithmetic on the MEANING. Take the “king” vector, remove the “man” vector, add the “woman” vector: you land near the “queen” vector. The direction “masculine → feminine” is a coherent arrow in space. Same: Paris − France + Italy ≈ Rome. The “capital of” relation is also a geometric direction. » Insist: this result comes from work on word embeddings (word2vec, 2013) and illustrates the general principle — in modern transformers, the embeddings are contextual and richer, but the intuition remains valid.
Where do these positions come from? “No one placed them by hand. They emerge from training: words that appear in the same contexts end up being neighbors. “We recognize a word by its associations” – that’s literally the principle (distributional hypothesis in linguistics). »
Web demo. Open the “Embedding space” tab: show the three clouds (animals, colors, professions). Drag the view, hover over the points. Point out that “veterinarian” sits between the professions cloud and the animals cloud — embeddings capture multiple memberships.

Frequently asked question #5: “Is it like a thesaurus? » Answer: more powerful. A thesaurus is binary (synonymous or not). The space of embeddings is continuous: it captures degrees of proximity, directions of meaning (genre, time, size, etc.), and relationships that no dictionary lists (“Paris is to France what Rome is to Italy”).

Professional application to mention (prepares for future sessions): “Semantic search in business works like this: we convert your documents AND your question into embeddings, then we look for documents whose vector is close to that of the question — even if they do not use any words in common. Searching for “billing problem” finds a document that says “payment anomaly”. This is the basis of RAG (Retrieval-Augmented Generation), which we will see in detail in a dedicated session. »

Transition to break:

“Let’s recap: the text is divided into tokens, each token becomes a point in a space of meaning. But an isolated point is not enough — “avocado” the fruit and “avocado” the profession, same word, different meanings. How does the model decide? Answer in 5 minutes, after the break: this is the attention mechanism. »

1:05 – 1:19 | Part D — Caution

Key concept: each token dynamically evaluates the relevance of all other tokens in the context, and adjusts its representation accordingly.

What to say:

The problem of ambiguity. Write on the board: “The avocado eats an avocado. » “The word “lawyer” appears twice — a profession, a fruit. How does the model know? Thanks to the context: “eat” points to the fruit for the second, “L” + subject position points to the person for the first. Attention is the mechanism that formalizes this “look towards the context”. »
Intuitive mechanics. “For each token, the model asks the question: what other tokens in the sentence help me clarify my meaning here, now? Each token distributes “attention weights” — a percentage of importance — over all other tokens. Then it updates its own representation by mixing the information from the tokens deemed relevant. And this is done in parallel for all the tokens, over dozens of successive layers. »
The example of the pronoun. " Sentence : The trophy does not fit in the suitcase because it is too big. — “he” designates what? The trophy. Change “large” to “small”: because she is too small — now it’s the suitcase. Solving “he/she” requires weighing the entire context. This is exactly what attention does, and this is what pre-2017 architectures missed on long sentences. »
Web demo. “Attention” tab: Show the heat map on the example sentence. Point out the dark boxes: “he” looks strongly towards “trophy”. Click on different words to visualize their distribution of attention.
Link to Part A. “Remember the table: attention = long memory, parallelism = scale . You have just seen long memory in action. And as each token calculates its attention independently of the others, everything is done in parallel – hence the massive exploitation of GPUs. The circle is closed. »

Frequently asked question #6: “Is it understanding or calculation? » Honest answer to give: “It’s calculation — matrix multiplications — whose result functionally resembles contextual understanding. The course will not resolve the philosophical debate; what matters to us: the mechanism is powerful, measurable, and explains the capabilities AND the limits. »

Educational trap to avoid: Do NOT go into Query/Key/Value details at this level. If an advanced participant asks, respond: “Great question — technically, each token issues a “query” and “keys,” and their compatibility gives the attention weights. See me at the break for details, or see the appendix. » Don't waste the remaining 90% of the room.

Transition to Exercise 2:

“A little game before moving on to training the models: you will manipulate the arithmetic of meaning yourselves. »

1:19 – 1:25 | Exercise 2 — The game of analogies

See exercise sheet, Exercise 2. In pairs, 6 minutes + 2 minutes of collective correction.

Role of the instructor: start the clock, then correct as a group by asking the pairs to explain Why the analogy works geometrically (what “direction” of meaning is at stake).

1:25 – 1:39 | Part E — Training pipeline & inference

Key concepts: pre-training → post-training (including RLHF) → fine-tuning; then training/inference distinction; Inference is limited by memory bandwidth.

What to say — the analogy of training a professional:

Pre-workout = college. “We make the model read an immense part of the available text: thousands of billions of tokens (⚠ orders of magnitude in constant growth). The task is disarmingly simple: predict the next token . “The cat sleeps on the ___” → “sofa”. By practicing billions of times, the model is forced to absorb grammar, facts, styles, basic reasoning — because all of this helps predict better. Result: a very learned but rough model – like a brilliant graduate who has never spoken to a client. He completes the text, he does not “respond”. Cost: tens to hundreds of millions of dollars of calculation for the largest models (⚠ figures changing rapidly). »
Post-training / RLHF = finishing school. “RLHF: Reinforcement Learning from Human Feedback. Annotators compare pairs of responses (“which is more useful, more honest, less dangerous?”) and the model is adjusted to produce more responses of the preferred type. This is what turns a completion engine into an assistant: tone, format, refusal of dangerous requests, ability to follow instructions. Analogy: finishing school — the knowledge is already there, we learn the manners, the format, the ethics. »
Fine-tuning = business integration. “Additional adjustment to an area or task: YOUR firm's legal vocabulary, YOUR brand's tone, YOUR reporting format. Like a brilliant new hire who is trained in internal procedures during his onboarding. Much less expensive than pre-training — we start with an already trained model. »

Summary table to project (slide 23):

Stage	Analogy	What we learn	Cost order
Pre-workout	University	Language, facts, reasoning	Very high ⚠
Post-workout (RLHF)	Finishing school	Usefulness, tone, security	Average ⚠
Fine tuning	Onboarding in business	Job specialization	Low ⚠

Inference — and the memory bottleneck. “Once trained, using the model is called inference : your prompt enters, the tokens come out one by one. Counterintuitive point: what limits speed is not computing power, it is memory bandwidth . For what ? To generate EACH token, all of the model parameters – tens or hundreds of gigabytes – must be transferred from memory to the calculation units. The calculation itself is fast; it’s the moving of the data that takes time. Analogy: a starred chef (calculation) who must fetch each ingredient from a warehouse on the other side of the city (memory). The boss is never the problem; the shuttle, yes. » Practical consequence to be given: “this is why chip manufacturers are fighting over memory bandwidth (HBM — High Bandwidth Memory), and why small models respond faster: fewer parameters to move per token. »

Frequently asked question #7: “Does the model learn from my conversations? » Answer: “Not during inference — the parameters are frozen. Your conversation fits in the context window, a temporary working memory. Depending on the providers and your settings, your conversations may however be collected for FUTURE training (⚠ policies vary depending on providers and offers — important point in business, we will come back to this in the governance/security session). »

Transition to Part E-bis:

“We have just seen HOW we coach. A question that no one asks and yet which decides everything: we train him on WHAT, exactly? Let's go down to the cellar. »

1:39 – 1:49 | Part E-bis — Behind the scenes: the data pipeline (according to Karpathy)

Key concepts: where the pre-training data actually comes from; the CommonCrawl pipeline → extraction → filtering → deduplication; “good data” is measured by ablations, not by eye; the quality of the dataset is the quality ceiling of the model.

Why this part. In his reference video Deep Dive into LLMs like ChatGPT (around 6 hours, 2025), Andrej Karpathy — co-founder of OpenAI, former AI director of Tesla — starts not with the architecture, but with the data . Its entire “pre-training” part is based on public work from HuggingFace: the FineWeb dataset and its technical report. It is one of the rare windows open to what laboratories keep secret: the data recipe. This part gives your participants what 99% of ChatGPT users don't know.

What to say — the journey from a web page to the template:

Source: CommonCrawl. “Where does the text come from? Almost everyone starts from the same source: CommonCrawl, a non-profit organization that has been crawling the web since 2007 and publishing a new crawl every 1 to 2 months — around 200 to 400 TiB of raw HTML per crawl, or around 2.7 billion web pages for a recent crawl (⚠ 2024 figures, growing). FineWeb is built on 96 crawls. Analogy: CommonCrawl is the grape harvest — everything the vine gave, magnificent clusters and rotten grapes mixed together. The work that follows is winemaking. » (The report is also called decanting the web — decant the web.)
Extract text from HTML. “A web page is 90% noise: menus, cookie banners, advertisements. First step: extract the useful text from the raw HTML (WARC format) with a dedicated tool. Discovery of FineWeb: the “ready-made” text version provided by CommonCrawl (WET files) gives a dataset 25% BIGGER… and clearly WORSE: the excess is navigation boilerplate. First counterintuitive lesson: more data ≠ better . »
Filter. “Then, we throw away — massively: URL blocklist (adult content), language classifier (we only keep the detected English pages with a score ≥ 0.65 for FineWeb v1), heuristic quality filters: too many repeated lines, lines too short, pages without punctuation, lists of SEO keywords… After this first sorting: around 36,000 billion tokens. »
Deduplicate — and the surprise. “The web is full of copies: mirrors, aggregators, template pages. We deduplicate with MinHash, a fuzzy fingerprinting technique that identifies documents that are ~75% or more similar. And there, a result which surprised the authors themselves: deduplicating the ENTIRE corpus globally (result: 4,000 billion tokens) gives a WORSE model than deduplicating each crawl independently (20,000 billion tokens). For what ? Global deduplication mostly deleted good recurring content and mechanically over-represented old, poor quality data — on the oldest crawl tested, the 10% “retained” was worse than the 90% deleted! Second lesson: in data, intuition is not enough — we must measure . »
Measure: the ablation method. “Precisely, how do we know that a version of the dataset is “better”? You can't see it with the eye. The method: train small twin models (1.8 billion parameters, ~28 billion tokens each) on each variant of the dataset — only the data changes — then compare them on early signal benchmarks (HellaSwag, MMLU, ARC, PIQA, etc.). Each pipeline choice was validated in this way. It’s experimental science applied to data: hypothesis → ablation → measurement. »
The result: FineWeb. “At the end of the pipeline: 15 trillion tokens, 44 TB of text — one of the best open pre-training datasets. Order of magnitude to be felt: it’s tens of millions of times War and Peace . »
FineWeb-Edu — AI that sorts for AI. “Last tip, the most modern: what if we only kept the pages educational ? We ask a large model (Llama-3-70B) to rate 500,000 pages from 0 to 5 on their educational value, then we train a small, fast classifier on these annotations, and we apply it to the 15,000 billion tokens (cost: ~6,000 hours of H100 GPU). Keeping the scores ≥ 3: FineWeb-Edu, 1,300 billion “very educational” tokens, which outperforms all open web datasets on knowledge benchmarks (MMLU, ARC, OpenBookQA). This is exactly the technique – until now secret – used for Llama 3 and Phi-3: one AI filters data from the next . »
Karpathy's message. “Remember his formula: an LLM is a lossy compression of internet . When ChatGPT responds to you, it statistically restores this decanted web. Direct consequence for you: the quality of the dataset IS the quality ceiling of the model — no architecture makes up for poor data. And this is also true on YOUR scale: your RAG and fine-tuning projects will obey the same law. Garbage in, garbage out — on a web scale. »

Point of vigilance to mention if time permits: recent crawls contain more and more AI-generated text (the authors measure this by the frequency of ChatGPT catchwords like delve Or rich tapestry , rising sharply since 2023). On a small scale this does not degrade performance, but the question of web self-contamination remains open (⚠ subject in progress).

Frequently asked question n°7-bis: “So my web pages / my posts are in ChatGPT? » Answer: “If your pages are public and crawlable, most likely yes, to some extent — that’s the whole point of ongoing legal debates over copyright and robots.txt files that now block AI crawlers (⚠ shifting legal landscape). We will come back to this in the governance session. »

Sources: Andrej Karpathy — Deep Dive into LLMs like ChatGPT (2025); Penedo et al., FineWeb: decanting the web for the finest text data at scale (HuggingFace, NeurIPS 2024) — https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1

Transition to Part F:

“Final piece of the puzzle: why do models get better every year? Is it luck? No — that’s predictable. Literally. »

1:49 – 1:53 | Part F — Scaling laws

Key concept: the performance of the models improves regularly and predictable when we increase three ingredients: data, calculation, parameters.

What to say:

“Remarkable discovery of the 2020s: if you plot the performance of a model as a function of its size (parameters), the amount of training data and the calculation invested, you get regular curves. We can predict the performance of a 10× larger model before training it. This is what gave laboratories the confidence to invest billions: they do not bet blindly, they extrapolate a curve. »
Analogy: “It’s like building bridges: you don’t build a 2 km bridge and hope it will hold – the material strength equations predict it. Scaling laws are the resistance equations of AI. »
Nuances to give (intellectual honesty): “Three clarifications. One: the curves predict a statistical metric (the prediction quality of the next token), not directly the appearance of specific capabilities — some capabilities seem to emerge in stages. Two: all three ingredients must grow together — a giant model underpowered by data is wasteful (this is the so-called “Chinchilla” lesson, named after a 2022 research model). Three: the debate remains open on the indefinite continuation of these curves, and the industry is exploring other axes, such as calculation at the time of inference (⚠ rapidly evolving landscape). »

Frequently asked question #8: “So bigger = always better? » Answer: “Better on the benchmarks (test benches), yes, in trend. But not always relevant : A 10x larger model is slower and more expensive for inference — remember memory bandwidth. In business, the right model is the smallest that achieves the level of quality required for the task. »

1:53 – 2:00 | Closing: Quick quiz, Exit Tickets, teaser

Distribute or show the quiz (quiz/quiz.md ) — in session, select 4–5 questions to do freehand; the complete quiz can be done independently.
Complete Exit Tickets (below) on paper or form — 3 minutes.
Teaser Session 2: “Now you know HOW the machine works. Next time: how to TALK to him to get the most out of him — the art and science of prompting. »

4. Exit Tickets (5 verification questions)

To be completed individually at the end of the session. Objective: verify key knowledge and detect misunderstandings before session 2.

ET-1. Explain in one or two sentences why a language model can go wrong when counting the letters in a word.

Model answer: The model does not see the letters: the text is divided into tokens (fragments of ~3-4 characters on average) which are opaque blocks for it. Counting letters requires access to individual characters that its internal representation does not directly provide.

AND-2. What does “king − man + woman ≈ queen” mean and what does this tell us about embeddings?

Model answer: Words are represented by vectors (lists of numbers); we can therefore do arithmetic on it. This equation shows that meaning relationships (here masculine→feminine) correspond to coherent directions in the space of embeddings: the meaning is encoded geometrically.

ET-3. Name the two problems of pre-2017 architectures that transformers solved, and by what means.

Model answer: (1) Forgetting long-distance dependencies in sentences → solved by the attention mechanism, where each token weighs the relevance of all the others. (2) The impossibility of parallelizing training (word-by-word sequential processing) → resolved by a fully parallelizable architecture, allowing massive exploitation of GPUs.

ET-4. Match each stage of the pipeline to its analogy and function: pre-training, post-training/RLHF, fine-tuning.

Model answer: Pre-training = university: massive acquisition of language and knowledge by prediction of the next token. Post-training/RLHF (reinforcement learning from human feedback) = finishing school: transformation into a helpful, polite and safe assistant thanks to human preferences. Fine-tuning = corporate onboarding: specialization in a specific area or format.

ET-5. True or false, and justify: “To accelerate the inference of a model, you especially need more powerful processors in calculation. »

Model answer: Fake. Inference is mainly limited by memory bandwidth: for each token generated, all of the parameters must pass from memory to the calculation units. It is this transfer, not the calculation, that constitutes the bottleneck — hence the importance of high-speed memory (HBM) and the speed advantage of small models.

5. Tips for the instructor

The “strawberry” moment is your best ally. Failing to count letters is counterintuitive and memorable — it’s the emotional hook of the session. If possible, demonstrate LIVE with a model (have a backup screenshot ready in case the model succeeds: newer models succeed more and more often ⚠ — in this case, explain that the model learned to work around by spelling, the structural limit remaining).
Make analogies, don't multiply them. An analogy by concept, held from start to finish: geography map (embeddings), university/finishing/onboarding (pipeline), leader and warehouse (bandwidth). Too many competing analogies confuse memorization.
Physically mark the red wire. At the end of each game, return to the diagram: text → tokens → vectors → attention → prediction of the next token. At the end of the session, the room should be able to recite this string.
Manage technical profiles. There may be a developer who wants to talk about softmax and multi-head attention. Magic formula: “Excellent question, it's beyond the scope of this session — come see me during the break. » Protect the rhythm of the group.
The numbers date, the structures last. Each time you cite a figure (vocabulary size, training cost, ratio of FR/EN tokens), indicate orally that it is a dated order of magnitude (the ⚠ on the slides are there for that). On the other hand, emphasize that the mechanisms (tokenization, embeddings, attention, pipeline) are stable: it is the lasting investment of the participants.
Time the web demo. The interactive page is captivating — it’s a risk. Set yourself a maximum of 3 minutes per tab for plenary demonstration; free exploration time belongs to exercises.
Anticipate the business question. Someone will ask “what use does it actually have for me?” ". Prepared response: “Three immediate benefits: (1) you will know why AI fails on certain tasks and how to reformulate; (2) you will understand token billing and context window consumption; (3) you will be able to read supplier announcements – model size, fine-tuning, inference speed – without being dazzled by marketing. »