Slides — Session 1: Foundations: Transformers & Tokenization

Program: Applied AI — Intermediate Level · Instructor: Yann Isola
Total: 30 slides · Duration: 120 minutes
Palette: ink #1A2230 · teal #0F7A6C · copper #B4612A · light teal #E9F6F3 · background #F4F7F6
Visual convention: titles in teal on #F4F7F6 background; “to remember” frames in light teal with copper edging; dated figures bear the marker .

Slide 1 — Title

Duration: 1 min

Content:

• Applied AI — Intermediate Level
• Session 1: Foundations: Transformers & Tokenization
• Yann Isola

Visual: ink background #1A2230, white title, teal subtitle. Discreet pattern of colored “tokens” (small rounded teal/copper rectangles) floating in the background — announces Demo 1.

Speaker Notes: Welcome. A framing sentence: “In 2 hours, you will be able to explain to a colleague how a language model really works — without magic.”

Slide 2 — The session contract

Duration: 2 mins

Content:

• 7 acquired in 2 hours: history · tokens · embeddings · attention · training pipeline · inference · scaling laws
• Golden rule: all acronyms will be explained — otherwise, raise your hand
• 3 interactive demos + 2 exercises + 1 quiz

Visual: horizontal frieze of the 7 acquired, each block in light teal, numbered. Copper freehand pictogram next to the golden ruler.

Speaker Notes: Show of Hands Poll: “Who uses an AI assistant every week?” — calibrates the examples. Announce the mid-term break.

Slide 3 — Part A · The period era (1950s–1990s)

Duration: 3 mins

Content:

• The idea: write all the rules by hand
• Example: spam filter — “if the email contains FREE → spam”
• Failure: “GR4TUIT” passes. Manual rules don't scale.

Visual: if/then organization chart drawn in a “whiteboard” style, with an email “GR4TUIT!!!” which passes through the filter — copper cross on the filter.

Speaker notes: Insist: failure is not the intelligence of engineers, it is the combinatorics of reality. Each added rule creates exceptions.

Slide 4 — Classical machine learning (1990s–2010s)

Duration: 3 mins

Content:

• Reversal: we show examples, the machine discovers the rules
• ML = Machine Learning
• The filter learns from 100,000 labeled emails
• Limit: a human still chooses what to observe (word frequency, links, etc.)

Visual: two-column diagram — “Before: rules → program” vs “After: examples → rules learned”. Copper reversal arrow.

Speaker notes: This is THE philosophical shift in the entire field: from explicit programming to learning by example. The rest of the course follows from this.

Slide 5 — Deep learning (2012–2017)

Duration: 3 mins

Content:

• Deep learning: the network also learns what to observe
• 2012: AlexNet dominates the ImageNet image competition
• But for language, two obstacles persist:
• RNNs (Recurrent Neural Networks) read word for word → forgetting the beginnings of long sentences
• sequential processing → impossible to parallelize, slow training

Visual: chain of wagons (words) read one by one; the first wagons fade into gray (oblivion).A copper stopwatch symbolizes slowness.

Speaker Notes: Take 20 seconds on the wagon metaphor: when the RNN gets to word 50, word 3 is diluted. Ask the room: “How would you not forget anything?” → natural transition.

Slide 6 — 2017: “Attention Is All You Need”

Duration: 3 mins

Content:

• Google article, 2017 — birth of transformer
• Two problems solved at once:

1. Attention → each word “looks” at all the others, even distant ones
2. Parallelism → everything is calculated at the same time → massive exploitation of GPUs (Graphics Processing Unit, graphics processor)

• GPT = Generative Pre-trained Transformer (pre-trained generative transformer)

Visual: stylized cover of the article + diagram: all the words of a sentence linked together by teal arcs (attention), and a stack of GPUs below (parallelism).

Speaker notes: Relaxing anecdote: no connection with the robots in the film — “transform” = transform a sequence into a sequence. Claude, Gemini, Grok, Muse Spark (ex-Llama): all transformers.

Slide 7 — Takeaways (Part A)

Duration: 1 min

Content:

Transformers = attention (long memory) + parallelism (hardware scaling)

• Rules → examples → learned characteristics → attention: 70 years of increasing delegation to the machine

Visual: full screen light teal frame with formula, copper border. Minimalist timeline at the bottom: 1950 · 1990 · 2012 · 2017.

Speaker Notes: Also write it physically on the board — we'll come back to this in Part D. Transition: “When you type a sentence, what is the VERY first thing that happens? The model does not read your words or your letters…”

Slide 8 — Part B · What is a token?

Duration: 3 mins

Content:

• The model cuts the text into tokens: fragments of ~3–4 characters on average
• Neither words nor letters
• Frequent words = 1 token · rare words = several fragments
• Typical vocabulary: ~100,000 fragments

Visual: the sentence “Artificial intelligence is transforming our jobs.” cut into rounded colored blocks (alternating teal/copper/ink on light backgrounds) — reproduce the rendering of Demo 1.

Speaker notes: Immediately switch to the interactive page (Tokenizer tab) and type the sentence live. Live is better than slide.

Slide 9 — Why tokens and not words?

Duration: 2 mins

Content:

• Vocabulary problem: millions of words, proper nouns, mistakes, neologisms — impossible to store everything
• Lego solution: few bricks, infinite constructions — “Smurffissime” = assembly of known fragments
• Why not letters? Sequences too long (50 words ≈ 300 letters vs ~70 tokens) → too many calculation steps

Visual: stylized Lego bricks (rounded rectangles) which come together to form a rare word. Length comparison: three bars (letters / tokens / words) with the token in the “golden middle” position highlighted in teal.

Speaker Notes: The token is an engineering compromise: manageable vocabulary × short sequences. Not a linguistic truth — a practical choice.

Slide 10 — The “strawberry” mystery

Duration: 4 mins****Content:

• Trick question: “How many “r”s in strawberry?” → frequent model errors
• The model sees: [str][aw][berry] — three opaque blocks, never the 10 letters
• Same cause: reversing a word, spelling backwards, manipulating “REF-88472-XL-2024”

Visual: large word “strawberry” displayed twice: at the top letter by letter (human vision), at the bottom in 3 blocks locked with padlocks (model vision). Copper arrow: “what you see / what he sees”.

Speaker Notes: KEY MOMENT of the session — do the demo live if possible; Backup screenshot ready. If the model succeeds: explain that it circumvents by spelling first (recent models learn strategies ) — the structural limit remains. Analogy: counting the “r”s in a word written in Chinese characters.

Slide 11 — Practical consequences of tokens

Duration: 2 mins

Content:

• API billing: you pay by token — French costs ~1.2 to 2x more tokens than English for equal content
• Context window: consumed more quickly in “expensive” languages in tokens
• Business rule: precision to the nearest character (references, counts) → classic code, not the model alone

Visual: two piles of coins (English vs French) of different heights, made of copper. Window icon that fills.

Speaker notes: This is the first “business” outcome of the session — budget managers are waking up here. API = application programming interface (remember orally).

Slide 12 — Instructions Exercise 1: Safari tokens

Duration: 1 min (launch) + 10 min (exercise)

Content:

• Open the interactive page · Tokenizer tab
• Test the 6 entries of the exercise sheet, note the number of tokens
• 8 min of exploration + 2 min of sharing

Visual: screenshot of Demo 1 + the 6 entries listed in light teal.

Speaker Notes: Circulate. Typical blockage: “I don’t see anything special” → suggest rare vs. frequent words, numbers, mistakes. Pooling: 2–3 surprising oral discoveries.

Slide 13 — Part C · From tokens to numbers: embeddings

Duration: 3 mins

Content:

• Computers only calculate numbers → each token becomes a vector (list of numbers)
• Requirement: the conversion must preserve the meaning
• Embedding = an address in a space of meaning

Visual: a “cat” token which transforms into a column of numbers [0.21, -1.4, 0.88, …], then into a point on a map. Three steps connected by teal arrows.

Speaker notes: Launch the map analogy: “cat” and “dog” neighbors like Lyon and Villeurbanne; “cat” and “carburetor” are polar opposites.

Slide 14 — A space with a thousand dimensions

Duration: 2 mins

Content:

• Space does not have 2 dimensions but hundreds to thousands
• Each dimension captures a nuance: animate/inanimate, masculine/feminine, concrete/abstract, technical/everyday…
• Nobody puts words by hand: the positions emerge from training
• “We recognize a word by its frequentations” (distributional hypothesis)

Visual: 2D map with 3 labeled clouds (teal animals, copper colors, ink professions) — resume Demo 2. Corner mention: “2D projection of a space with ~1000 dimensions ”.

Speaker Notes: Switch to Demo 2: drag view, hover over points.Show “veterinarian” between professions and animals — multiple membership.

Slide 15 — The arithmetic of meaning: king − man + woman ≈ queen

Duration: 4 mins

Content:

• Numbers → we can calculate on the meaning
• roi − homme + femme ≈ reine: the “masculine→feminine” direction is a consistent arrow
• Paris − France + Italie ≈ Rome: “capital of” is also a direction
• Historical result of word embeddings (word2vec, 2013) — the intuition still holds in transformers (embeddings that have become contextual)

Visual: parallelogram: four points (king, queen, man, woman), two parallel copper arrows man→woman and king→queen. Animation of the appearance of arrows (use the “Animate” button in Demo 2).

Speaker Notes: Trigger Demo 2 animation live. Emphasize: no one programmed this — geometry emerges from co-occurrence statistics.

Slide 16 — Professional application: semantic search

Duration: 3 mins

Content:

• Documents AND questions converted into embeddings → we look for close vectors
• “billing problem” finds “payment anomaly” — zero words in common
• Basis of RAG (Retrieval-Augmented Generation, generation augmented by recovery of information) → dedicated session later
• Limit: “avocado” (fruit? profession?) — a single static vector mixes the senses…

Visual: diagram: question → point teal; documents → point cloud; copper proximity circle around the documents found. At the bottom, the word “lawyer” torn between and .

Speaker Notes: The “advocate” point is the scripted transition to attention. Question to the room: “how to decide?” → answer after the break. Announce: break 5 min.

Slide 17 — Part D · The problem of ambiguity

Duration: 3 mins

Content:

• “The avocado eats an avocado.” — same token, two meanings
• The context decides: “eats” → fruit; subject position → person
• Attention = the mechanism which formalizes this “look towards the context”

Visual: the sentence in broad terms; from the first “lawyer” comes an arrow towards “L’”; from the second, arrows towards “eat” — arrows of thickness proportional to the weight of attention.

Speaker Notes: Regain post-break energy with this fun phrase. Have a participant read it aloud.

Slide 18 — The mechanics of attention

Duration: 4 mins

Content:

• Each token asks the question: “what other tokens clarify my meaning here?”
• It distributes attention weights (importance percentages) over all others
• Then updates its representation by mixing the relevant information
• In parallel for all tokens · on dozens of successive layers

Visual: a central token with teal light beams of different intensities towards the other tokens in the sentence — “spotlight” style. Small stack of layers on the right (layer 1, 2, … n).

Speaker Notes: DO NOT enter Query/Key/Value here. If expert question: “see me at break”. Keep the spotlight as image: attention is a dimmable beam.

Slide 19 — The pronoun test

Duration: 3 mins

Content:

• “The trophy does not fit in the suitcase because it is too big.” → he = trophy
• Change the ending: “…because she is too small.” → she = suitcase
• Solve “he/she” = weigh ALL context — exactly what attention calculates
• This is what pre-2017 architectures missed on long sentences

Visual: Demo 3 heat map: “he” line with dark box on “trophy”. The two variants of the sentence side by side with the slurs reversed.

Speaker Notes: Switch to Demo 3: click on "it", show the dark box towards "trophy". Then load sentence 2 (the lawyer) and show that the SAME word receives different attentions.

Slide 20 — The circle is closed

Duration: 2 mins

Content:

• Reminder Slide 7: attention = long memory · parallelism = scale
• Each token calculates its attention independently → everything is parallelized → GPUs fully exploited
• Honest question: understanding or calculation? → matrix multiplications whose result functionally resembles understanding

Visual: repeat of the table from Slide 7, now “validated” with two copper checkmarks. Small philosophical box in italics at the bottom.

Speaker Notes: Don't settle the philosophical debate — say it explicitly. What matters: the mechanism is powerful, measurable, and explains capabilities AND limits.

Slide 21 — Instructions Exercise 2: the game of analogies

Duration: 1 min (launch) + 8 min (exercise)

Content:

• In pairs · 6 min + 2 min of correction
• Part A: complete 5 analogies A − B + C ≈ ? and name the direction
• Part B: create 2 analogies (including 1 from your profession)
• Part C: the lawyer's trick question

Visual: example solved in large size: Paris − France + Japon ≈ Tokyo with the “country→capital” arrow in copper.

Speaker notes: Chrono visible. When correcting, have MANAGEMENT verbalize each time — this is the targeted skill. Part C brings back attention: educational loop.

Slide 22 — Part E · Pre-workout: college

Duration: 4 mins

Content:

• Only one task: predict the next token — “The cat sleeps on the ___”
• On trillions of text tokens
• To predict well, the model MUST absorb: grammar, facts, styles, basics of reasoning
• Result: very learned but crude — he completes the text, he does not “respond”
• Cost: tens to hundreds of millions of dollars of calculation

Visual: a graduate hat on a stylized teal brain; below, a scrolling text with holes. Copper cost meter with .

Speaker Notes: Emphasize the paradox: a trivial task × a massive scale = general capabilities. The “brilliant graduate who has never spoken to a client” prepares the next slide.

Slide 23 — Post-workout: finishing school

Duration: 4 mins

Content:

• RLHF = Reinforcement Learning from Human Feedback
• Annotators compare pairs of answers: “which is the most useful/honest/safe?”
• The model is adjusted towards the preferred answers
• Transforms a completion engine into a assistant: tone, format, refusal of dangerous requests

Visual: two response bubbles side by side, a human thumb which designates the best one; update arrow to model. Clean teal/copper style.

Speaker notes: Properly articulate the entire acronym RLHF, in French — golden rule of the course.Analogy: the knowledge is already there, we learn manners and ethics.

Slide 24 — Fine-tuning: onboarding in business + summary table

Duration: 3 mins

Content:

• Fine-tuning: specialize an already trained model — in-house vocabulary, brand tone, internal formats

Step	Analogy	What we learn	Cost
Pre-workout	University	Language, facts, reasoning	Very high
Post-workout (RLHF)	Finishing School	Usefulness, tone, security	Medium
Fine-tuning	Onboarding	Job specialization	Low

Visual: the table in full width, alternating light teal / white lines, “Analogy” column with pictograms ( / / ).

Speaker notes: This table is THE reference for future sessions — invite you to photograph it. Question to anticipate: “can we fine-tuner from scratch?” → economic nonsense for 99.9% of companies (see Exercise 3, scenario 1).

Slide 25 — Inference and the memory bottleneck

Duration: 4 mins

Content:

• Inference = use the trained model (fixed parameters): the tokens come out one by one
• Counter-intuition: speed is limited by memory bandwidth, not calculation
• Each token generated = transfer ALL parameters (tens to hundreds of GB ) from memory to calculation
• Hence: the race for HBM (High Bandwidth Memory) · small models respond faster

Visual: the starred chef (calculation) waiting for the shuttle coming from the warehouse (memory) — illustration in 2 comic strip style panels, copper shuttle, teal chef. “Go per token” counter with .

Speaker Notes: Common question to address here: "Is the model learning from my conversations?" → No during inference (fixed parameters); the conversation lives in the context window; collection for FUTURE workouts depends on provider and settings — governance topic, dedicated session.

Slide 25a — Part E-bis · What do we train on, exactly?

Duration: 4 mins

Content:

• Karpathy (Deep Dive into LLMs, ~6 hrs) starts with DATA, not architecture
• Almost universal source: CommonCrawl — non-profit organization, crawling the web since 2007
• 1 crawl every 1–2 months = 200–400 TiB of HTML, ~2.7 billion pages ( 2024)
• FineWeb (HuggingFace): the PUBLIC recipe — 96 crawls “decanted” like a wine

Visual: funnel in 5 teal/copper stages: 96 HTML crawls → text extraction → filtering → deduplication → 15,000 billion tokens (44 TB).

Speaker notes: Emphasize the harvest/winemaking analogy: CommonCrawl = everything the vine gave, including rotten grapes. The name of the report — decanting the web — is no coincidence. This is one of the only data recipes published in full.

Slide 25b — Decant the web: filter, deduplicate, measure

Duration: 4 mins

Content:

• Extraction: the “ready-made” text version (WET) = 25% bigger but WORSE → more data ≠ better
• Filtering: URL blocklist, language (score ≥ 0.65), quality heuristics → ~36,000 billion tokens
• MinHash deduplication — surprise: deduplicating EVERYTHING globally (4T) is WORSE than crawl by crawl (20T)
• How do we know? Ablations: small twin models (1.8 billion params, ~28 billion tokens) compared on HellaSwag, MMLU, ARC…Visual: two columns “intuition” (copper cross) vs “measurement” (teal check); Mini stylized benchmark curves.

Speaker notes: The overall dedup result vs. per crawl surprised the authors themselves: on the oldest crawl, the 10% retained were worse than the 90% deleted. Lesson to hammer home: in data, we don't guess, we measure.

Slide 25c — FineWeb-Edu & Karpathy’s message

Duration: 2 mins

Content:

• FineWeb-Edu: Llama-3-70B notes 500,000 pages (0–5, educational value) → small classifier → applied to 15T tokens (~6,000 h of H100)
• Threshold ≥ 3 → 1,300 billion “very educational” tokens → best MMLU / ARC / OpenBookQA scores
• The (secret) technique of Llama 3 and Phi-3: one AI filters the data of the next one
• Karpathy: an LLM = lossy compression of the Internet → the quality of the dataset IS the ceiling of the model

Visual: stylized loop “AI → filter → data → next AI”; at the bottom, large quote: “garbage in, garbage out — on a web scale”.

Speaker notes: Bridging the gap with the participants' daily lives: their future RAG/fine-tuning projects obey the same law. If time permits: mention the increase in AI-generated text in recent crawls (proxy: frequency of “delve”, “rich tapestry” — self-contamination of the web, open subject). Sources: Karpathy, Deep Dive into LLMs like ChatGPT (2025); Penedo et al., FineWeb (NeurIPS 2024).

Slide 26 — Part F · Scaling laws

Duration: 2 mins

Content:

• Empirical observation: performance = regular and predictable function of 3 ingredients: data · calculation · parameters
• We can predict the performance of a 10× larger model BEFORE training it
• This is what justifies massive investments: extrapolation of curve, not blind bet
• Analogy: material strength equations for bridges

Visual: stylized log-log graph: three smooth rising curves (data/calculation/parameters), extrapolation zone in copper dotted lines with a predicted “future model” point.

Speaker Notes: The bridge analogy works well: you don't build a 2 km bridge “hoping”. Scaling laws are the resistance equations of AI.

Slide 27 — Scaling laws: the nuances

Duration: 2 mins

Content:

• Curves predict a statistical metric (prediction quality), not precise capabilities — some emerge in increments
• The 3 ingredients must grow together (“Chinchilla” lesson, 2022) — a giant undernourished with data = waste
• Open debate on the continuation of curves; other axes explored (e.g. calculation to inference)
• Bigger ≠ still relevant: slower, more expensive for inference → the right model = the smallest that achieves the required quality

Visual: three cursors (data/calculation/parameters) aligned = teal check mark; unbalanced = copper cross. Cost/quality balance at the bottom.

Speaker notes: Link to slide 25: recall memory bandwidth to justify “bigger = slower”. Intellectual honesty: the landscape is changing quickly, report the .

Slide 28 — The complete chain (summary)

Duration: 2 mins

Content:

Text → Tokens → Vectors (embeddings) → Attention → Next token prediction

• The one-line session.Each link explains strengths AND limits:
• Tokens → letter counting failures, costs per language
• Embeddings → semantic search, analogies
• Attention → disambiguation, long memory
• Prediction → entire training pipeline

Visual: repeat the animated frieze from the interactive page (Summary section): 5 light teal blocks connected by copper arrows, appearing in a cascade.

Speaker's notes: Have the room recite the chain, in chorus if the atmosphere permits. This is the mental deliverable of the session.

Slide 29 — Quick Quiz & Exit Tickets

Duration: 5 mins

Content:

• Quiz: 4–5 questions freehand (selection of the complete quiz — 12 questions independently)
• Exit Tickets: 5 written questions, 3 minutes, collected when leaving
• Without notes — only to adjust Session 2

Visual: quiz pictograms (✓/✗) and stylized copper exit ticket.

Speaker Notes: Choose quiz questions based on perceived weaknesses during the session. Exit Tickets are your radar: scan them BEFORE preparing for session 2.

Slide 30 — Next session + resources

Duration: 2 mins

Content:

• Session 2: the art and science of prompting — “You know HOW the machine works. Next time: how to TALK to him to get the most out of it.”
• To do until then: Exercise 3 (consultant triage) if not done in session · play again with the interactive page · complete independent quiz
• Instructor contact + access to supports

Visual: ink background like slide 1 (closing the visual loop), title of session 2 in teal, token pattern in the background.

Speaker Notes: Finishing on time — it signals professionalism. Thank. Stay 5 minutes for individual questions (including postponed technical questions: Query/Key/Value, etc.).

Timing Summary

* The 5 min break is placed after slide 16 (end of Part C). Integrated margins: ±3 min absorbed by the exercises (see plan B in the teacher guide: Exercise 3 → homework).

Slides — Applied AI, Intermediate Level, Session 1. © Yann Isola. Version 1.0.