FrançaisApplied AI · Intermediate level · Session 1
Foundations: Transformers & Tokenization
Three interactive demonstrations to see with your own eyes how a language model reads, represents and connects text.
Demo 1
🧩 The tokenizer: how the model cuts your text
Type any text: it is cut into tokens — fragments, not words. Frequent words fit into a token; rare words, mistakes and numbers are fragmented. (Simplified educational tokenizer: real tokenizers follow the same logic with a learned vocabulary of ~100,000 fragments ⚠.)
0
tokens
0
characters
0
characters / token
To remember: the model never sees individual letters — only these blocks.
This is why counting the “r”s in “strawberry” or reversing a word is structurally difficult for it,
whereas writing or summarizing (tasks at the token level) is easier.
Demo 2
🗺️ The space of embeddings: meaning becomes geometry
Each token becomes a vector — a position in a space of meaning. Words close in meaning are neighbors in space. Drag to explore and hover over a point to read its label. (Educational 2D projection: real spaces have hundreds of dimensions ⚠.)
Animals Colors Professions
Observe: “veterinarian” is placed between the cloud of professions and that of animals — embeddings
capture multiple memberships. And the animation shows that the “masculine → feminine” relationship
is a direction in space: the same arrow connects man → woman and king → queen.
Demo 3
🔦 Attention: each word “looks” at the others
To clarify its meaning, each token assigns an importance weight to all other tokens in the context. Click a word in the sentence or read the heat map: the darker a box, the stronger the attention. (Illustrative weights, simplified for educational purposes.)
Click a word to see where its attention is focused.
The key point: the weights are recalculated dynamically for each sentence.
It is this mechanism — introduced by transformers in 2017 — that resolves long-distance dependencies
(connect “it” to “trophy” despite the distance) where previous architectures “forgot”.
Summary
🎯 The complete pipeline
The common thread of the session — what happens with each message you send: