# Quiz — Session 1: Foundations: Transformers & Tokenization **Program:** Applied AI — Intermediate Level **Instructor:** Yann Isola **Format:** 12 multiple choice questions — only one correct answer per question. **Recommended duration:** 18 minutes independently, or 4–5 questions selected by show of hands at the end of the session. **Recommended passing threshold:** 8/12. --- ### Question 1 — (Memorization) The innovation of 2017 Which 2017 publication introduced “transform” architecture? - A. *ImageNet Classification with Deep Convolutional Neural Networks* - B. *Attention Is All You Need* - C. *Playing Atari with Deep Reinforcement Learning* - D. *Language Models are Few-Shot Learners* **✅ Answer: B.** **Explanation:** The article *Attention Is All You Need* (Google, 2017) introduced the transformer, an architecture based entirely on the attention mechanism and fully parallelizable. A is the AlexNet article (2012, computer vision); C concerns reinforcement learning; D is the article by GPT-3 (2020), later. --- ### Question 2 — (Understanding) What the model actually “reads” When you send the phrase “Hello, how are you?” » to a language model, what does it actually process? - A. Each letter, one by one - B. Each whole word, like in a dictionary - C. Tokens: fragments of text of around 3–4 characters on average ⚠ - D. An image of the sentence, analyzed by optical recognition **✅ Answer: C.** **Explanation:** The text is cut by a tokenizer into fragments (tokens) from a fixed vocabulary (~100,000 entries, variable order of magnitude depending on the models ⚠). Frequent words fit into one token, rare words are assembled from several fragments. Neither individual letters nor systematic whole words. --- ### Question 3 — (Application) The riddle of counting letters A model gets it wrong when counting the number of “r”s in “strawberry”. What is the MOST accurate explanation? - A. The model never encountered this word during training - B. The word is divided into opaque tokens: the model has no direct access to individual characters - C. The model does not understand English - D. The model's memory is saturated by the conversation **✅ Answer: B.** **Explanation:** “strawberry” is a common word that the model “knows” very well statistically (A is wrong). But he sees it as a few blocks (e.g. [str][aw][berry]), not as 10 letters: counting characters requires a granularity that tokenization hides. This is a structural limitation — one that recent models sometimes get around by spelling out the word ⚠ first. --- ### Question 4 — (Understanding) Embeddings What does the famous equation “king − man + woman ≈ queen” mean? - A. The model stores a database of royal family relationships - B. Words are vectors, and meaning relations correspond to coherent directions in vector space - C. The model applies a hand-coded grammar rule for gender - D. It's a meaningless coincidence, often misquoted **✅ Answer: B.** **Explanation:** Each token is converted into a vector (list of numbers); nearby senses occupy nearby positions. Vector arithmetic reveals that relationships like masculine→feminine or country→capital form regular directions of space.Nothing is coded by hand: these regularities emerge from training. --- ### Question 5 — (Understanding) The role of attention What is the purpose of the attention mechanism in a transformer? - A. To filter inappropriate content before responding - B. To allow each token to dynamically evaluate the relevance of all other tokens in the context - C. To permanently memorize user conversations - D. To compress the text to save memory **✅ Answer: B.** **Explanation:** Attention assigns to each pair of tokens a relevance weight, dynamically recalculated according to the context: in “The avocado eats an avocado”, the second “avocado” is pulled towards the meaning “fruit” by “eats”. It is this mechanism that resolves long-distance dependencies (e.g. attaching a pronoun to its distant antecedent). --- ### Question 6 — (Application) The pipeline analogy Your company wants an existing model to learn the exact format of your internal audit reports. In the course analogy, this step corresponds to: - A. University (pre-training) - B. Finishing school (post-training / RLHF) - C. Corporate onboarding (fine-tuning) - D. The final exam (evaluation / benchmark) **✅ Answer: C.** **Explanation:** Fine-tuning specializes a model already trained in a specific area, tone or format - like a new hire who has already graduated and is trained in in-house procedures. Pre-training (A) builds general knowledge; post-training/RLHF (B, Reinforcement Learning from Human Feedback) shapes assistant behavior. --- ### Question 7 — (Memorization) The pre-training task What is the unique task that pretrains a large language model? - A. Correctly answer questions asked by humans - B. Translate texts between several languages - C. Predict the next token in a text - D. Classify texts into categories (spam, sentiment, subject, etc.) **✅ Answer: C.** **Explanation:** All the model's knowledge comes from a seemingly trivial task: guessing the next token out of trillions of text tokens ⚠. To predict well, the model is forced to absorb grammar, facts, styles and basics of reasoning. “Question answering” behaviors (A) come next, post-training. --- ### Question 8 — (Application) The inference bottleneck A chatbot in production responds too slowly. The service provider offers to double the raw computing power of the servers. Why is this proposal likely ineffective? - A. Because the inference speed is mainly limited by the memory bandwidth: each generated token requires passing all the model parameters from memory - B. Because the models have a fixed speed defined by their user license - C. Because doubling the calculation also doubles the response time - D. Because the slowness always comes from the internet network **✅ Answer: A.** **Explanation:** For each token produced, all the parameters (tens to hundreds of gigabytes ⚠) must flow from the memory to the calculation units. Calculation awaits memory — lesson analogy: the star chef (calculation) waits for the warehouse shuttle (memory).Real levers: faster memory (HBM — High Bandwidth Memory), smaller model, inference optimizations. --- ### Question 9 — (Understanding) Scaling laws What do “scaling laws” describe? - A. European regulations governing the maximum size of AI models - B. The regular and predictable improvement in performance when we increase data, calculation and parameters together - C. Year-over-year growth in AI vendor pricing - D. The speed at which a model forgets old information **✅ Answer: B.** **Explanation:** The scaling laws are empirical regularities: the quality of prediction improves according to regular curves when we increase the three ingredients in a balanced way. This is what allows laboratories to predict the performance of a future model before training it — and justifies their massive investments. Nuance: they predict a statistical metric, not the appearance of specific abilities. --- ### Question 10 — (Application/synthesis) The double contribution of transformers Before 2017, Recurrent Neural Networks (RNNs) processed text word by word. What TWO problems did the transformers solve simultaneously? - A. The cost of electricity and the size of data centers - B. Automatic translation and voice recognition - C. The forgetting of distant dependencies in the text (resolved by attention) and the impossibility of parallelizing the training (resolved by a parallel architecture exploiting GPUs) - D. Hallucinations and model biases **✅ Answer: C.** **Explanation:** This is the structural summary of the session: attention = each token accesses the entire context, even remote ones (long memory); parallelism = all tokens are calculated at the same time, allowing thousands of GPUs (Graphics Processing Unit) to be exploited and reaching current model sizes. The hallucinations and biases (D) were NOT resolved by the transformers — we will talk about them again in the following sessions. --- ### Question 11 — (Understanding) The raw material of LLMs In the FineWeb type data pipeline presented in the session (the one that Andrej Karpathy uses in his reference video), what is the correct order of steps to go from the raw web to the pre-training dataset? - A. Tokenization → crawl → filtering → training - B. Crawl (CommonCrawl) → extraction of text from HTML → filtering (language, quality) → deduplication - C. Deduplication → crawl → human annotation of each page → extraction - D. Filtering → tokenization → crawl → deduplication **✅ Answer: B.** **Explanation:** We start from web crawls (CommonCrawl: 200–400 TiB of HTML per crawl ⚠), we extract the useful text from the HTML, we filter (URL blocklist, language, quality heuristics), then we deduplicate (MinHash). Result for FineWeb: 15,000 billion tokens, 44 TB. Tokenization occurs afterwards, when preparing the training; and no page-by-page human annotation is possible at this scale. --- ### Question 12 — (Analysis) The deduplication paradox The FineWeb authors compared two strategies: deduplicating all 96 crawls globally, or deduplicating each crawl independently. What did they notice? - HAS.Global deduplication results in better models because it removes more duplicates - B. Both strategies give identical results, deduplication is useless - C. Crawl by crawl deduplication gives better models: the global version over-represented old, poor quality data - D. Global deduplication is better but too expensive, so they chose the crawl version to save money **✅ Answer: C.** **Explanation:** Counterintuitive result verified by ablations (small twin models compared on the same benchmarks): global dedup (4,000 billion tokens) produced worse models than dedup per crawl (20,000 billion), because on old crawls it mainly retained content of poor quality — the 10% “kept” there were worse than the 90% deleted. Lesson from Part E-bis: in data curation, we don't trust intuition, we measure. --- ## Quick correction grid | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | Q11 | Q12 | |----|----|----|----|----|----|----|----|----|-----|-----|-----| | B | C | B | B | B | C | C | A | B | C | B | C | **Interpretation of scores:** - **11–12:** complete mastery — ready for Session 2. - **8–10:** solid knowledge — reread the explanations of the missed questions. - **6–7:** review the corresponding sections of the support; repeat exercises 1 and 3. - **< 6:** resume the interactive page and session support before Session 2; do not hesitate to contact the instructor. --- *Quiz — Applied AI, Intermediate Level, Session 1. © Yann Isola. Version 1.0.*