# Quiz — Session 4: RAG, giving memory to the model

**Program:** Applied AI — Yann Isola · Intermediate level
**11 questions · one correct answer per question · explained answer key at the end**

Reminder: RAG = *Retrieval-Augmented Generation*: retrieve relevant evidence before generating an answer.

---

**Q1.** Which of the following statements about knowledge of an LLM (Large Language Model) used **without** RAG is FALSE?

- A. Its knowledge ends at the training cut-off date.
- B. It has never seen your company's internal documents.
- C. It stores knowledge in compressed, lossy model weights.
- D. It can recite verbatim any public document from its training data.

**Q2.** The central analogy of the RAG seen in class is:

- A. Give glasses to a myopic model.
- B. Transform a closed-memory exam into an open-book exam.
- C. Replace the brain of the model with a database.
- D. Teach the model a new language.

**Q3.** What is the correct sequence of the **ingest** pipeline (offline)?

- A. Document embedding → chunking → indexing.
- B. Cutting into chunks → embedding of each chunk → indexing of the vectors.
- C. Indexing of documents → division → generation.
- D. Question embedding → retrieval → generation.

**Q4.** In the **query** pipeline (online), why do we vectorize the question with the **same embedding model** as that used for the chunks?

- A. To save memory.
- B. Because it's faster.
- C. So that question and chunks live in the same geometric space, condition for proximity to measure similarity of meaning.
- D. Because embedding models only work on questions.

**Q5.** A chunk that is much too large (for example 5,000 tokens) mainly poses what problem?

- A. It mixes several subjects: its embedding becomes a fuzzy “average”, less discriminating for retrieval.
- B. It is illegal in most vector bases.
- C. It makes the embedding model slower to train.
- D. None: the larger the chunk, the better the retrieval.

**Q6.** Why do we add an **overlap** between consecutive chunks?

- A. To artificially increase the size of the index.
- B. To prevent information straddling a cutting boundary from being lost or cut in two.
- C. So that the model generates longer answers.
- D. To compress documents.

**Q7.** When cutting, the rule regarding **tables** is:

- A. Cut them into individual lines for smaller chunks.
- B. Delete them, because the models do not read the tables.
- C. Keep them whole in a single chunk: a line separated from its header becomes unintelligible.
- D. Systematically convert them into images.

**Q8.** A user searches for "error REF-2024-8812" and the purely vector RAG finds nothing relevant. What is the most likely explanation and solution?

- A. The generation model is too small; you need a larger model.
- B. Exact codes have little semantic content: their embedding does not distinguish them well. Solution: hybrid search, combining vectors and keyword search.
- C. The context window is saturated; it must be emptied.
- D. The code is too new for embedding; the model must be retrained.

**Q9.** Your RAG answers next to the question. You inspect and see that the retrieved chunks are off-topic. The correct diagnosis is:

- A. The generation model hallucinates; it must be replaced.
- B. Retrieval failed: the model worked correctly with bad ingredients. The division, index or search must be corrected.
- C. The user asked the wrong question; nothing to correct.
- D. It's a random bug, rerunning the query will be enough.

**Q10.** What is the role of the instruction “Answer only from the context provided; if the answer is not there, say so”?

- A. Accelerate generation.
- B. Reduce the cost of queries.
- C. Prevent the model from filling in the gaps with its internal memory, and allow honest refusal rather than plausible invention.
- D. Force the model to answer even when evidence is absent.

**Q11.** A self-correction step asks the same model whether its answer is grounded. Why is that insufficient to open the evidence gate?

- A. A model can never reread its own answer.
- B. The same error can be repeated; require verifiable citations and thresholds calibrated on an external evaluation set.
- C. Only HyDE can verify an answer.
- D. Reranking replaces evaluation entirely.

---

## Answer key with comments

**Q1 — D.** Knowledge is stored *lossy* (lossy): the model retains the general idea, not the exact text — like you after a book read ten years ago. A, B, C are precisely the three limits covered in this session (fixed, public, with losses).

**Q2 — B.** The LLM alone takes the closed-book exam; RAG lets it bring the correct documents. We do not replace the model (C) or re-train it (D): we enrich its prompt at question time.

**Q3 — B.** Ingestion: cut → vectorize each chunk (the meaning becomes geometry, see Session 1) → index the vectors. Answer D describes the *request* pipeline, not ingestion.

**Q4 — C.** Searching for the “k chunks closest to the question” only makes sense if everyone is in the same space. Two different embedding models produce incompatible spaces: the distances mean nothing.

**Q5 — A.** A catch-all chunk has an “average” embedding that does not strongly resemble any specific question. It also wastes the context window on generation. The good range is typically 300–800 tokens ⚠. Split by structure.

**Q6 — B.** Without overlap (typically 10–20% ⚠), a key phrase straddling two chunks may not be complete in either of them — therefore not found.

**Q7 — C.** “130 € / night | Nominative invoice” without the header or the “Charge type” column is unusable. The entire table + its introductory sentence = a chunk.

**Q8 — B.** `REF-2024-8812` is an arbitrary string: semantically close to any other code. Lexical search (by keywords, type BM25) finds it exactly. Hence the **hybrid** search: vectors for the meaning, keywords for the exact.

**Q9 — B.** RAG debugging reflex #1: inspect the chunks. Here the proof is made — it is retrieval. Changing the model (A) would be useless: the cook can't do anything if the clerk brings the wrong ingredients.

**Q10 — C.** This framing reduces unsupported answers and enables **honest refusal**: “I cannot find this information in the supplied documents” is a valid answer—far better than a plausible invention.

**Q11 — B.** Self-correction can be a routing signal, but the same model can confirm its own error. The evidence gate requires verifiable citations and thresholds calibrated on an **external evaluation set**; neither HyDE nor reranking replaces that proof.

---

### Indicative scale

| Score | Reading |
|---|---|
| 10–11 | Excellent: can select a RAG intervention and demand external evidence. |
| 8–9 | Solid. Review missed questions with the explanations. |
| 6–7 | Revisit the pipelines, robust cascade, and failure modes. |
| < 6 | Repeat Exercises 2 and 4, then retake the quiz. |