# Trainer’s Guide — Session 4 (Intermediate level) ## RAG: give memory to the model **Program:** Applied AI — Yann Isola **Duration:** 2 hours **Audience:** professionals who have followed sessions 1 to 3 (embeddings, context window, prompting) **Source module:** Module 3, part 1 --- ## 1. Educational objectives At the end of the session, each participant should be able to: 1. **Explain** why the knowledge of an LLM (Large Language Model) is limited: frozen at the cut-off date, public only, stored in a compressed manner and with losses in the weights of the model. 2. **Describe** the principle of RAG (Retrieval-Augmented Generation): transform each question into an “open book exam”. 3. **Draw** the two pipelines: ingestion (offline) and query (online), with their respective stages. 4. **Make chunking decisions**: chunk size, overlap, structure cutting, entire tables. 5. **Diagnose** a RAG failure by separating retrieval errors from generation errors. 6. **Select and evaluate** a robust intervention—multi-query, HyDE, reranking, or an evidence gate—without ignoring latency, cost, or privacy. --- ## 2. Prerequisites and materials | Element | Detail | |---|---| | Participant prerequisites | Session 1 (embeddings: meaning becomes geometry), Session 2-3 (context, prompting) | | Training material | Video projector, session slides, interactive web page `webpage/index.html` (works offline) | | Participant materials | Laptop recommended for exercises 2 and 3 (paper possible) | | Documents to print | Worksheet, quiz, and six rotating exit tickets | **Pre-session check:** open `webpage/index.html`; exercise the pipeline simulator, chunking visualizer, and robust-RAG cascade. No internet connection is required. --- ## 3. Timed course (120 minutes) ### Block A — The problem: a frozen memory (0:00 → 0:20, 20 min) | Time | Activity | Slides | |---|---|---| | 0:00–0:05 | Home + express reminder Session 1: “meaning becomes geometry” (embeddings) | 1–3 | | 0:05–0:15 | The three limits of the memory of an LLM | 4–7 | | 0:15–0:20 | Mini-demo: question about a fictitious internal fact → the model cannot know | 8 | **Trainer notes:** - Open with a question to the group: *“If I ask a model how many leave days you have left, what will it answer?”* Expected answers: it does not know, or worse, it invents an answer. Both cases illustrate the problem. - Hammer the **three limits**: 1. **Fixed**: knowledge stops on the training cutoff date (*knowledge cutoff*). Anything later does not exist for the model. 2. **Public**: the model was trained on public data. Your internal documents, contracts, procedures, tickets — never seen before. 3. **Lossy**: even public knowledge is stored with loss in the model weights. Analogy: the model has “read” Wikipedia, but it cannot recite it word for word, like you after reading a book ten years ago. - Establish the central analogy now: **closed-book exam vs open-book exam**. The LLM alone is a student taking an exam from memory. RAG lets that student bring the right documents. - Frequent trap: participants think that *fine-tuning* (partial re-training) is the solution to inject knowledge. Note the objection on the board, come back to it at the end of block B: fine-tuning learns *behaviors* (style, format), not reliable and up-to-date *facts*; it is expensive and must be redone for each document update. The RAG updates the index in seconds. --- ### Block B — The principle of RAG and the two pipelines (0:20 → 0:55, 35 min) | Time | Activity | Slides | |---|---|---| | 0:20–0:30 | Definition of RAG + overview of both pipelines | 9–11 | | 0:30–0:40 | Ingestion pipeline (offline): slice → vectorize → index | 12–14 | | 0:40–0:50 | Query pipeline (inline): vectorize the question → retrieve k chunks → assemble the prompt → generate | 15–17 | | 0:50–0:55 | Interactive demo: pipeline simulator on web page | 17 | **Trainer notes:** - **Define each term on first use** — course rule: - RAG = *Retrieval-Augmented Generation*: retrieve relevant evidence before generating an answer. - Chunk = fragment of document (we will keep the English word, standard in the profession, by translating it once: “piece”). - Embedding = vector embedding, seen in Session 1: a text becomes a point in a geometric space where proximity = similarity of meaning. - Top-k = the k closest results (k is a number we choose, often 3 to 10 ⚠). - **Insist on the temporal separation of the two pipelines**: - *Ingestion*: is done **once** (then each time the documents are updated), **offline**, without a user. This is the preparation of the library. - *Query*: is done **with each question**, **online**, in a few hundred milliseconds ⚠. This is the library consultation. - Diagram on the board (do it by hand, even if it is in the slides — the gesture helps memorization): ``` INGESTION (offline, once) Documents → Chunking → Embedding each chunk → Vector index QUERY (online, for each question) Question → Question embedding → Retrieve the k closest chunks → Assemble the prompt (instruction + chunks + question) → Generate ``` - **Conceptual key point**: the question and the chunks live in *the same geometric space*. This is why “search for chunks close to the question” makes sense. Link explicitly to Session 1. - Concrete example to be given orally from start to finish: *“What is the telework policy for new hires?”* 1. The question becomes a vector. 2. The index returns 4 chunks: two extracts from the internal regulations, an extract from the 2025 teleworking agreement, an extract from the onboarding guide. 3. The assembled prompt: “Answer only from the following context. [4 chunks] Question: …” 4. The model generates a response citing the telework agreement. - **Demo** (5 min): project `webpage/index.html`, “Pipeline simulator” tab. Type a question, go through the 4 steps one by one. Ask the group to predict which chunks will come out before clicking. --- ### Break (0:55 → 1:05, 10 min) --- ### Block C — Chunking decisions and metadata (1:05 → 1:30, 25 min) | Time | Activity | Slides | |---|---|---| | 1:05–1:15 | Chunking: size, overlap, structure, tables | 18–21 | | 1:15–1:20 | Metadata: source, section, date, access level | 22 | | 1:20–1:30 | Exercise 1 in pairs: strategy for cutting up a real document | — | **Trainer notes:** - **Chunking is the #1 quality lever of a RAG.** Tell it like it is. - The four decisions: 1. **Size**: typically 300 to 800 tokens ⚠ (reminder: 1 token ≈ 0.75 words in English, a little less in French ⚠). Too small = the chunk loses its context (“he” — who is that, “he”?). Too big = the chunk mixes several subjects and its embedding becomes a fuzzy average. 2. **Overlap**: make the chunks overlap by 10 to 20% ⚠ so as not to cut information right at the border. 3. **Split by structure**: titles, sections, paragraphs — never in the middle of a sentence. A chunk = ideally a unit of meaning. 4. **Whole tables**: never cut a table in two. A table row without its header is unreadable (example: “42 | 15% | yes” — what are we talking about?). - Analogy: divide a book into revision sheets. A good sheet is self-contained (understandable on its own), neither too short nor too long, and does not cut a conjugation table in two. - **Metadata**: each chunk carries a label — *source* (which document), *section*, *date*, *access level*. Three uses: - **Filtering** before searching: “only search in HR documents after 2024”. - **Citations**: The answer can point to the source document — essential for trust and verification. - **Security**: a salesperson must not retrieve chunks from the payroll file. The filter by access level is done **at retrieval**, not after generation. - **Exercise 1** (10 min, pairs): see exercise sheet. Distribute the extract of the document provided. Circulate between pairs. Express debrief: 2 pairs present their division, compare the choices on the integrated table. --- ### Block D — Hybrid search, robust RAG, and failure modes (1:30 → 1:50, 20 min) | Time | Activity | Slides | |---|---|---| | 1:30–1:35 | Hybrid search: vectors + keywords | 23–24 | | 1:35–1:44 | Robust-RAG intervention ladder + interactive cascade | 25–29 | | 1:44–1:49 | Diagnosis, evidence gate, and refusal | 30–33 | | 1:49–1:50 | Decide: answer, reformulate, or refuse | 33 | **Trainer notes:** - **Hybrid search:** embeddings capture meaning but can miss exact strings. `REF-2024-8812` carries little semantic meaning; BM25 retrieves it exactly. Merge vector and lexical candidates before ranking. - **Intervention ladder—do not enable everything by default:** 1. *Multi-query:* produce 2–4 reformulations, retrieve for each, then merge and deduplicate. 2. *HyDE* (*Hypothetical Document Embeddings*): generate a hypothetical document and use its embedding as a retrieval probe. It is **never evidence**. 3. *Broad retrieval:* optimize recall so useful evidence is not discarded too early. 4. *Reranking:* a **cross-encoder** reads each question–chunk pair and reranks candidates only. It is slower and more expensive than the first-stage bi-encoder. 5. *Evidence gate:* answer only when authorized, current, non-contradictory passages support the response; otherwise reformulate or refuse. - **Name the loops without presenting them as guarantees:** *Corrective RAG* grades passages and then reformulates or retries retrieval; *Adaptive RAG* chooses a simple search, a deeper cascade, or refusal based on the query; *Self-RAG* asks the model for reflection signals during generation. Those self-assessments are **routing signals**, not **external evidence**. - **Measure before adding:** compare variants on the same annotated **evaluation set**. Track retrieval recall/precision, citation support, correct refusal, **latency**, and **cost**. A model's self-score is not external evidence. - **Privacy:** web search creates a new data boundary. Never send confidential queries or excerpts to an external service without an explicit policy, filtering, and authorization. - **Diagnosis:** inspect retrieved chunks before blaming generation. If the document is absent from the index, multi-query, HyDE, and reranking cannot recreate it. - **Honest refusal:** “I cannot find this information in the supplied documents” is expected when the evidence gate remains closed. --- ### Block E — Quiz, summary, and exit tickets (1:50 → 2:00, 10 min) | Time | Activity | |---|---| | 1:50–1:57 | Quiz: 11 multiple-choice questions | | 1:57–1:59 | Summary: six ideas to retain | | 1:59–2:00 | Exit tickets | **Six ideas to retain:** 1. LLM memory is frozen, public, and lossy; RAG creates an open-book exam. 2. Two pipelines: offline ingestion and online query. 3. Chunking is the first lever: overlap, structural boundaries, and whole tables. 4. Hybrid search combines vectors for meaning with keywords for exact strings. 5. Robust RAG means reformulate → retrieve broadly → rerank → gate on evidence. 6. When RAG fails, inspect the chunks first, then answer or refuse. --- ## 4. Exit tickets (six questions) Give each learner one rotating question. Answers should fit in one or two sentences. **Ticket 1.** Name the three limits of a language model's knowledge without RAG. *Expected: frozen at the training cutoff, limited to training data, and stored lossily in weights.* **Ticket 2.** How do ingestion and query pipelines differ? *Expected: ingestion prepares the index offline; the query pipeline runs for each user request.* **Ticket 3.** Why should a table not be split during chunking? *Expected: a row without its headers loses meaning.* **Ticket 4.** A RAG answer is wrong. What is your first diagnostic action? *Expected: inspect the retrieved chunks and confirm the expected document exists in the index.* **Ticket 5.** Why combine vector and keyword search? *Expected: vectors capture meaning; lexical search retrieves exact codes and strings.* **Ticket 6.** Why is a positive model self-assessment insufficient to open the evidence gate? *Expected: the same model can repeat its error; require verifiable citations and thresholds calibrated on an external evaluation set.* --- ## 5. Anticipated difficulties and responses | Difficulty | Response | |---|---| | “Why not put every document in the prompt?” | Context is limited; extra text adds noise, latency, and cost. Retrieval selects relevant passages. | | Confusing chunk and query embeddings | They must use the same embedding model and geometric space. | | “Fine-tuning does the same thing” | Fine-tuning changes behavior; it is not an updatable factual store with citations and access control. | | “More stages always mean better RAG” | False. Each stage must fix a measured failure while respecting latency, cost, and privacy budgets. | | “The model says its answer is grounded” | Self-assessment is not external evidence. Require citations, verification, and an annotated evaluation set. | | “RAG eliminates hallucinations” | It can reduce them; retrieval and generation can still fail. | --- ## 6. Extensions - Compare small and large k values, then inspect recall, noise, latency, and cost. - Run the robust cascade in `webpage/index.html`; ask each group which stages they would retain after evaluation. - Ask which company documents should enter a RAG first and which metadata fields are critical. - **Next in the program—Session 5:** tools and function calling, allowing a model to act through controlled interfaces.