Français

Slide 1 — Title

RAG: give memory to the model

Applied AI · Intermediate level · Session 4 · 2 h

RAG = Retrieval-Augmented Generation — generation grounded in retrieved evidence

Slide 2 — What you will be able to do in 2 hours

  1. Explain the three limits of an LLM memory
  2. Draw the two pipelines of the RAG (ingestion / request)
  3. Make chunking decisions
  4. Justify hybrid search
  5. Select a robust-RAG intervention: multi-query, HyDE, reranking, or an evidence gate
  6. Diagnose a failing RAG without blaming the model by default

Slide 3 — Reminder Session 1: meaning becomes geometry

  • Embedding transforms a text into a vector (list of numbers)
  • Texts with a close meaning → near points in space
  • “leave” and “vacation”: neighbors. “holidays” and “SQL server”: distant

Slide 4 — Thought experiment

“How many vacation days do I have left?”

What does a language model alone answer?

❌ The model cannot know… or worse: it invents

Slide 5 — Limit n°1: a frozen memory

  • Knowledge ends on the cutoff date of the training (knowledge cutoff)
  • Any subsequent event does not exist for the model
  • Example: an internal reorganization of March 2026 is invisible for a model trained before

Slide 6 — Limit n°2: a public memory

  • Training on public data (web, books, code, etc.)
  • Your contracts, procedures, tickets, internal wikis: never seen
  • The model does not know your business — by construction

Slide 7 — Limit no. 3: a memory with losses

  • Knowledge is stored compressed, with losses (lossy) in weights
  • The model has “read” Wikipedia… but cannot recite it
  • Like you with a book read ten years ago: the idea remains, the exact text does not

Slide 8 — Mini-demo: the wall

Question to the model without RAG: “What is the expense reimbursement procedure at this fictitious company?”

Expected result: a generic invention or an honest refusal.

The model is taking a closed-book exam.

Slide 9 — The idea of RAG

Turn every question into an open book exam

RAG = Retrieval-Augmented Generation
(retrieve evidence first, then generate)

Instead of relying on model memory:
we find the right documents, we give them to the model, and it answers from that evidence.

Slide 10 — Why not put everything in the prompt?

  • Context window limited (and your documents = thousands of pages)
  • Very long context = quality which degrades + cost per request which explodes
  • The RAG selects: only the relevant passages enter the prompt

Slide 11 — Overview: two pipelines

Ingestion Query
When Offline, Upstream Online, with every question
Frequency Once + updates Thousands of times a day
Role Prepare the library Consult the library

Slide 12 — Ingestion pipeline (1/3): cut

Documents → chunks

  • An entire document = too big, too heterogeneous
  • We cut it into fragments of 300 to 800 tokens ⚠
  • Each chunk = ideally a unit of meaning

Slide 13 — Ingestion pipeline (2/3): vectorize

Each chunk → one embedding

  • The meaning of each chunk becomes a point in space (Session 1)
  • Chunks of close direction → neighboring points

Slide 14 — Ingestion pipeline (3/3): index

Vectors → a vector index

  • Specialized database to quickly answer: “what are the k points closest to this one?”
  • Result of ingestion: a library searchable by sense

Slide 15 — Request pipeline (1/2)

To each question:

  1. Embed the question with the same embedding model used for chunks
  2. Retrieve the nearest k chunks (often 3–10 ⚠)

Slide 16 — Query pipeline (2/2)

  1. Assemble the prompt:
Instruction: "Answer only from the supplied context."
+ Context: [chunk 1] [chunk 2] [chunk 3]
+ User question
  1. Generate an open-book answer with citations

Slide 17 — Demo: the pipeline simulator

🖥️ webpage/index.html — RAG Pipeline Simulator tab

Question → embedding → top-3 retrieval → assembled prompt → cited answer

Slide 18 — Chunking: lever n°1

The quality of a RAG is first determined by cutting

4 decisions:
size · overlap · structure · tables

Slide 19 — Decision 1 & 2: size and overlap

  • Too small: “It must be validated first.” — what must be validated? The chunk has lost its context
  • Too large: several subjects mixed → “medium” embedding, blurry, poorly retrieved
  • Good range: 300–800 tokens ⚠
  • 10–20% overlap ⚠: no information dies cut off at the border

Slide 20 — Decision 3: cut on the structure

  • Cut on headings, sections, paragraphs — never in full sentence
  • A chunk = an autonomous unit of meaning
  • Tip: prefix each chunk with its breadcrumbs
    (“Chapter 4 > Export > Supported formats”)

Slide 21 — Decision 4: tables remain whole

A line cut from its header:

“130 € / night | Named invoice” — what are we talking about?

Rule: one table = never cut. Entire table + its introductory sentence = one chunk.

Slide 22 — Metadata: the label of each chunk

Each chunk carries: source · section · date · access level

Usage Example
Filtering “search only in HR docs ≥ 2024”
Quotes the answer points to the source document — verifiable
Security a salesperson does not retrieve chunks from the payroll file

Slide 23 — Vectors miss exact codes

Search for “error REF-2024-8812”:

  • Semantically, this code “means nothing”
  • Its embedding ≈ that of REF-2024-8811, ERR-4471, any code
  • Vector search returns almost anything

Slide 24 — The solution: hybrid search

Vectors (the meaning) + keywords (the exact) → fusion of results

  • Vector search: “paternity leave” finds “birth leave”
  • Lexical search (BM25): “ERR-4471” is found exactly
  • The standard for serious RAGs: both, combined

Slide 25 — From vanilla RAG to robust RAG

A poor result does not justify stacking every available technique.

Intervention ladder:
weak query → multi-query / HyDE → broad retrieval → reranking → evidence gate → answer or refusal

Slide 26 — Stage 1: improve the search query

  • Multi-query: produce 2–4 reformulations, retrieve for each, then merge and deduplicate
  • HyDE (Hypothetical Document Embeddings): generate a hypothetical answer document and search using its embedding
  • The hypothetical text is a retrieval probe—never evidence and never the final answer

Slide 27 — Stage 2: retrieve broadly, then rerank

  1. A bi-encoder retrieves a broad candidate set quickly
  2. A cross-encoder reads each question–chunk pair together
  3. Reranking keeps the few best-supported passages

Why two stages? Recall first; precision second.

Slide 28 — Stage 3: the evidence gate

Before generation, ask:

  • Do the passages actually answer the question?
  • Are the sources authorized, current, and non-contradictory?
  • Can every material claim be cited?

Enough evidence → answer. Insufficient evidence → reformulate or refuse.

Slide 29 — The cost of robustness

Every added stage can increase:

  • latency: multiple searches and model calls;
  • cost: query rewrites, HyDE, and cross-encoder scoring;
  • privacy risk: never send confidential data to web search without an explicit policy;
  • evaluation complexity: more routes create more failure modes.

Slide 30 — When RAG fails, who is responsible?

Wrong reflex: “The model hallucinated; replace it.”

Correct reflex: inspect the retrieved chunks first

In many incidents, the failure is in retrieval, not generation.

Slide 31 — Failure mode 1: the wrong chunks

  • Ambiguous query, poor chunking, or k too small
  • The model receives off-topic context and answers off-topic
  • Escalate gradually: rechunk, add hybrid search, try multi-query, then rerank when useful candidates exist but rank poorly

Slide 32 — Failure mode 2: a missing document

  • The information was never ingested
  • No model can retrieve what is not in the index
  • Example: an unindexed 2025 leave amendment leaves the system citing the outdated 2022 agreement

Fixes: index refresh process, date metadata, and source-age alerts.

Slide 33 — Framing and honest refusal

Instruction: “Answer only from the supplied context. If the answer is absent, say so.”

  • Reduces reliance on internal model memory without guaranteeing zero hallucinations
  • “I cannot find this information in the supplied documents” is a valid answer

Honest refusal beats plausible invention.

Slide 34 — Summary: six ideas to retain

  1. LLM memory is frozen, public, and lossy → RAG creates an open-book exam
  2. Two pipelines: ingestion offline; query online
  3. Chunking is the first lever: overlap, structure, whole tables
  4. Hybrid search: vectors for meaning, keywords for exact strings
  5. Robust RAG: reformulate → retrieve → rerank → gate on evidence
  6. When it fails, inspect the chunks first, then answer or refuse

Slide 35 — Quiz, exit tickets, and next session

  • 📝 Quiz: 11 questions, 7 minutes
  • 🎫 Exit ticket: one question answered in two sentences
  • Session 5: tools and function calling—letting a model act through controlled interfaces

Notes: Welcome. Announce the common thread: at the end of the session, everyone will be able to draw a complete RAG pipeline and diagnose why a RAG responds poorly. Remember that this session builds directly on Session 1 (embeddings).

Notes: LLM = Large Language Model — redefine even if the group knows. Each objective corresponds to a block of the session.

Notes: Ask the group for 30 seconds: “Who can redefine an embedding for me? ". This reminder is the foundation of the entire session — vector retrieval is ONLY geometry. Without that, the rest is voodoo.

Notes: Let the group respond. The two outcomes (admission of ignorance / plausible invention) illustrate the problem. Continue: why can't the model know? → next slide.

Notes: Analogy: a printed newspaper — excellent on the day, never updated afterwards. Point out that re-training a model costs millions ⚠: we don't do it to refresh facts.

Notes: This is the most important limit in a professional context. Question to the group: “What documents from YOUR daily life has a model never seen?” List 3-4 concrete examples; we will reuse them in the chunking block.

Notes: This is the least intuitive limit. Consequence: even on the public, the model can distort details (dates, numbers, names). Hence the plausible hallucinations. The three limits together justify the RAG.

Notes: Use a live demo when available. Establish the phrase “closed-book exam”; RAG turns the next question into an open-book exam.

Notes: THE pivot slide. We do not modify the model, we do not re-train it: we enrich its prompt at the time of the question. The model remains the same — it is its context that changes.

Notes: Frequent objection to defusing right away. Analogy: to answer an exam question, you don't bring the entire library — you bring the 3 correct cards.

Notes: Fundamental temporal separation — participants systematically confuse them. Ingestion is done WITHOUT user; the query takes place in a few hundred milliseconds ⚠.

Notes: Reminder: 1 token ≈ 0.75 words in English, a little less in French ⚠. The word “chunk” will remain in English (trade standard) – translate it once: “piece”. The fine cutting decisions arrive at block C.

Notes: Insist: this is the SAME mechanism as in Session 1, applied to each piece of document. A corpus of 10,000 chunks = 10,000 points in a space of several hundred dimensions ⚠.

Notes: Do not go into the details of implementing vector bases — outside the scope. Remember the function: search for nearest neighbors, fast, on a large scale.

Notes: Question and chunks must share the same vector space. Two different embedding models produce incompatible distances. Small k can miss evidence; large k adds noise, latency, and cost.

Notes: Show that there is NO magic: RAG is prompt assembly. The model does not “connect” to anything — it reads what is put in front of it, nothing more.

Notes: 5 minutes. Type “How many days of telework per week?” Before each step, ask the group to predict what will come out. End with the trap question “pricing policy for NGOs” → honest refusal (teaser for Block D). Then pause.

Notes: Resume after the break. State the point plainly: a mediocre RAG almost always has mediocre chunking. Keep the same analogy throughout: cutting a book into revision sheets.

Notes: Demo the chunking viewer (tab 2): fixed tiny size → orphan sentences; then enable overlap and show the copper highlighted area.

Notes: Breadcrumbs make the chunk understandable on its own AND improve its retrieval (the words in the title count in the embedding). Show the “structural” strategy in the viewer: sections remain intact.

Notes: Deliberately striking example. In the viewer, the “fixed size” strategy cuts the scale table right in the middle — show it. Real cases: price scales, compatibility matrices, HR scales.

Notes: Emphasize security: the filter by access level applies TO RETRIEVAL, never after generation (if the chunk enters the prompt, the information can leak into the response). Quotes = condition of trust in business. Start exercise 1 just after this slide.

Notes: Shock demo to the similarity explorer (tab 3): click ERR-4471 vs ERR-4472 → similarity almost 1 while they designate different things. Other victims: rare proper names, product references, legal article numbers.

Notes: BM25 = classic keyword search algorithm (cite it without detailing it). Message: vector and lexical search are not competitors; they are complementary, and each covers the other's blind spots.

Notes: “Vanilla RAG” means the simple pipeline covered so far. Treat the following techniques as conditional tools: add a stage only when an evaluation set exposes the failure it is meant to fix.

Notes: “How does it work for new people?” can become “telework conditions for new hires,” “telework seniority requirement,” and “telework eligibility exceptions.” HyDE can help with short queries, but it can also introduce the wrong angle. Compare it on an evaluation set.

Notes: A cross-encoder is slower, so it is applied to candidates rather than the entire corpus. Do not promise universal improvement: measure recall, precision, latency, and cost on the same queries.

Notes: A score produced by the same model is not external evidence. Calibrate the decision on an annotated evaluation set containing expected answers, expected absences, and known contradictions. Distinguish the families: **Corrective RAG** grades passages and retries retrieval; **Adaptive RAG** chooses a route based on the query; **Self-RAG** emits reflection signals. In all three, a self-score is a **routing signal**, never external evidence.

Notes: The best pipeline is not the most complex one. It is the simplest pipeline that meets defined quality, security, latency, and cost thresholds on a representative evaluation set.

Notes: Use the cook-and-runner analogy: replacing the cook does not save a dessert when the runner brought salt instead of sugar. Avoid unsupported percentages; diagnose each incident from evidence.

Notes: Do not add HyDE or reranking automatically. If the document is absent from the index, neither technique can recreate it.

Notes: A clean, cited answer can still be stale. Corpus freshness is a system property, not a model capability.

Notes: The evidence gate decides whether the system may answer; the prompt then constrains generation. Model self-evaluation replaces neither verifiable citations nor an external evaluation set.

Notes: Ask learners to restate points 4–6 with the technique, expected evidence, and added cost.

Notes: Do not promise a nonexistent RAG-focused Session 5. Robust-RAG improvement is now part of this session through the intervention ladder and diagnostic exercise.