Trainer’s Guide — Session 4 (Intermediate level)

RAG: give memory to the model

Program: Applied AI — Yann Isola Duration: 2 hours Audience: professionals who have followed sessions 1 to 3 (embeddings, context window, prompting) Source module: Module 3, part 1

1. Educational objectives

At the end of the session, each participant should be able to:

Explain why the knowledge of an LLM (Large Language Model) is limited: frozen at the cut-off date, public only, stored in a compressed manner and with losses in the weights of the model.
Describe the principle of RAG (Retrieval-Augmented Generation): transform each question into an “open book exam”.
Draw the two pipelines: ingestion (offline) and query (online), with their respective stages.
Make chunking decisions: chunk size, overlap, structure cutting, entire tables.
Diagnose a RAG failure by separating retrieval errors from generation errors.
Select and evaluate a robust intervention—multi-query, HyDE, reranking, or an evidence gate—without ignoring latency, cost, or privacy.

2. Prerequisites and materials

Element	Detail
Participant prerequisites	Session 1 (embeddings: meaning becomes geometry), Session 2-3 (context, prompting)
Training material	Video projector, session slides, interactive web page `webpage/index.html` (works offline)
Participant materials	Laptop recommended for exercises 2 and 3 (paper possible)
Documents to print	Worksheet, quiz, and six rotating exit tickets

Pre-session check: open webpage/index.html; exercise the pipeline simulator, chunking visualizer, and robust-RAG cascade. No internet connection is required.

3. Timed course (120 minutes)

Block A — The problem: a frozen memory (0:00 → 0:20, 20 min)

Time	Activity	Slides
0:00–0:05	Home + express reminder Session 1: “meaning becomes geometry” (embeddings)	1–3
0:05–0:15	The three limits of the memory of an LLM	4–7
0:15–0:20	Mini-demo: question about a fictitious internal fact → the model cannot know	8

Trainer notes:

Open with a question to the group: “If I ask a model how many leave days you have left, what will it answer?” Expected answers: it does not know, or worse, it invents an answer. Both cases illustrate the problem.
Hammer the three limits:

Fixed: knowledge stops on the training cutoff date (knowledge cutoff). Anything later does not exist for the model.
Public: the model was trained on public data. Your internal documents, contracts, procedures, tickets — never seen before.
Lossy: even public knowledge is stored with loss in the model weights. Analogy: the model has “read” Wikipedia, but it cannot recite it word for word, like you after reading a book ten years ago.

Establish the central analogy now: closed-book exam vs open-book exam. The LLM alone is a student taking an exam from memory. RAG lets that student bring the right documents.
Frequent trap: participants think that fine-tuning (partial re-training) is the solution to inject knowledge. Note the objection on the board, come back to it at the end of block B: fine-tuning learns behaviors (style, format), not reliable and up-to-date facts; it is expensive and must be redone for each document update. The RAG updates the index in seconds.

Block B — The principle of RAG and the two pipelines (0:20 → 0:55, 35 min)

Time	Activity	Slides
0:20–0:30	Definition of RAG + overview of both pipelines	9–11
0:30–0:40	Ingestion pipeline (offline): slice → vectorize → index	12–14
0:40–0:50	Query pipeline (inline): vectorize the question → retrieve k chunks → assemble the prompt → generate	15–17
0:50–0:55	Interactive demo: pipeline simulator on web page	17

Trainer notes:

Define each term on first use — course rule:
RAG = Retrieval-Augmented Generation: retrieve relevant evidence before generating an answer.
Chunk = fragment of document (we will keep the English word, standard in the profession, by translating it once: “piece”).
Embedding = vector embedding, seen in Session 1: a text becomes a point in a geometric space where proximity = similarity of meaning.
Top-k = the k closest results (k is a number we choose, often 3 to 10 ⚠).
Insist on the temporal separation of the two pipelines:
Ingestion: is done once (then each time the documents are updated), offline, without a user. This is the preparation of the library.
Query: is done with each question, online, in a few hundred milliseconds ⚠. This is the library consultation.
Diagram on the board (do it by hand, even if it is in the slides — the gesture helps memorization):

INGESTION (offline, once)
Documents → Chunking → Embedding each chunk → Vector index

QUERY (online, for each question)
Question → Question embedding → Retrieve the k closest chunks
        → Assemble the prompt (instruction + chunks + question) → Generate

Conceptual key point: the question and the chunks live in the same geometric space. This is why “search for chunks close to the question” makes sense. Link explicitly to Session 1.
Concrete example to be given orally from start to finish: “What is the telework policy for new hires?”

The question becomes a vector.
The index returns 4 chunks: two extracts from the internal regulations, an extract from the 2025 teleworking agreement, an extract from the onboarding guide.
The assembled prompt: “Answer only from the following context. [4 chunks] Question: …”
The model generates a response citing the telework agreement.

Demo (5 min): project webpage/index.html, “Pipeline simulator” tab. Type a question, go through the 4 steps one by one. Ask the group to predict which chunks will come out before clicking.

Break (0:55 → 1:05, 10 min)

Block C — Chunking decisions and metadata (1:05 → 1:30, 25 min)

Time	Activity	Slides
1:05–1:15	Chunking: size, overlap, structure, tables	18–21
1:15–1:20	Metadata: source, section, date, access level	22
1:20–1:30	Exercise 1 in pairs: strategy for cutting up a real document	—

Trainer notes:

Chunking is the #1 quality lever of a RAG. Tell it like it is.
The four decisions:

Size: typically 300 to 800 tokens ⚠ (reminder: 1 token ≈ 0.75 words in English, a little less in French ⚠). Too small = the chunk loses its context (“he” — who is that, “he”?). Too big = the chunk mixes several subjects and its embedding becomes a fuzzy average.
Overlap: make the chunks overlap by 10 to 20% ⚠ so as not to cut information right at the border.
Split by structure: titles, sections, paragraphs — never in the middle of a sentence. A chunk = ideally a unit of meaning.
Whole tables: never cut a table in two. A table row without its header is unreadable (example: “42 | 15% | yes” — what are we talking about?).

Analogy: divide a book into revision sheets. A good sheet is self-contained (understandable on its own), neither too short nor too long, and does not cut a conjugation table in two.
Metadata: each chunk carries a label — source (which document), section, date, access level. Three uses:
Filtering before searching: “only search in HR documents after 2024”.
Citations: The answer can point to the source document — essential for trust and verification.
Security: a salesperson must not retrieve chunks from the payroll file. The filter by access level is done at retrieval, not after generation.
Exercise 1 (10 min, pairs): see exercise sheet. Distribute the extract of the document provided. Circulate between pairs. Express debrief: 2 pairs present their division, compare the choices on the integrated table.

Block D — Hybrid search, robust RAG, and failure modes (1:30 → 1:50, 20 min)

Time	Activity	Slides
1:30–1:35	Hybrid search: vectors + keywords	23–24
1:35–1:44	Robust-RAG intervention ladder + interactive cascade	25–29
1:44–1:49	Diagnosis, evidence gate, and refusal	30–33
1:49–1:50	Decide: answer, reformulate, or refuse	33

Trainer notes:

Hybrid search: embeddings capture meaning but can miss exact strings. REF-2024-8812 carries little semantic meaning; BM25 retrieves it exactly. Merge vector and lexical candidates before ranking.
Intervention ladder—do not enable everything by default:
1. Multi-query: produce 2–4 reformulations, retrieve for each, then merge and deduplicate.
2. HyDE (Hypothetical Document Embeddings): generate a hypothetical document and use its embedding as a retrieval probe. It is never evidence.
3. Broad retrieval: optimize recall so useful evidence is not discarded too early.
4. Reranking: a cross-encoder reads each question–chunk pair and reranks candidates only. It is slower and more expensive than the first-stage bi-encoder.
5. Evidence gate: answer only when authorized, current, non-contradictory passages support the response; otherwise reformulate or refuse.
Name the loops without presenting them as guarantees: Corrective RAG grades passages and then reformulates or retries retrieval; Adaptive RAG chooses a simple search, a deeper cascade, or refusal based on the query; Self-RAG asks the model for reflection signals during generation. Those self-assessments are routing signals, not external evidence.
Measure before adding: compare variants on the same annotated evaluation set. Track retrieval recall/precision, citation support, correct refusal, latency, and cost. A model’s self-score is not external evidence.
Privacy: web search creates a new data boundary. Never send confidential queries or excerpts to an external service without an explicit policy, filtering, and authorization.
Diagnosis: inspect retrieved chunks before blaming generation. If the document is absent from the index, multi-query, HyDE, and reranking cannot recreate it.
Honest refusal: “I cannot find this information in the supplied documents” is expected when the evidence gate remains closed.

Block E — Quiz, summary, and exit tickets (1:50 → 2:00, 10 min)

Time	Activity
1:50–1:57	Quiz: 11 multiple-choice questions
1:57–1:59	Summary: six ideas to retain
1:59–2:00	Exit tickets

Six ideas to retain:

LLM memory is frozen, public, and lossy; RAG creates an open-book exam.
Two pipelines: offline ingestion and online query.
Chunking is the first lever: overlap, structural boundaries, and whole tables.
Hybrid search combines vectors for meaning with keywords for exact strings.
Robust RAG means reformulate → retrieve broadly → rerank → gate on evidence.
When RAG fails, inspect the chunks first, then answer or refuse.

4. Exit tickets (six questions)

Give each learner one rotating question. Answers should fit in one or two sentences.

Ticket 1. Name the three limits of a language model’s knowledge without RAG. Expected: frozen at the training cutoff, limited to training data, and stored lossily in weights.

Ticket 2. How do ingestion and query pipelines differ? Expected: ingestion prepares the index offline; the query pipeline runs for each user request.

Ticket 3. Why should a table not be split during chunking? Expected: a row without its headers loses meaning.

Ticket 4. A RAG answer is wrong. What is your first diagnostic action? Expected: inspect the retrieved chunks and confirm the expected document exists in the index.

Ticket 5. Why combine vector and keyword search? Expected: vectors capture meaning; lexical search retrieves exact codes and strings.

Ticket 6. Why is a positive model self-assessment insufficient to open the evidence gate? Expected: the same model can repeat its error; require verifiable citations and thresholds calibrated on an external evaluation set.

5. Anticipated difficulties and responses

Difficulty	Response
“Why not put every document in the prompt?”	Context is limited; extra text adds noise, latency, and cost. Retrieval selects relevant passages.
Confusing chunk and query embeddings	They must use the same embedding model and geometric space.
“Fine-tuning does the same thing”	Fine-tuning changes behavior; it is not an updatable factual store with citations and access control.
“More stages always mean better RAG”	False. Each stage must fix a measured failure while respecting latency, cost, and privacy budgets.
“The model says its answer is grounded”	Self-assessment is not external evidence. Require citations, verification, and an annotated evaluation set.
“RAG eliminates hallucinations”	It can reduce them; retrieval and generation can still fail.

6. Extensions

Compare small and large k values, then inspect recall, noise, latency, and cost.
Run the robust cascade in webpage/index.html; ask each group which stages they would retain after evaluation.
Ask which company documents should enter a RAG first and which metadata fields are critical.
Next in the program—Session 5: tools and function calling, allowing a model to act through controlled interfaces.