# Exercises — Session 4: RAG, giving memory to the model **Program:** Applied AI — Yann Isola · Intermediate level **Reminder:** RAG = *Retrieval-Augmented Generation*: retrieve relevant evidence before generating an answer. --- ## Exercise 1 — Chunking strategy on a real document **Duration:** 10 minutes · **Format:** pairs · **Material:** paper or text editor ### Context You must ingest the following document into an enterprise RAG system. Your job: decide **where to cut**, and justify each choice. ### Document to cut > **Professional expense reimbursement policy — v3.2 (updated: March 2026)** > > **1. General principles** > Any employee can request reimbursement of expenses incurred as part of their missions. Requests are made via the NotaFrais tool within 30 days after the expense. After this period, the request is refused unless waived by the manager. > > **2. Reimbursement scales** > > | Fee Type | Ceiling | Proof required | > |---|---|---| > | Meals (travel) | 25 € / meal | Ticket or invoice | > | Hotel (France) | 130 € / night | Named invoice | > | Hotel (foreign) | 180 € / night | Named invoice | > | Kilometers (personal vehicle) | 0.52 € / km | Journey report | > > These ceilings are revised each year in January. Exceptional overruns must be validated **before** the expenditure by the department manager. > > **3. Special cases** > Customer invitations follow a separate procedure: form REP-07 must be attached to the request, with the nominal list of guests. The ceiling is €60 per guest. Any invitation exceeding €500 in total requires validation from the sales director. > > **4. Contact** > For any questions: accounting department, accounts@example.fr, ext. 4412. ### Questions 1. **Propose a division** into chunks. For each chunk: indicate its content (by section or paragraph numbers) and its approximate size. 2. **Justify**: why these borders and not others? 3. **The reimbursement table**: what do you do with it, and why? 4. **Metadata**: list at least 4 metadata fields that you would attach to each chunk, and give a concrete use for each (filtering, citation, security, etc.). 5. **Trick question**: a user asks, “what is the ceiling for a hotel in Berlin?” Which chunk(s) should be retrieved? Does your chunking strategy allow that? ### Indicative correction 1. **Recommended cutting (4 chunks):** - Chunk A: Entire Section 1 (general principles) — a complete unit of meaning, ~80 tokens ⚠. - Chunk B: Entire Section 2 **with the table intact and its introduction + the two sentences that follow** (annual review, overruns) — the table and its associated rules form a whole, ~120 tokens ⚠. - Chunk C: Section 3 (special cases / customer invitations) — separate subject, ~90 tokens ⚠. - Chunk D: Section 4 (contact) — very short; acceptable to merge it with chunk C or keep it alone with good metadata. Both answers are valid. 2. **Rationale:** cut out of the **structure** (the section titles), never in the middle of a sentence; each chunk is self-contained — understandable without reading the others. The document is short: there is no need to aim for 800 tokens, we favor unity of meaning. 3. **Table: never cut it.** A line like “130 € / night | Named invoice” is unusable when separated from the header and the “Fee Type” column. Keep the whole table in one chunk with its introductory sentence. Advanced tip: prefix the chunk with a short summary such as “Expense reimbursement scales: meals, hotels, mileage.” 4. **Expected metadata:** `source` (Reimbursement Policy v3.2) → citation in the answer; `section` (1, 2, 3, 4) → precise citation; `date` (March 2026) → filter out outdated versions; `version` (3.2) → detect duplicates across versions; `access_level` (all employees) → security filtering at retrieval; `type` (internal policy / HR-finance) → topical filtering. 5. **“Hotel in Berlin”** should retrieve chunk B (scales, line “Hotel (foreign): €180”). Trap: “Berlin” does not appear in the document. The embedding must bring “Berlin” close enough to “foreign”. If the table has been cut away from its context, retrieval fails. This case justifies keeping the table and context together. --- ## Exercise 2 — Complete design of a RAG pipeline ⭐ (central exercise) **Duration:** 25 minutes · **Format:** groups of 3–4 · **Materials:** large sheet or shared document ### Context You are the team responsible for designing “AssistDoc”, an internal customer support assistant for a software publisher. Corpus available: - **450 product sheets** (heterogeneous formats, some with compatibility tables); - **1,200 support tickets resolved** (customer question + agent response); - **1 user manual** of 300 pages (PDF structured in chapters and sections); - **80 dated release notes** (changelogs), some of which contradict old product sheets; - **1 error code base**: 600 codes of the type `ERR-4471` with their explanation. Support agents will ask questions like: *“The customer has error ERR-4471 on version 12.3, what should I do?”* or *“Is the export module compatible with SSO?”* (SSO = *Single Sign-On*, single authentication). ### Your mission — design the pipeline by responding to the 6 components **Part 1 — Ingestion.** Draw the (offline) ingestion pipeline from end to end: which steps, in what order, for which documents? **Part 2 — Differentiated Chunking.** The same division for the 5 types of documents would be an error. Propose a strategy **by document type** (size, borders, case of tables, case of question/answer tickets, case of error codes). **Part 3 — Metadata.** Define the metadata schema: which fields, and for what use (filtering, citation, management of contradictions between changelogs and old files)? **Part 4 — Query.** Draw the query pipeline (online) for the question *“The customer has error ERR-4471 on version 12.3, what should I do?”* Why is a **purely vector** search likely to fail on this specific question? What do you suggest? **Part 5 — Prompt framing.** Write the system instruction you would give to the model at generation time (3–5 lines), including: restriction to provided context, citation requirement, and behavior when the answer is absent. **Part 6 — Test plan.** Propose 3 test questions that each check a different risk (one for exact codes, one for version contradictions, one for absence of information). ### Indicative correction **Part 1.** Collection → conversion into clean text (PDF → text, retaining the structure of titles and tables) → division into chunks according to the strategy of part 2 → attachment of metadata → calculation of the embedding of each chunk (the meaning becomes geometry) → indexing in the vector base + lexical index (keywords) in parallel. Scheduled re-ingest for each new release note. **Part 2.** - *Product sheets*: 1 sheet = 1 to 3 chunks, cut out on the sections; **entire compatibility tables**, never cut. - *Resolved tickets*: **1 ticket = 1 chunk** (the question/answer pair is the unit of meaning; separating the question from its answer destroys the value). Filter low quality tickets before ingestion. - *300 page manual*: cutting on the chapter → section → subsection hierarchy, chunks of 300–800 tokens ⚠ with overlap of 10–20% ⚠; prefix each chunk of its breadcrumbs (“Chapter 4 > Export > Supported formats”) so that it remains autonomous. - *Version notes*: 1 note = 1 chunk if short; cut by section otherwise; the **date and version number in mandatory metadata**. - *Error codes*: **1 code = 1 chunk** (small, self-contained), with the code itself repeated in the text and in metadata. - Any answer that applies a single size to the entire corpus must be challenged in the debrief. **Part 3.** `source_type` (sheet / ticket / manual / changelog / error_code), `produit`, `version_produit`, `date`, `chapitre_section`, `code_erreur` (if applicable), `qualite` (for tickets). Uses: filter by version (“retrieve only documents ≤ 12.3” or favor the most recent), cite the exact source in the response, **resolve contradictions by giving priority to the most recent changelog** — rule expressible using `date` + `source_type`. **Part 4.** Question → embedding → top-k vector search **+ keyword search** on `ERR-4471` (and metadata filter `version_produit`) → merging results → assembly of the prompt → generation. Pure vector risk: `ERR-4471` is an arbitrary string without semantic content; its embedding is almost indistinguishable from that of `ERR-4472`. Lexical (exact) search finds the right code for sure → **hybrid search** essential here. **Part 5.** Expected example: > “You are the customer support assistant. Answer **only** from the excerpts provided below. Always cite the source of each statement (document name and section). If the excerpts do not contain the answer, respond: "I cannot find this information in the documentation provided" and suggest rephrasing or checking the source document. If two excerpts contradict each other, favor the most recent one and report the contradiction.” **Part 6.** Examples: 1. *Exact code*: “What does ERR-4471 mean?” → verifies that hybrid search finds the correct code (and not ERR-4417). 2. *Contradiction*: “Is the export module SSO compatible?” when a 2024 file says no and the 12.2 changelog says yes → checks priority to the most recent source and explicit contradiction reporting. 3. *Absence*: “What is the pricing policy for NGOs?” (absent from the corpus) → verifies **honest refusal** instead of invention. --- ## Exercise 3 — Autopsy of a failed RAG **Duration:** 15 minutes · **Format:** individual then shared ### Context You audit the internal RAG of a company. For each incident below, you are given: the question, the chunks retrieved (summaries), and the response generated. **Diagnose: is the fault in the retrieval or in the generation? What to correct?** ### Incident A - **Question:** “What is the withdrawal period for a contract signed online?” - **Chunks retrieved:** [1] electronic signature procedure; [2] archiving of contracts; [3] commercial follow-up email template. - **Answer:** “The withdrawal period is 14 days.” *(exact in French law, but absent from chunks)* ### Incident B - **Question:** “What is the escalation procedure for incident P1?” - **Chunks retrieved:** [1] definition of priorities P1–P4; [2] complete P1 escalation procedure; [3] on-call directory. - **Answer:** “In the event of P1, notify your manager within 4 hours.” *(chunk 2 says: within 15 minutes, on-call, not the manager)* ### Incident C - **Question:** “How many days of paternity leave with us in 2026?” - **Chunks retrieved:** [1] leave agreement 2022 (28 days); [2] intranet page 2021; [3] HR FAQ 2022. - **Answer:** “28 days, according to the leave agreement.” *(a 2025 amendment increases the leave to 32 days, but it has never been ingested)* ### Incident D - **Question:** “Does the AlphaSync product support the ISO 27001 standard?” (ISO = International Organization for Standardization) - **Chunks retrieved:** [1] AlphaSync file — security; [2] BetaSync sheet — certifications (mentions ISO 27001); [3] generic press release. - **Answer:** “Yes, AlphaSync is ISO 27001 certified.” *(BetaSync is)* ### Questions 1. For each incident: **retrieval or generation?** Justify in one sentence. 2. For each incident: **a concrete corrective action.** 3. Summary question: what systematic debugging reflex do these four cases illustrate? ### Indicative correction **Incident A — Faulty retrieval + absent framing.** The chunks are off-topic; the model responded from its internal memory (the response is just by chance, but the system is out of control). Fixes: improve retrieval (is the relevant legal document indexed? is the breakdown good?) **and** add the instruction “answer only from context” — without it you cannot distinguish a sourced answer from a made-up answer. **Incident B — Failed generation.** The retrieval worked perfectly (chunk 2 contains the correct procedure); the model was poorly rendered. This is the rarest case but it does exist. Corrections: strengthen the prompt (require textual citation of the procedure), test a more capable model, reduce noise in the context (was chunk 3 useful?). **Incident C — Failed retrieval: missing document.** The model correctly used the provided chunks, but they were all out of date. No model can find an amendment that has never been ingested. Fixes: update the index process (re-ingest after each new agreement); use metadata `date` and favor recent documents; alert on the age of cited sources. **Incident D — Partially failed retrieval + faulty generation.** The retrieval mixed two products (a BetaSync chunk retrieved for an AlphaSync question), and the model merged the information without distinguishing the sources. Corrections: metadata `product` + filtering at retrieval; generation instruction requiring attribution for each statement; non-confusion test between products with similar names. **Expected summary:** before blaming the model, **always inspect the retrieved chunks**. RAG debugging starts with retrieval: are the right documents indexed, well chunked, correctly filtered, and actually retrieved? --- ## Exercise 4 — From vanilla RAG to robust RAG **Duration:** 15 minutes · **Format:** groups of three · **Goal:** choose the smallest evidence-backed correction ### Situation AssistDoc's vanilla RAG uses one query, hybrid top-3 retrieval, and generation. On an annotated **evaluation set** of 40 questions: - precise error-code questions work; - vague questions from new employees often miss the right document; - the correct chunk sometimes appears between ranks 6 and 12; - when no passage supports an answer, the model still calls itself “confident.” ### Task For each symptom, choose **one first intervention** and the evidence required to retain it: 1. vague query: multi-query, HyDE, both, or neither? 2. useful candidate present but poorly ranked: increase k or add cross-encoder reranking? 3. insufficient evidence: what must the evidence gate check before answering? 4. why is self-correction by the same model not external evidence? 5. complete the scorecard: retrieval quality, citation support, correct refusal, latency, cost, and privacy. ### Suggested answer 1. Start with **multi-query** because its reformulations remain visible and auditable. Test HyDE separately if short queries still fail to match document-shaped passages. A hypothetical document never becomes a source. 2. When the useful chunk is already among the candidates, retrieve broadly and apply **cross-encoder reranking**. Measure the precision gain and latency cost; do not rerank the whole corpus. 3. The **evidence gate** checks relevance, authorization, freshness, contradictions, and whether every material claim can be cited. If the gate remains closed, reformulate once or refuse. 4. The same model can confirm its own mistake. Self-correction is a routing signal, not proof; require verifiable citations and thresholds calibrated on the external evaluation set. 5. Keep a stage only when it improves the targeted metrics without exceeding **latency** and **cost** budgets or crossing an unauthorized **privacy** boundary. Web search over confidential data requires an explicit policy.