Exercises — Session 4: RAG, giving memory to the model

Program: Applied AI — Yann Isola · Intermediate level Reminder: RAG = Retrieval-Augmented Generation: retrieve relevant evidence before generating an answer.

---

Exercise 1 — Chunking strategy on a real document

Duration: 10 minutes · Format: pairs · Material: paper or text editor

Context

You must ingest the following document into an enterprise RAG system. Your job: decide where to cut, and justify each choice.

Document to cut

Professional expense reimbursement policy — v3.2 (updated: March 2026)

1. General principles Any employee can request reimbursement of expenses incurred as part of their missions. Requests are made via the NotaFrais tool within 30 days after the expense. After this period, the request is refused unless waived by the manager.

2. Reimbursement scales

Fee Type	Ceiling	Proof required
Meals (travel)	25 € / meal	Ticket or invoice
Hotel (France)	130 € / night	Named invoice
Hotel (foreign)	180 € / night	Named invoice
Kilometers (personal vehicle)	0.52 € / km	Journey report

These ceilings are revised each year in January. Exceptional overruns must be validated before the expenditure by the department manager.

3. Special cases Customer invitations follow a separate procedure: form REP-07 must be attached to the request, with the nominal list of guests. The ceiling is €60 per guest. Any invitation exceeding €500 in total requires validation from the sales director.

4. Contact For any questions: accounting department, accounts@example.fr, ext. 4412.

Questions

1. Propose a division into chunks. For each chunk: indicate its content (by section or paragraph numbers) and its approximate size.
2. Justify: why these borders and not others?
3. The reimbursement table: what do you do with it, and why?
4. Metadata: list at least 4 metadata fields that you would attach to each chunk, and give a concrete use for each (filtering, citation, security, etc.).
5. Trick question: a user asks, “what is the ceiling for a hotel in Berlin?” Which chunk(s) should be retrieved? Does your chunking strategy allow that?

Indicative correction

1. Recommended cutting (4 chunks):

• Chunk A: Entire Section 1 (general principles) — a complete unit of meaning, ~80 tokens ⚠.
• Chunk B: Entire Section 2 with the table intact and its introduction + the two sentences that follow (annual review, overruns) — the table and its associated rules form a whole, ~120 tokens ⚠.
• Chunk C: Section 3 (special cases / customer invitations) — separate subject, ~90 tokens ⚠.
• Chunk D: Section 4 (contact) — very short; acceptable to merge it with chunk C or keep it alone with good metadata. Both answers are valid.

2. Rationale: cut out of the structure (the section titles), never in the middle of a sentence; each chunk is self-contained — understandable without reading the others. The document is short: there is no need to aim for 800 tokens, we favor unity of meaning. 3.Table: never cut. A line “130 € / night | Nominative invoice” separated from its header and the “Type of expense” column is unusable. We keep the entire table in a single chunk, with the introductory sentence. Advanced tip: prefix the chunk with a summary (“Expense reimbursement scales: meals, hotel, kilometers”).
3. Expected metadata: source (Refund Policy v3.2) → quote in answer; section (1, 2, 3, 4) → precise quote; date (March 2026) → filtering of outdated versions; version (3.2) → detection of duplicates between versions; niveau_acces (all employees) → security filtering upon retrieval; type (internal policy / HR-finance) → thematic filtering.
4. “Hotel in Berlin” must output chunk B (scales, line “Hotel (foreign): €180”). Trap: “Berlin” does not appear anywhere in the document – ​​it is the embedding which must bring “Berlin” closer to “foreign”. If the table has been cut out of context, the retrieval fails. This is exactly the kind of case that warrants keeping table + context together.

---

Exercise 2 — Complete design of a RAG pipeline ⭐ (central exercise)

Duration: 25 minutes · Format: groups of 3–4 · Materials: large sheet or shared document

Context

You are the team responsible for designing “AssistDoc”, an internal customer support assistant for a software publisher. Corpus available:

• 450 product sheets (heterogeneous formats, some with compatibility tables);
• 1,200 support tickets resolved (customer question + agent response);
• 1 user manual of 300 pages (PDF structured in chapters and sections);
• 80 dated release notes (changelogs), some of which contradict old product sheets;
• 1 error code base: 600 codes of the type ERR-4471 with their explanation.

Support agents will ask questions like: “The customer has error ERR-4471 on version 12.3, what should I do?” or “Is the export module compatible with SSO?” (SSO = Single Sign-On, single authentication).

Your mission — design the pipeline by responding to the 6 components

Part 1 — Ingestion. Draw the (offline) ingestion pipeline from end to end: which steps, in what order, for which documents?

Part 2 — Differentiated Chunking. The same division for the 5 types of documents would be an error. Propose a strategy by document type (size, borders, case of tables, case of question/answer tickets, case of error codes).

Part 3 — Metadata. Define the metadata schema: which fields, and for what use (filtering, citation, management of contradictions between changelogs and old files)?

Part 4 — Query. Draw the query pipeline (online) for the question “The customer has error ERR-4471 on version 12.3, what should I do?”. Why is a purely vector search likely to fail on this specific question? What do you suggest?

Part 5 — Prompt framing. Write the system instruction you would give to the model at generation time (3–5 lines), including: restriction to provided context, citation requirement, and behavior when the answer is absent.

Part 6 — Test plan. Propose 3 test questions that each check a different risk (one for exact codes, one for version contradictions, one for absence of information).

Indicative correction

Part 1. Collection → conversion into clean text (PDF → text, retaining the structure of titles and tables) → division into chunks according to the strategy of part 2 → attachment of metadata → calculation of the embedding of each chunk (the meaning becomes geometry) → indexing in the vector base + lexical index (keywords) in parallel. Scheduled re-ingest for each new release note.

Part 2.

• Product sheets: 1 sheet = 1 to 3 chunks, cut out on the sections; entire compatibility tables, never cut.
• Resolved tickets: 1 ticket = 1 chunk (the question/answer pair is the unit of meaning; separating the question from its answer destroys the value). Filter low quality tickets before ingestion.
• 300 page manual: cutting on the chapter → section → subsection hierarchy, chunks of 300–800 tokens ⚠ with overlap of 10–20% ⚠; prefix each chunk of its breadcrumbs (“Chapter 4 > Export > Supported formats”) so that it remains autonomous.
• Version notes: 1 note = 1 chunk if short; cut by section otherwise; the date and version number in mandatory metadata.
• Error codes: 1 code = 1 chunk (small, self-contained), with the code itself repeated in the text and in metadata.
• Any answer that applies a single size to the entire corpus must be challenged in the debrief.

Part 3. source_type (sheet / ticket / manual / changelog / error_code), produit, version_produit, date, chapitre_section, code_erreur (if applicable), qualite (for tickets). Uses: filter by version (“retrieve only documents ≤ 12.3” or favor the most recent), cite the exact source in the response, resolve contradictions by giving priority to the most recent changelog — rule expressible using date + source_type.

Part 4. Question → embedding → top-k vector search + keyword search on ERR-4471 (and metadata filter version_produit) → merging results → assembly of the prompt → generation. Pure vector risk: ERR-4471 is an arbitrary string without semantic content; its embedding is almost indistinguishable from that of ERR-4472. Lexical (exact) search finds the right code for sure → hybrid search essential here.

Part 5. Expected example:

“You are the customer support assistant. Answer only from the excerpts provided below. Always cite the source of each statement (document name and section). If the excerpts do not contain the answer, respond: “I cannot find this information in the documentation provided” and suggest rephrasing or checking the source document. If two excerpts contradict each other, favor the most recent one and report the contradiction.”

Part 6. Examples:

1. Exact code: “What does ERR-4471 mean?” → verifies that hybrid search finds the correct code (and not ERR-4417).
2. Contradiction: “Is the export module SSO compatible?” when a 2024 file says no and the 12.2 changelog says yes → checks priority to the most recent source and explicit contradiction reporting.
3. Absence: “What is the pricing policy for NGOs?” (absent from the corpus) → verifies honest refusal instead of invention.

---

Exercise 3 — Autopsy of a failed RAG

Duration: 15 minutes · Format: individual then shared

Context

You audit the internal RAG of a company. For each incident below, you are given: the question, the chunks retrieved (summaries), and the response generated. Diagnose: is the fault in the retrieval or in the generation? What to correct?

Incident A

• Question: “What is the withdrawal period for a contract signed online?”
• Chunks retrieved: [1] electronic signature procedure; [2] archiving of contracts; [3] commercial follow-up email template.
• Answer: “The withdrawal period is 14 days.” (exact in French law, but absent from chunks)

Incident B

• Question: “What is the escalation procedure for incident P1?”
• Chunks retrieved: [1] definition of priorities P1–P4; [2] complete P1 escalation procedure; [3] on-call directory.
• Answer: “In the event of P1, notify your manager within 4 hours.” (chunk 2 says: within 15 minutes, on-call, not the manager)

Incident C

• Question: “How many days of paternity leave with us in 2026?”
• Chunks retrieved: [1] leave agreement 2022 (28 days); [2] intranet page 2021; [3] HR FAQ 2022.
• Answer: “28 days, according to the leave agreement.” (a 2025 amendment increases the leave to 32 days, but it has never been ingested)

Incident D

• Question: “Does the AlphaSync product support the ISO 27001 standard?” (ISO = International Organization for Standardization)
• Chunks retrieved: [1] AlphaSync file — security; [2] BetaSync sheet — certifications (mentions ISO 27001); [3] generic press release.
• Answer: “Yes, AlphaSync is ISO 27001 certified.” (BetaSync is)

Questions

1. For each incident: retrieval or generation? Justify in one sentence.
2. For each incident: a concrete corrective action.
3. Summary question: what systematic debugging reflex do these four cases illustrate?

Indicative correction

Incident A — Faulty retrieval + absent framing. The chunks are off-topic; the model responded from its internal memory (the response is just by chance, but the system is out of control). Fixes: improve retrieval (is the relevant legal document indexed? is the breakdown good?) and add the instruction “answer only from context” — without it you cannot distinguish a sourced answer from a made-up answer.

Incident B — Failed generation. The retrieval worked perfectly (chunk 2 contains the correct procedure); the model was poorly rendered. This is the rarest case but it does exist. Corrections: strengthen the prompt (require textual citation of the procedure), test a more capable model, reduce noise in the context (was chunk 3 useful?).

Incident C — Failed retrieval: missing document. The model correctly used the provided chunks — which were all out of date. No model can find an endorsement that has never been ingested. Fixes: index update process (re-ingestion with each new agreement); metadata date + favor recent documents; alert on the age of the cited sources.Incident D — Partially failed retrieval + faulty generation. The retrieval mixed two products (BetaSync chunk retrieved for an AlphaSync issue) and the model merged the information without distinguishing the sources. Corrections: metadata produit + filtering on retrieval; generation instruction requiring attribution of each statement to its source; non-confusion test between products with similar names.

Expected summary: before blaming the model, always inspect the retrieved chunks. RAG debugging starts with retrieval: are the right documents indexed, well chunked, correctly filtered, and actually retrieved?

---

Exercise 4 — From vanilla RAG to robust RAG

Duration: 15 minutes · Format: groups of three · Goal: choose the smallest evidence-backed correction

Situation

AssistDoc’s vanilla RAG uses one query, hybrid top-3 retrieval, and generation. On an annotated evaluation set of 40 questions:

• precise error-code questions work;
• vague questions from new employees often miss the right document;
• the correct chunk sometimes appears between ranks 6 and 12;
• when no passage supports an answer, the model still calls itself “confident.”

Task

For each symptom, choose one first intervention and the evidence required to retain it:

1. vague query: multi-query, HyDE, both, or neither?
2. useful candidate present but poorly ranked: increase k or add cross-encoder reranking?
3. insufficient evidence: what must the evidence gate check before answering?
4. why is self-correction by the same model not external evidence?
5. complete the scorecard: retrieval quality, citation support, correct refusal, latency, cost, and privacy.

Suggested answer

1. Start with multi-query because its reformulations remain visible and auditable. Test HyDE separately if short queries still fail to match document-shaped passages. A hypothetical document never becomes a source.
2. When the useful chunk is already among the candidates, retrieve broadly and apply cross-encoder reranking. Measure the precision gain and latency cost; do not rerank the whole corpus.
3. The evidence gate checks relevance, authorization, freshness, contradictions, and whether every material claim can be cited. If the gate remains closed, reformulate once or refuse.
4. The same model can confirm its own mistake. Self-correction is a routing signal, not proof; require verifiable citations and thresholds calibrated on the external evaluation set.
5. Keep a stage only when it improves the targeted metrics without exceeding latency and cost budgets or crossing an unauthorized privacy boundary. Web search over confidential data requires an explicit policy.