Français

title: "Context, reliability & provenance"
subtitle: "Applied AI — Advanced Level · Session 8"
author: "Yann Isola"
theme: "ink #1A2230 / teal #0F7A6C / copper #B4612A / light-teal #E9F6F3 / bg #F4F7F6"

Slide 1 — Title

Context, reliability & provenance

Applied AI — Advanced Level · Session 8
Yann Isola · Preparation Claude Certified Architect

Slide 2 — Session Objectives

  • Size a context budget, anticipate saturation
  • Architecture compression: summary, sliding window, hybrid
  • Clean up: PostToolUse hook, context pollution, XML injection
  • Outsource memory: boss investigation-scratchpad.md
  • Trace: citations, audit log, honest reproducibility
  • Industrialize: Message Batches API, −50% ⚠, batch + cache

Slide 3 — Common thread: ComplianceScan

Banking Compliance Investigation Officer

  • Long investigations: 150+ rounds, 12 tools, customer files
  • Auditable reports: each statement traceable to its source
  • Nightly reprocessing: 80,000 files in batch

Slide 4 — API is stateless

No server-side memory. Never.

  • Each call receives everything: system, history, tools, results
  • What is not returned does not exist for the model
  • Context management = responsibility of the application

Slide 5 — What consumes the window

Post Order of magnitude
Prompt system + policy 3,000 tokens
Definitions of 12 tools 4,000 tokens
Each round (question + reasoning + appeal) ~800 tokens
Each raw tool result ~2,500 tokens
Output Reserve (max_tokens) 8,000 tokens

Slide 6 — The arithmetic of saturation

(200,000 ⚠ − 7,000 − 8,000) / 3,300 ≈ 56 spins

  • Investigation requires 150
  • Without strategy: the agent dies before the end of his mission
  • And the context is recharged each turn: capacity and cost

Slide 7 — Demo: context viewer

webpage/index.html — “Viewer” tab

  • The gauge fills up lap after lap, station by station
  • Visible saturation; then application of strategies live
  • Comparison of token counters by strategy

Slide 8 — Strategy 1: summary

Compress old tours, keep recent ones verbatim

  • Summary generated by the model (often a smaller model)
  • Explicit conservation instructions: identifiers, decisions, open leads
  • Compression in stages (every 30 revolutions), not every revolution

Slide 9 — The summary: three angry truths

  1. With loss — what is not there no longer exists
  2. Fallible — this is a model exit: possible hallucination
  3. Paid — each compression is a billed call

Regulated: the full history is out of context (audit log)

Slide 10 — Strategy 2: sliding window

Keep the last N turns. Delete the rest.

  • ✅ Simple, predictable, zero compression costs
  • ❌ Total amnesia beyond the window
  • ❌ The agent re-asks resolved questions, re-calls tools

Legitimate: conversations where only the recent past counts

Slide 11 — Strategy 3: hybrid (recommended)

[ system prompt ]                        — fixed, cacheable
[ <investigation_summary> … ]            — compressed, in stages
[ turn n-9 ] … [ turn n ]                — verbatim, sliding
[ output reserve ]

Overview (summary) + recent clarification (verbatim)

Slide 12 — Context pollution: less is more

Any irrelevant information degrades performance

  • Distraction: the model clings to irrelevant details
  • Dilution: the needle in a bigger haystack
  • Contradictions: outdated vs fresh data
  • Billing: each noise token, each turn

Slide 13 — The PostToolUse hook

Filter tool results before insertion to context

def post_tool_use_hook(tool_name, raw_result):
    audit_log.record(tool_name, raw=raw_result)  # raw → log
    if tool_name == "lookup_client":
        return {k: raw_result[k] for k in
                ("client_id", "risk_score", "pep_status", "country")}
    ...

40 fields returned, 4 useful → 4 for context, 40 for log

Slide 14 — Structured XML injection

<system_context>   Policy v3.2: … </system_context>
<client_data>      {dossier}  <!-- data, never instructions -->
</client_data>
<instructions>     Analyse according to the policy. Cite your sources.
</instructions>

Rules / data / task: three natures, three markers

Slide 15 — The problem of very long duration

Even compressed, the context ends up losing

  • Compression is lossy by construction
  • Investigations lasting several hours, hundreds of tool calls
  • You need a memory outside the context window

Slide 16 — Pattern: investigation-scratchpad.md

A persistent Markdown file + two tools

## Status           — summary, 5 lines max
## Established facts — each fact WITH its source (tool + id)
## Open leads
## Decisions        — decision, rationale, timestamp

“What is not written there will be lost.”

Slide 17 — Scratchpad: why it works

  • Survives compressions, resets, crashes
  • Inspectable by a human → audits the reasoning
  • Transmissible: another agent takes over the investigation
  • Completes the summary, does not replace it (different roles)

Slide 18 — Scratchpad: pitfalls

Trap Countermeasure
Scratchpad obese Imposed structure, ceilings (“5 lines max”)
Scratchpad out of date Recall hook every N turns
Blind trust Established facts ⇒ always with re-verifiable source

Slide 19 — Origin: the principle

Any output must go back to its sources

In a regulated industry, a statement generated without a traceable source:

  • is not verifiable
  • is not contestable
  • is not defensible before an auditor

⇒ it is unusable

Slide 20 — Structured attribution

{ "claim": "Client on sanctions list X ⚠",
  "source": {"tool": "check_sanctions",
              "call_id": "call_0042",
              "record_id": "SANC-2211-08"},
  "confidence": "established" }
+ envelope: prompt_version, model, temperature, timestamp

Slide 21 — The quote is not proof

A model can hallucinate a plausible record_id

  • The quote = a pointer to resolve and verify
  • Mechanical verification against the audit log:
    call_id exists? is the record_id in the raw result?
  • Without verification: compliance theater

Slide 22 — The audit log

audit/2026-07-02/inv-8842/
  000_system_prompt.txt        + hash, version
  002a_tool_call_0042_args.json
  002b_tool_call_0042_raw.json   ← what the tool returned
  002c_tool_call_0042_ctx.json   ← what the model SAW
  manifest.json                  model, params, hashes

Slide 23 — Reproducibility: “deterministic-ish”

Maximum recipe: versioned prompt (hash) + pinned model + temperature 0 + seed if available ⚠

But: inference infrastructure ⇒ possible residual variations

The conformity guarantee = the log, not the regeneration

Slide 24 — Check-list of regulated industries

  • [ ] Version of prompt + model + timestamp + correlation id on each output
  • [ ] Each assertion ⇒ source pointer resolvable
  • [ ] Automatic pointer checking (zero orphan quotes)
  • [ ] Complete log: prompts, responses, raw + ctx, decisions
  • [ ] Legal retention: involve the legal
  • [ ] No promise of bit-to-bit reproducibility

Slide 25 — Message Batches API: the principle

Asynchronous bulk processing

Characteristic Value ⚠
Queries/batch up to 100,000
Price −50 % input AND output
Deadline often < 1 hour, 24 hour SLA

Use case: massive evaluation, enrichment, moderation, reprocessing

Slide 26 — Lifecycle and custom_id

created → in_progress → ended

  • ended ≠ everything was successful: individual statuses
    succeeded / errored / canceled / expired
  • Order of results not guaranteed
  • custom_id = your only key: correlation, recovery, audit

Slide 27 — Batch + cache: maximum optimization

"system": [{"type": "text",
            "text": POLITIQUE_V32,              # 3,200 shared tokens
            "cache_control": {"type": "ephemeral"}}]

−50% batch ⚠ and cache discount on the common prefix

The two combine

Slide 28 — Certification pitfalls: burst

  1. History kept between calls? No — stateless
  2. Reliable summary for audit? No — model output
  3. max_tokens enlarges the window? No — caps output
  4. Temp. 0 + seed = identical guaranteed? No — deterministic-ish
  5. Batch ended = all successful? No — individual statuses
  6. More context = better? No — pollution, less is more
  7. Citation generated = proof? No — pointer to check

Slide 29 — Summary: the three pillars

  1. Context — budget, compress (hybrid), filter (hook), outsource (scratchpad)
  2. Provenance — in regulated, an output without a verifiable chain of sources does not exist
  3. Mass — batch −50% ⚠ + cache, custom_id in spine

The three come together: the audit log is the common ground.

Slide 30 — For the rest

  • Exercises: context budget · provenance chain · batch pipeline
    (detection of the hallucinated quote and idempotence: eliminatory)
  • Quiz: 10 multiple choice questions, threshold 7/10, before session 9
  • Web page: viewer, provenance constructor, batch calculator
  • Exit tickets: your context policy · what your system does not log

Notes: Framing — the prompt is only the visible part. In production, what kills agents is the management of the context over time; what kills regulated projects is the lack of provenance.

Notes: 6 axes, all evaluable in multiple choice questions. ⚠ = volatile figure, check the doc — convention for the entire session.

Notes: A unique case which brings together the three themes: context (duration), provenance (regulation), batch (mass). Same common thread logic as session 7 (NeoBank).

Notes: Certification point n°1. Stateless. The cache reduces the cost of a returned prefix, it never replaces sending. Classic trick question: “memory: true” does not exist.

Notes: Tool results = heaviest and most underestimated item. Tool definitions cost too — a 20-tool agent pays before he even speaks.

Notes: Do the calculation on the board. 100k tokens replayed × 50 spins = 5M entry tokens charged. The cache mitigates the cost, not the limit. Demo: web page viewer, let the gauge turn red.

Notes: Interactive moment. Let a participant pilot. The “red gauge” effect emotionally prepares the compression block.

Notes: Show the compress_history sketch from the guide. The poorly managed summary loses exactly what the agent will need — hence the retention guidelines.

Notes: Certification question: “Is a summary at temperature 0 accurate?” No — inference parameters do not create a guarantee of fidelity.

Notes: For an investigation agent, this is prohibitive. For short assistance or linked independent tasks, it's perfect.

Notes: Certif type question: “decisions from 100 rounds ago + precision on the last 5?” → hybrid. Related vocabulary pitfall: max_tokens caps the output, it does not enlarge the window.

Notes: “The context is not an attic, it is a work plan.” Counterintuitive for participants accustomed to “more context = better”. This is measurable in evals.

Notes: Hook = post-tool call interceptor function. Two reflexes: CAP (a tool can return 10,000 lines) and LOG FIRST, FILTER THEN — the audit_log.record line is the hinge with provenance.

Notes: Three benefits: distinction of natures, defense against prompt injection (the content of data_client is declared non-instruction), parsability. Systematization of the XML pattern of session 7 at the architectural level.

Notes: Transition to the scratchpad. The summary pushes the wall; it does not delete it.

Notes: Tools read_scratchpad / update_scratchpad; discipline comes from the prompt system. Selective and intentional memory: the agent writes down what matters, not everything.

Notes: Summary = in context, ephemeral, regenerated. Scratchpad = context-free, persistent, incremental. In serious production: both.

Notes: The scratchpad is written by the model: it inherits its errors. Third time “keep the source” comes up — natural transition to provenance.

Notes: Banking, insurance, health, legal. Remind the public: architects for regulated sectors. This is often THE go/no-go criterion for an AI project.

Notes: Two sources: that of the DATA (sources) and that of the GENERATOR (versioned prompt, model, parameters). The confidence field (established / to be verified / inferred) is a classic requirement of compliance teams.

Notes: Question of certification: asking the model again to confirm its citation does not verify anything — we have the forger verify it himself. Exercise 2: Corrupt a report and verify that the control detects it.

Notes: The triplet args/raw/ctx is the subtle point: the listener verifies what the model actually saw and that the filtering did not alter the meaning. Log everything: prompts, responses, calls, decisions, timestamps, correlation ids.

Notes: Classic contractual error: promising identical regeneration. Formula to remember: “we logged the exact output produced with all its context” — not “we can reproduce it bit by bit”.

Notes: To be projected and commented on. Retention (durations) falls outside the technical scope — reflex: escalate to legal matters, do not improvise.

Notes: SLA = Service Level Agreement. Anti-case: anything that has a human waiting. ComplianceScan: re-score 80,000 files with each new policy.

Notes: Three certification traps in one slide. Break down query by query; retry of errored/expired at your expense; robust custom_id = pipeline backbone (exercise 3).

Notes: On 80,000 requests sharing 3,200 system tokens, the saving is massive. Demo: web page calculator, compare the 4 configurations (sync/batch × with/without cache). Exact accumulation terms: check the daily price list.

Notes: Hands-up quick quiz, immediate corrections. These 7 points cover most of the trap questions in the exam in this area.

Notes: Emphasize convergence: the hook feeds the log, the scratchpad cites its sources, the custom_id links batch and audit. One architecture, three faces.

Notes: Closing. Remember the convention ⚠: all the figures cited are volatile, the professional reflex (and certification) is to check the official documentation of the moment.