# Exercises — Advanced Level, Session 8
# “Context, reliability & provenance”

**Program:** Applied AI — Yann Isola
**Audience:** Solutions Architects — preparation *Claude Certified Architect*
**Estimated total duration:** 3:30 to 4:30 hours (individual or pair)
**Prerequisites:** Anthropic API key, Python 3.10+, SDK (SDK = Software Development Kit) `anthropic`, session 8 followed, interactive web page of the session to check your calculations.

> ⚠ All pricing figures, window sizes, quotas and SLAs (Service Level Agreement) used below are **volatile**. Always note the date and source of the price list you use — it's even an explicit requirement of exercise 1.

---

## Exercise 1 — Context budget calculator (60-75 min)

### Context

Your team deploys “ComplianceScan”, the compliance investigation agent seen in session. Before any line of agent code, the architect that you are must answer three questions: **how many turns can the agent last? how much does an investigation cost? which context strategy is required?** You will build the tool that answers — a reusable context budget calculator.

### Input data (load profile)

| Post | Value |
|-------|--------|
| Model Context Window | 200,000 tokens ⚠ |
| Prompt system + compliance policy | 3,200 tokens |
| Tool definitions (12 tools) | 4,100 tokens |
| Average user/agent tour (question + reasoning + tool call) | 800 tokens |
| Average raw tool result | 2,500 tokens |
| Tool result filtered by hook PostToolUse | 400 tokens |
| Output Reserve (`max_tokens`) | 8,000 tokens |
| Typical investigation | 150 revolutions, 1 tool call per revolution |

### Work requested

1. **Calculation model.** Write a Python module `context_budget.py` with a function `simulate(profile, strategy) -> Report` which simulates the size of the context turn by turn and returns: the saturation round (or `None`), the total of input tokens charged cumulatively on the investigation (reminder: in each round, **all the context is returned and re-billed**), and the final distribution by position (system / tools / history / tool results).

2. **Scenario A — raw, no strategy.** Raw tool results, no filtering, no compression. In what turn does the agent saturate? Is the investigation of 150 rounds possible?

3. **Scenario B — PostToolUse hook.** Filtered results (400 tokens instead of 2,500). New saturation trick? Winning factor?

4. **Scenario C — hybrid (hook + summary + sliding window).** In addition to filtering: every 30 rounds, rounds beyond the last 10 are compressed into a summary of 1,500 tokens (the previous summary is absorbed into the new one). Also model the **cost of the compression call** itself. Does the 150 round investigation pass? With what margin?

5. **Economic costing.** For each feasible scenario, calculate the cost of a complete investigation, then 200 investigations/day, using the official price list **of the day** (date and source ⚠). Bonus: add the “system prompt + cached tool definitions” variant and quantify the saving.

6.**Architecture note (1 page max).** Your recommendation: what strategy, what thresholds (compression level, verbatim window size, hook ceilings), what residual risks (what are we losing in the summaries? how does the scratchpad compensate?).

### Success criteria

- [ ] The simulator is parametric (no hard constant in the logic).
- [ ] Scenario A demonstrates figures to support the impossibility of 150 turns.
- [ ] The cost of the compression calls in scenario C is counted (forgetting it is the classic error).
- [ ] The prices used are dated and sourced.
- [ ] The architecture note explicitly arbitrates cost / capacity / risk of loss of information.

### To go further

Compare your heuristic estimates (~4 characters/token in English, different ratio in French) with the official token counting endpoint on 5 real messages. What gap? What safety margin do you deduce for your simulator?

---

## Exercise 2 — Designing a chain of provenance (75-90 min)

### Context

The compliance team agrees to deploy ComplianceScan **on one condition** set by internal audit: *“For any report generated, we must be able to trace each statement back to its primary source, reconstruct what the model saw, and know which version of the system produced what. »* You design and prototype the complete chain of provenance.

### Work requested

1. **Output schema with attribution.** Define a report JSON (JSON = JavaScript Object Notation) schema where each `finding` carries: the assertion, a resolvable source pointer (`tool`, `call_id`, `record_id(s)`), a confidence level (`établi` / `à vérifier` / `inféré`), and where the envelope carries: `prompt_version` (hash + semantic identifier), full model identifier, temperature, timestamp, investigation correlation identifier.

2. **Generation prompt.** Write the prompt (system + injection XML structure separating `<contexte_systeme>`, `<donnees>`, `<instructions>`) which constrains the model to produce this schema, with the explicit rule: *no assertion without source pointer; in the absence of a source, the assertion becomes `inféré` with justification.*

3. **Audit log.** Implement a `AuditLog` (JSON files in a `audit/<date>/<investigation_id>/` tree) that logs: system prompt (with hash), each message, each tool call — **arguments, raw result, filtered result entered at context** (the args/raw/ctx triple), and the final manifest (model, version, parameters, hashes).

4. **Citation checker.** Write `verify_provenance(report, audit_log) -> VerificationResult` which, for each `finding`: (a) checks that `call_id` exists in the journal; (b) verifies that the `record_id` cited appear in the **raw result** of this call; (c) flag any orphan citations. Test it by deliberately corrupting a report (invent a plausible `record_id` — simulate the quote hallucination) and verify that the check detects it.

5. **Mock audit scenario.** In pairs: one plays the auditor and chooses a statement at random from a generated report; the other must, in less than 5 minutes and only with the log, produce: the primary source, what the model actually saw (ctx vs raw), and the version of the prompt. Document what was missed or slowed down.

6.**Basic question (10 lines).** Your management asks: “Can we guarantee the identical regeneration of a report to prove its origin? » Write the architect's honest response: what temperature 0 + pinned version + seed ⚠ guarantee, what they don't guarantee ("deterministic-ish"), and why the audit log — not regeneration — is the real guarantee of compliance.

### Success criteria

- [ ] The diagram distinguishes between the provenance of the **data** (sources) and the provenance of the **generator** (prompt, model, parameters).
- [ ] The checker detects the hallucinatory quote from the corruption test.
- [ ] The journal allows us to answer “what exactly did the model see?” » (ctx ≠ raw).
- [ ] The mock audit is completed in < 5 min.
- [ ] The answer on reproducibility does not over-promise.

### To go further

Add integrity chaining: each log entry includes the hash of the previous entry (hash chain). What do we gain from an auditor? What have we still not proven?

---

## Exercise 3 — Batch processing pipeline (60-90 min)

### Context

New compliance policy v3.3: the **80,000 customer files** must be re-rated this week. Synchronous calls = too slow, too expensive, rate limits. You build the complete pipeline on the **Message Batches API**: submission, monitoring, processing, error recovery — with batch optimization + caching and encryption to support it.

*(For the exercise, work on 200 synthetic files; the code must be sized for 80,000.)*

### Work requested

1. **Generation of the game.** Script `make_dossiers.py`: 200 synthetic JSON files (id, country, declared activity, list of transactions, various flags) — including ~10 deliberately malformed (missing field, questionable encoding) to test downstream robustness.

2. **Submission.** Script `submit_batch.py`:
- builds queries with a robust **`custom_id`** (convention to document: e.g. `v33-{dossier_id}-{hash_court_du_contenu}` — justify each component);
- places the compliance policy (long and identical system block) in **cache** (`cache_control`);
- batch division if necessary (limit of 100,000 requests/batch ⚠ — your code must manage division even if the test set fits in one batch);
- locally persists the submission manifest: `batch_id`, list of `custom_id`, timestamp, prompt version (origin!).

3. **Monitoring.** Script `poll_batch.py`: polls status at reasonable intervals (gradual backoff, no tight loop), logs transitions `in_progress` → `ended`.

4. **Counting and recovery.** Script `collect_results.py`:
- analyzes the results by attaching each result to its file via `custom_id` (reminder: **the order is not guaranteed**);
- route by type: `succeeded` → results base; `errored` → retry file with error details; `expired` / `canceled` → resubmission queue;
- is **idempotent**: relaunched after crashing halfway through without double-processing (justify the mechanism — progress markers, upsert by `custom_id`...);
- produces a completeness report: submitted / passed / failed / missing, with the nominative list of those missing.

5.**Comparative costing.** With today's price list (dated, sourced ⚠), calculate for 80,000 files (shared system 3,200 tokens, variable input ~900 tokens, output ~350 tokens) the cost in 4 configurations: synchronous without cache / synchronous + cache / batch without cache / **batch + cache**. Present the table and the total savings factor. Check your orders of magnitude with the calculator on the session web page.

6. **Operations note (½ page).** 24-hour SLA ⚠: what are the consequences on planning (night window, margin, criticality)? What happens if 3% of requests time out? What is your “campaign completed” criteria?

### Success criteria

- [ ] The convention of `custom_id` is documented and allows correlation + idempotence + audit.
- [ ] The processing processes the 4 terminal statuses and survives the 10 malformed files.
- [ ] `collect_results.py` restarted twice in a row does not double-process anything.
- [ ] The encryption shows the batch + cache accumulation, dated and sourced prices.
- [ ] The completeness report designates by name any missing file.

### To go further

Plug in the output of exercise 2: each batch result feeds the audit log and the provenance schema. A `custom_id` then becomes the thread that connects file → request → response → report → audit. This is exactly the architecture expected of a *Claude Certified Architect*: the three building blocks of the session become one.

---

## Indicative scale (if assessment noted)

| Exercise | Points | Of which |
|---------|--------|------|
| 1 — Context budget | 30 | Simulator 12, scenarios 10, architecture note 8 |
| 2 — Chain of provenance | 40 | Diagram 8, log 10, verifier 12, mock audit 5, reproducibility 5 |
| 3 — Batch pipeline | 30 | Submission 8, counting/rework 12, costing 6, exploitation 4 |

Validation threshold: 60/100. The detection of the hallucinated quote (ex. 2.4) and the idempotence of the analysis (ex. 3.4) are **eliminatory** if absent: these are the two non-negotiable reflexes of the profession.