Exercises — Advanced Level, Session 8

“Context, reliability & provenance »

Program : Applied AI — Yann Isola Audience : Solution architects — preparation Claude Certified Architect Estimated total duration: 3:30 to 4:30 a.m. (individual or pair) Prerequisites: Anthropic API key, Python 3.10+, SDK (SDK = Software Development Kit) anthropic , session 8 followed, the interactive web page of the session to check your calculations.

⚠ All pricing figures, window sizes, quotas and SLAs (Service Level Agreement) used below are volatile . Systematically note the date and source of the price list you use – it is even an explicit requirement of exercise 1.

Exercise 1 — Context Budget Calculator (60-75 min)

Context

Your team deploys “ComplianceScan”, the compliance investigation agent seen in session. Before any line of agent code, the architect that you are must answer three questions: how many turns can the agent last? how much does an investigation cost? what context strategy is required? You'll build the tool that answers — a reusable context budget calculator.

Input data (load profile)

Job	Value
Model Context Window	200,000 tokens ⚠
Prompt system + compliance policy	3,200 tokens
Tool definitions (12 tools)	4,100 tokens
Average user/agent tour (question + reasoning + tool call)	800 tokens
Average raw tool result	2,500 tokens
Tool result filtered by PostToolUse hook	400 tokens
Output reserve (`max_tokens` )	8,000 tokens
Typical investigation	150 revolutions, 1 tool call per revolution

Work requested

Calculation model. Write a Python module context_budget.py with a function simulate(profile, strategy) -> Report which simulates the size of the context turn by turn and returns: the saturation turn (or None ), the total of cumulative invoiced entry tokens for the investigation (reminder: at each round, all context is returned and re-invoiced ), and the final distribution by position (system / tools / history / tool results).
Scenario A — raw, no strategy. Raw tool results, no filtering, no compression. In what turn does the agent saturate? Is 150 round investigation possible?
Scenario B — PostToolUse hook. Filtered results (400 tokens instead of 2,500). New saturation trick? Winning factor?
Scenario C — hybrid (hook + summary + sliding window). In addition to filtering: every 30 rounds, rounds beyond the last 10 are compressed into a summary of 1,500 tokens (the previous summary is absorbed into the new one). Also model the compression call cost himself. Does the 150 round investigation pass? With what margin?
Economic costing. For each feasible scenario, calculate the cost of a complete investigation, then 200 investigations/day, using the official price list of the day (date and source ⚠). Bonus: add the “system prompt + cached tool definitions” variant and quantify the savings.
Architectural note (1 page max). Your recommendation: what strategy, what thresholds (compression level, verbatim window size, hook ceilings), what residual risks (what are we losing in the summaries? how does the scratchpad compensate?).

Success criteria

The simulator is parametric (no hard constant in the logic).
Scenario A demonstrates figures to support the impossibility of 150 turns.
The cost of the compression calls in scenario C is counted (forgetting it is the classic mistake).
The prices used are dated and sourced.
The architecture note explicitly arbitrates cost / capacity / risk of loss of information.

To go further

Compare your heuristic estimates (~4 characters/token in English, different ratio in French) with the official token counting endpoint on 5 real messages. What gap? What safety margin do you deduce for your simulator?

Exercise 2 — Designing a chain of provenance (75-90 min)

Context

Compliance team agrees to deploy ComplianceScan on one condition asked by internal audit: “For any report generated, we need to be able to trace each statement back to its primary source, reconstruct what the model saw, and know which version of the system produced what. » You design and prototype the complete chain of provenance.

Work requested

Output schema with attribution. Define a report JSON (JSON = JavaScript Object Notation) schema where each finding carries: the assertion, a resolvable source pointer (tool , call_id , record_id(s)), a level of confidence (établi / à vérifier / inféré), and where the envelope bears: prompt_version (hash + semantic identifier), full model identifier, temperature, timestamp, investigation correlation identifier.
Prompt generation. Write the prompt (system + injection XML structure separating <contexte_systeme>, <donnees>, <instructions>) which constrains the model to produce this diagram, with the explicit rule: no assertion without a source pointer; in the absence of a source, the assertion passes into inféré with justification.
Audit log. Implement a AuditLog (JSON files in a tree audit/<date>/<investigation_id>/) which logs: system prompt (with hash), each message, each tool call — arguments, raw result, filtered result entered in context (the args/raw/ctx triple), and the final manifest (model, version, parameters, hashes).
Citation checker. Write verify_provenance(report, audit_log) -> VerificationResult which, for each finding : (a) verifies that the call_id exists in the newspaper; (b) verifies that the record_id cited appear well in the gross result of this call; © flags any orphan quotations. Test it by deliberately corrupting a report (invent one record_id plausible — simulate the quote hallucination) and verify that the control detects it.
Blank audit scenario. In pairs: one plays the listener and chooses a statement at random from a generated report; the other must, in less than 5 minutes and only with the log, produce: the primary source, what the model actually saw (ctx vs raw), and the version of the prompt. Document what was missed or slowed down.
Substantive question (10 lines). Your management asks: “Can we guarantee the identical regeneration of a report to prove its origin? » Write the architect's honest response: what temperature 0 + pinned version + seed ⚠ guarantee, what they don't guarantee ("deterministic-ish"), and why the audit log — not regeneration — is the real guarantee of compliance.

Success criteria

The diagram distinguishes origin of data (sources) and origin of generator (prompt, model, parameters).
The checker detects the hallucinatory quote from the corruption test.
The diary allows us to answer “what exactly did the model see?” » (ctx ≠ raw).
The blank audit ends in < 5 mins.
The answer on reproducibility does not overpromise.

To go further

Add integrity chaining: Each log entry includes the hash of the previous entry (hash chain). What do we gain from an auditor? What have we still not proven?

Exercise 3 — Batch processing pipeline (60-90 min)

Context

New compliance policy v3.3: 80,000 customer files must be re-graded this week. Synchronous calls = too slow, too expensive, rate limits. You build the complete pipeline on the Message Batches API : submission, monitoring, processing, error recovery — with batch optimization + caching and supporting encryption.

(For the exercise, work on 200 synthetic files; the code must be sized for 80,000.)

Work requested

Game generation. Script make_dossiers.py : 200 synthetic JSON files (id, country, declared activity, list of transactions, various flags) — including ~10 deliberately malformed (missing field, questionable encoding) to test downstream robustness.
Submission. Script submit_batch.py :
- builds the queries with a custom_id robust (agreement to be documented: e.g. v33-{dossier_id}-{hash_court_du_contenu} — justify each component);
- places the compliance policy (long and identical system block) in hidden (cache_control ) ;
- batch division if necessary (limit of 100,000 requests/batch ⚠ — your code must manage division even if the test set fits in one batch);
- locally persists the submission manifest: batch_id , list of custom_id , timestamp, prompt version (origin!).
Follow up. Script poll_batch.py : queries the status at reasonable intervals (progressive backoff, no tight loop), logs transitions in_progress → ended .
Counting and recovery. Script collect_results.py :
- analyzes the results by attaching each result to its file via custom_id (reminder : order is not guaranteed ) ;
- road by type: succeeded → results base; errored → retry file with error details; expired / canceled → resubmission queue;
- East idempotent : restartable after crashing halfway through without double-processing (justify the mechanism — progress markers, upsert by custom_id …) ;
- produces a completeness report: submitted / passed / failed / missing, with the nominative list of those missing.
Comparative costing. With today's price list (dated, sourced ⚠), calculate for 80,000 files (shared system 3,200 tokens, variable input ~900 tokens, output ~350 tokens) the cost in 4 configurations: synchronous without cache / synchronous + cache / batch without cache / batch + cache . Present the table and the total savings factor. Check your orders of magnitude with the calculator on the session web page.
Operating note (½ page). 24 hour SLA ⚠: what are the consequences on planning (night window, margin, criticality)? What happens if 3% of requests time out? What is your “campaign completed” criteria?

Success criteria

The agreement of custom_id is documented and allows correlation + idempotence + audit.
The processing processes the 4 terminal statuses and survives the 10 malformed files.
collect_results.py restarted twice in a row does not double-process anything.
The costing shows the batch + cache accumulation, dated and sourced prices.
The completeness report designates by name any missing file.

To go further

Plug in the output of exercise 2: each batch result feeds the audit log and the provenance schema. A custom_id then becomes the thread that connects file → request → response → report → audit. This is exactly the expected architecture of a Claude Certified Architect : the three building blocks of the session become one.

Indicative scale (if assessment noted)

Exercise	Points	Of which
1 — Context budget	30	Simulator 12, scenarios 10, architecture note 8
2 — Chain of provenance	40	Diagram 8, log 10, verifier 12, mock audit 5, reproducibility 5
3 — Batch pipeline	30	Submission 8, counting/rework 12, costing 6, processing 4

Validation threshold: 60/100. The detection of the hallucinated quote (ex. 2.4) and the idempotence of the counting (ex. 3.4) are playoffs if absent: these are the two non-negotiable reflexes of the profession.