Exercises — Advanced Level, Session 9
“Certification scenarios”
Program : Applied AI — Yann Isola Audience : Solution architects — preparation Claude Certified Architect Estimated total duration: 3 h to 4 h (individual or pair work; exercise 2 benefits a lot from being done in adversarial pair) Prerequisites: sessions 1 to 8 of the advanced level. No API key is essential (design exercises on paper/Markdown), but an Anthropic key allows you to validate the points marked (optional: validate by code).
⚠ All exam parameters cited (60 questions, 90 minutes, threshold 72%, domain weightings) are volatile : check the official certification page before the exam. Model names in code examples are also volatile.
Exercise 1 — Design the complete architecture for Scenario 1: Customer Support Agent (75-90 min)
Context
You are the architect of a customer support agent for an e-retailer (500,000 orders/year). The agent, built on the Claude Agent SDK (SDK = Software Development Kit), must manage: product returns, billing disputes, account problems. It has MCP tools (MCP = Model Context Protocol): get_customer , lookup_order , process_refund , escalate_to_human .
Contractual objectives:
- First Contact Resolution > 80 %
- Zero reimbursement without verified identity (requirement of the fraud team — non-negotiable )
- Refunds > €200: human validation required (internal policy)
- Explicit request from a human: transfer < 30 seconds
It is Exactly the scope of Scenario 1 of the exam. Your production will also serve as a revision sheet.
Work requested
Produce an architectural document (architecture-s1.md ) containing the following 6 sections.
1. Flowchart (in text or Mermaid). The journey of a request, from receipt to resolution or escalation. Make it appear explicitly: the identity verification point, the escalation decision point, the agentic loop (stop_reason: "tool_use" → execution → return results).
2. Tool contracts. For each of the 4 tools, write the full description as it will be provided to the model: role, input format with examples, what the tool returns, when to use it and when not to use it (explicit differentiation get_customer vs. lookup_order ), borderline cases. Constraint: a proofreader who ONLY reads the 4 descriptions must be able to predict which tool the agent will choose for 5 typical requests that you provide.
3. Deterministic guarantees vs. probabilistic guidance. Classify each of the 4 contractual requirements in one of the two columns, and justify:
| Requirement | Mechanism (programmatic hook/precondition OR prompt/few-shot) | Justification |
|---|---|---|
| Zero reimbursement without verified identity | ? | ? |
| Human validation > 200 € | ? | ? |
| First Contact Resolution > 80 % | ? | ? |
| Transfer < 30 s on explicit request | ? | ? |
Then write it hook pseudo-code PreToolUse which blocks process_refund if (a) none customer_id verified does not exist in the session, or (b) the amount exceeds €200 — in this case redirecting to escalate_to_human accompanied by a structured pattern. (optional: validate by code — implement the actual hook in the SDK.)
4. Escalation Policy. Write the “escalation” section of the system prompt: explicit criteria (at least 4 triggers), 3 few-shot examples including 1 case where you should NOT climb despite a dissatisfied customer, and the transfer summary format (handoff): Client ID, reason, actions attempted, recommended action. Prohibition on using sentiment analysis or self-assessed confidence as an escalation criterion — explain why in 3 lines (this is an exam question).
5. Tool error management. Define the error response format of your MCP tools: errorCategory (transient / validation / business / permission), isRetryable , readable message. Give a complete JSON example for each category, including the trap case: lookup_order who does not find no order (valid empty result — how to distinguish it from an access failure?).
6. Self-attack. Write 3 certification questions (situation + 4 options + justified answer) that trap YOUR architecture. At least one must use a distractor from the “magic prompt” family and one from the “over-engineered” family.
Success criteria
- The diagram shows the identity precondition Before any refund, out of the “prompt” path
- The 4 tool descriptions allow routing prediction on the 5 typical requests
- The deterministic/probabilistic table is correctly filled (fraud requirements and €200 → hooks; resolution rate and tone → prompt)
- The hook pseudo-code handles both conditions and produces a structured pattern
- The escalation policy contains the counterexample “disgruntled ≠ escalation”
- The 4 error categories are illustrated, empty result ≠ access failure
- The 3 self-attack questions respect the exam format
Trap to avoid
Put the rule “zero reimbursement without identity” in the prompt system “because we also have the hook, belt and suspenders”. This is acceptable in practice — but on examination, if the question asks "what mechanism guarantees the rule”, the only valid answer is the programmatic mechanism. The prompt does not guarantee anything: it makes it probable.
Exercise 2 — Compare three approaches for Scenario 3: multi-agent search (60-75 min, pair recommended)
Context
An analysis firm must produce monitoring reports on broad issues (“impact of the MiCA regulation on European stablecoin issuers”, “adoption of generative AI in the banking sector”). Each report requires: complete coverage of the subject, quotes for each statement , management of contradictory sources, delay < 2 a.m.
Three architectures are on the table. Your job: compare them rigorously , like a multiple exam question would do.
Approach A — Single agent equipped. A single agent with web research, document reading and writing. No subagents.
Approach B — Fixed pipeline (prompt chaining). Four hard-coded sequential steps: (1) exhaustive web search → (2) analysis of collected documents → (3) synthesis → (4) formatting of the report. Each step is an independent call receiving the output of the previous one.
Approach C — Coordinator + dynamic subagents (hub-and-spoke). A coordinator decomposes the subject, generates parallel search sub-agents (via Task ), evaluates the coverage, raises the sub-agents on the gaps, then delegates the synthesis.
Work requested
1. Comparison grid. Complete (with justification of one sentence per cell):
| Criteria | A (single agent) | B (fixed pipeline) | C (dynamic coordinator) |
|---|---|---|---|
| Broad topic coverage | |||
| Risk of context saturation | |||
| Latency (parallelism possible?) | |||
| Preserving provenance (claim → source) | |||
| Recovering from component failure | |||
| Token cost | |||
| Observability/debugging | |||
| Implementation complexity |
2. The three breakdowns, three diagnoses. For each incident below, indicate which approach(es) are most vulnerable to it, the root cause, and the fix — in exam response format (2-4 sentences):
- Incident 1: The “AI in Banking” report only covers customer service chatbots, ignoring fraud detection, credit scoring and compliance.
- Incident 2: At the 60th tool call, the agent begins referring to “typical industry practices” instead of the documents actually collected, and two citations point to documents never opened.
- Incident 3: a web search timeout in step 1 causes the entire report to fail, even though 80% of the sources were already collected.
3. The question of consensus. In approach C, two sub-agents return contradictory figures (AI adoption: 40% according to a 2025 government report, 12% according to a 2023 sector study). Write it reconciliation rule that you give to the coordinator (5-8 prompt lines). It must cover: conservation of the two values, attribution, taking into account dates, and what appears in the final report. Then explain in 3 lines why “having 3 sub-agents vote and take the majority” is a false consensus for documentary facts (hint: consensus applies to judgments, not to sourced facts — 3 agents can read the same bad source).
4. Argued verdict. Recommend an approach (or a hybrid) for this firm, in 10 lines max, in the format: recommendation → 3 arguments → 1 residual risk assumed → confidence (🟢/🟡/🔴). In pairs: one defends B, the other C, 10 minutes of adversarial proceedings before writing the joint verdict.
Success criteria
- The grid is filled with justifications, not impressions (“C parallelizes via multiple
Taskin one lap” and not “It’s faster”) - Incident 1 → decomposition too narrow © or search prompt too vague (A/B) — the diagnosis points to the contractor , not the performers
- Incident 2 → context degradation/saturation (especially A); correction: isolated context subagents + scratchpad + obligatory structured quotes
- Incident 3 → error propagation (especially B, pipeline without recovery); correction: structured error context + partial results
- Reconciliation rule keeps both values with attribution and dates
- The verdict distinguishes between sourced facts (provenance) and judgments (possible consensus)
Trap to avoid
Conclude “It’s always better”. Fixed pipeline B is the Good review response when the steps are predictable and identical from one execution to another (standardized multi-aspect review). It is justified when the decomposition depends on what we discover . An exam question may well make B the correct answer.
Exercise 3 — Evaluation grid for Scenario 6: extraction of structured data (60 min)
Context
Your company deploys an extraction pipeline on 3 types of documents: supplier invoices (native PDFs and scans), service contracts (10-40 pages) and delivery notes (semi-structured, heterogeneous formats). Volume: 12,000 documents/month. Management asks: “When can we automate without human review?” »
Your job as an architect: build the evaluation grid which answers this question — that is, defining what we measure, how, with what thresholds, and what triggers a human review. This is the heart of Domain 5 applied to Scenario 6.
Work requested
1. Annotated extraction diagram. Define the JSON schema for extracting invoices (8-12 fields: supplier, number, dates, lines, totals excluding VAT/VAT/VAT, currency, IBAN — IBAN = International Bank Account Number). For each field, annotate:
- required / optional / nullable (and why - reminder: a required field on information that is sometimes absent forces hallucination);
- validation type syntactic (regex IBAN, date format) vs semantics (Σ lines = total excluding tax; excluding tax + VAT = including tax);
- business criticality (an IBAN error ≠ a wording error).
Include self-verification fields calculated_total / stated_total and explain in 2 lines their role.
2. Evaluation game. Specify the composition of a test set of 150 labeled documents. Constraints:
- stratification by document type (3 types) and by difficulty (clean native / degraded scan / edge case);
- at least 15 named borderline cases : multi-currency invoice, credit note (negative amount), multiple VAT, invoice without IBAN, illegible document, invoice in English, duplicate number, voluntary inconsistent total, “on the back” field missing from the scan, etc. — complete the list;
- for each limiting case: the expected behavior (extract, extract with
null, markunclear, reject for human review).
3. Metrics and thresholds. Define the decision grid:
| Metric | Precise definition | Proposed automation threshold | Justification |
|---|---|---|---|
| Precision per field (by doc type) | |||
| Reminder of critical fields (IBAN, TTC) | |||
Rate of null correct vs hallucinations |
|||
| Semantic reconciliation rate (totals) | |||
| Human review routing rate |
Rule imposed: no single global threshold . Explain in 5 lines why “97% overall accuracy” is a tricky criterion (masking minority document types and weak fields — this is THE exam question for Domain 5).
4. Trust and routing policy. Design the mechanism:
- confidence scores at field level (how to obtain them: self-assessment calibrated on the labeled set — and why uncalibrated raw self-assessment is not enough);
- 3 routing rules for human review (e.g.: critical field trust < threshold ; semantic reconciliation failure; type of document never seen);
- the retry loop with feedback: when it applies (correctable validation error) and when it is useless (information missing from the source) — give an example of each;
- L'continuous stratified sampling post-automation: what to sample, how often, to detect what (derived from supplier formats).
5. Return to management. Write the answer to “when do we automate?” » in 10 lines: criteria for entry into automation by segment (type of doc × field criticality), residual share of human review assumed, and monitoring system. Decision format, not course format.
Success criteria
- The diagram distinguishes nullable/required with anti-hallucination justification
- Syntactic and semantic validation clearly separated (the schema guarantees the syntax, never the semantics)
- The test set is stratified and the 15 borderline cases each have an expected behavior
- No single global threshold; metrics by type × field
- Routing policy covers low trust, semantic failure and novelty
- The “unnecessary retry” case is correctly identified (information absent ≠ extraction failed)
- Restitution is a segmented decision, not a global yes/no
Trap to avoid
Propose “we try 3 times then we escalate” as a universal policy. Retry with error feedback corrects extractions poorly done ; it will never reveal any information missing from the document . A typical exam question sets this trap for you with a statement like “the due date is only in the general conditions, separate document”.
Delivery
Drop architecture-s1.md , comparaison-s3.md And evaluation-s6.md in the session's shared folder. Cross-correction at the opening of session 10: each pair rereads the production of another pair with the criteria grid above, in certification review format (what is reported, what is ignored, seriousness).