# Exercises — Advanced Level, Session 9
# “Certification scenarios”

**Program:** Applied AI — Yann Isola
**Audience:** Solutions Architects — preparation *Claude Certified Architect*
**Estimated total duration:** 3 hours to 4 hours (individual work or in pairs; exercise 2 benefits a lot from being done in contradictory pairs)
**Prerequisites:** sessions 1 to 8 of the advanced level. No API key is essential (design exercises on paper/Markdown), but an Anthropic key allows you to validate the marked points *(optional: validate by code)*.

> ⚠ All exam parameters cited (60 questions, 90 minutes, threshold 72%, domain weightings) are **volatile**: check the official certification page before the exam. Model names in code examples are also volatile.

---

## Exercise 1 — Design the complete architecture of Scenario 1: customer support agent (75-90 min)

### Context

You are the architect of a customer support agent for an e-retailer (500,000 orders/year). The agent, built on the **Claude Agent SDK** (SDK = Software Development Kit), must manage: product returns, billing disputes, account problems. It has the MCP tools (MCP = Model Context Protocol): `get_customer`, `lookup_order`, `process_refund`, `escalate_to_human`.

**Contractual objectives:**
- First contact resolution > 80%
- Zero reimbursement without verified identity (requirement of the fraud team — **non-negotiable**)
- Reimbursements > €200: human validation required (internal policy)
- Explicit request from a human: transfer < 30 seconds

This is **exactly** the scope of Scenario 1 of the exam. Your production will also serve as a revision sheet.

### Work requested

Produce an architecture document (`architecture-s1.md`) containing the following 6 sections.

**1. Flowchart (in text or Mermaid).** The journey of a request from receipt to resolution or escalation. Explicitly display: the identity verification point, the escalation decision point, the agentic loop (`stop_reason: "tool_use"` → execution → return of results).

**2. Tool contracts.** For each of the 4 tools, write the **full description** as it will be provided to the model: role, input format with examples, what the tool returns, when to use it **and when not to use it** (explicit differentiation `get_customer` vs `lookup_order`), edge cases. Constraint: a proofreader who ONLY reads the 4 descriptions must be able to predict which tool the agent will choose for 5 typical requests that you provide.

**3. Deterministic guarantees vs probabilistic guidance.** Classify each of the 4 contractual requirements in one of the two columns, and justify:

| Requirement | Mechanism (programmatic hook/precondition OR prompt/few-shot) | Rationale |
|---|---|---|
| Zero reimbursement without verified identity | ? | ? |
| Human validation > €200 | ? | ? |
| First contact resolution > 80% | ? | ? |
| Transfer < 30 s on explicit request | ? | ? |

Then write the **pseudo hook code** `PreToolUse` which blocks `process_refund` if (a) no verified `customer_id` exists in the session, or (b) the amount exceeds €200 — in this case redirecting to `escalate_to_human` accompanied by a structured pattern. *(optional: validate by code — implement the actual hook in the SDK.)*

**4.Escalation policy.** Write the “escalation” section of the system prompt: explicit criteria (at least 4 triggers), 3 few-shot examples including **1 case where you should NOT escalate** despite a dissatisfied customer, and the **transfer summary format** (handoff): customer ID, reason, actions attempted, recommended action. Prohibition on using sentiment analysis or self-assessed confidence as escalation criteria — explain why in 3 lines (this is an exam question).

**5. Tool error management.** Define the error response format of your MCP tools: `errorCategory` (transient / validation / business / permission), `isRetryable`, readable message. Give a complete JSON example for each category, including the trap case: `lookup_order` which finds **no commands** (valid empty result — how to distinguish this from an access failure?).

**6. Self-attack.** Write 3 certification questions (situation + 4 options + justified answer) that trap YOUR architecture. At least one must use a distractor from the “magic prompt” family and one from the “over-engineered” family.

### Success criteria

- [ ] The diagram shows the identity precondition **before** any reimbursement, outside the “prompt” path
- [ ] The 4 tool descriptions allow routing prediction on the 5 typical requests
- [ ] The deterministic/probabilistic table is correctly filled (fraud requirements and €200 → hooks; resolution rate and tone → prompt)
- [ ] The hook pseudo-code handles both conditions and produces a structured pattern
- [ ] The escalation policy contains the counterexample “disgruntled ≠ escalation”
- [ ] The 4 error categories are illustrated, empty result ≠ access failure
- [ ] The 3 self-attack questions respect the exam format

### Trap to avoid

Put the rule “zero reimbursement without identity” in the prompt system “because we also have the hook, belt and suspenders”. This is acceptable in practice — but on examination, if the question asks "what mechanism **guarantees** the rule", the only valid answer is the programmatic mechanism. The prompt does not guarantee anything: it makes it probable.

---

## Exercise 2 — Compare three approaches for Scenario 3: multi-agent search (60-75 min, pair recommended)

### Context

An analysis firm must produce monitoring reports on broad issues (“impact of the MiCA regulation on European stablecoin issuers”, “adoption of generative AI in the banking sector”). Each report requires: complete coverage of the subject, **citations for each statement**, management of contradictory sources, deadline < 2 hours.

Three architectures are on the table. Your job: compare them **rigorously**, like a multiple exam question.

**Approach A — Single agent equipped.** A single agent with web research, document reading and writing. No subagents.

**Approach B — Fixed pipeline (prompt chaining).** Four hard-coded sequential steps: (1) exhaustive web search → (2) analysis of collected documents → (3) synthesis → (4) formatting of the report. Each step is an independent call receiving the output of the previous one.

**Approach C — Coordinator + dynamic subagents (hub-and-spoke).** A coordinator decomposes the subject, generates parallel search subagents (via `Task`), evaluates the coverage, relaunches subagents on the gaps, then delegates the synthesis.### Work requested

**1. Comparison grid.** Complete (with justification of one sentence per cell):

| Criterion | A (single agent) | B (fixed pipeline) | C (dynamic coordinator) |
|---|---|---|---|
| Broad topic coverage | | | |
| Risk of context saturation | | | |
| Latency (parallelism possible?) | | | |
| Preserving provenance (claim → source) | | | |
| Component Failure Recovery | | | |
| Token cost | | | |
| Observability/debugging | | | |
| Implementation complexity | | | |

**2. Three failures, three diagnoses.** For each incident below, indicate which approach(es) are most vulnerable to it, the root cause, and the fix — in exam response format (2-4 sentences):

- **Incident 1:** “AI in Banking” report only covers customer service chatbots, ignoring fraud detection, credit scoring and compliance.
- **Incident 2:** at the 60th tool call, the agent starts referring to "typical industry practices" instead of the documents actually collected, and two quotes point to documents never opened.
- **Incident 3:** a web search timeout in step 1 causes the entire report to fail, even though 80% of the sources were already collected.

**3. The question of consensus.** In approach C, two sub-agents return contradictory figures (AI adoption: 40% according to a 2025 government report, 12% according to a 2023 sector study). Write the **reconciliation rule** that you give to the coordinator (5-8 prompt lines). It must cover: conservation of the two values, attribution, taking into account dates, and what appears in the final report. Then explain in 3 lines why “having 3 sub-agents vote and take the majority” is a false consensus for **documentary facts** (hint: consensus applies to judgments, not to sourced facts — 3 agents can read the same bad source).

**4. Argued verdict.** Recommend an approach (or a hybrid) for this firm, in 10 lines max, in the format: recommendation → 3 arguments → 1 residual risk assumed → confidence (🟢/🟡/🔴). In pairs: one defends B, the other C, 10 minutes of adversarial proceedings before writing the joint verdict.

### Success criteria

- [ ] The grid is filled with justifications, not impressions (“C parallelizes via several `Task` in a turn” and not “C is faster”)
- [ ] Incident 1 → decomposition too narrow (C) or search prompt too vague (A/B) — the diagnosis points to the **ordering party**, not the performers
- [ ] Incident 2 → context degradation/saturation (especially A); correction: isolated context subagents + scratchpad + obligatory structured citations
- [ ] Incident 3 → error propagation (especially B, pipeline without recovery); correction: structured error context + partial results
- [ ] Reconciliation rule keeps both values with attribution and dates
- [ ] The verdict distinguishes between sourced facts (provenance) and judgments (possible consensus)

### Trap to avoid

Conclude “It’s always better”. Fixed Pipeline B is the **correct** review answer when the steps are predictable and identical across runs (standardized multi-aspect review). It is justified when the decomposition **depends on what we discover**.An exam question may well make B the correct answer.

---

## Exercise 3 — Evaluation grid for Scenario 6: extraction of structured data (60 min)

### Context

Your company deploys an extraction pipeline on 3 types of documents: **supplier invoices** (native PDF and scans), **service contracts** (10-40 pages) and **delivery notes** (semi-structured, heterogeneous formats). Volume: 12,000 documents/month. Management asks: “When can we automate without human review?” »

Your job as an architect: build the **evaluation grid** that answers this question — that is, define what we measure, how, with what thresholds, and what triggers a human review. This is the heart of Domain 5 applied to Scenario 6.

### Work requested

**1. Annotated extraction schema.** Define the JSON schema for extracting **invoices** (8-12 fields: supplier, number, dates, lines, totals excluding VAT/VAT, currency, IBAN — IBAN = International Bank Account Number). For each field, annotate:

- required / optional / **nullable** (and why - reminder: a required field on information that is sometimes absent forces hallucination);
- validation type **syntactic** (regex IBAN, date format) vs **semantic** (Σ lines = total excluding tax; excluding tax + VAT = including tax);
- business criticality (an IBAN error ≠ a wording error).

Include the self-verification fields `calculated_total` / `stated_total` and explain in 2 lines their role.

**2. Evaluation set.** Specify the composition of a test set of 150 labeled documents. Constraints:

- stratification by **document type** (3 types) and by **difficulty** (clean native / degraded scan / borderline case);
- at least 15 **named borderline cases**: multi-currency invoice, credit note (negative amount), multiple VAT, invoice without IBAN, illegible document, invoice in English, duplicate number, voluntary inconsistent total, “on the back” field missing from the scan, etc. — complete the list;
- for each borderline case: the **expected behavior** (extract, extract with `null`, mark `unclear`, reject for human review).

**3. Metrics and thresholds.** Define the decision grid:

| Metric | Precise definition | Proposed automation threshold | Rationale |
|---|---|---|---|
| Precision per field (by doc type) | | | |
| Reminder of critical fields (IBAN, TTC) | | | |
| `null` rates correct vs hallucinations | | | |
| Semantic reconciliation rates (totals) | | | |
| Routing rate in human review | | | |

Rule imposed: **no single global threshold**. Explain in 5 lines why “97% overall accuracy” is a tricky criterion (masking minority document types and weak fields — this is THE exam question for Domain 5).

**4. Trust and routing policy.** Design the mechanism:

- confidence scores **at field level** (how to obtain them: self-assessment calibrated on the labeled set — and why raw uncalibrated self-assessment is not enough);
- 3 routing rules with human review (ex.: critical field confidence < threshold; semantic reconciliation failure; type of document never seen);
- the retry loop with feedback: when it applies (correctable validation error) and when it is **useless** (information absent from the source) — give an example of each;
- **continuous stratified sampling** post-automation: what to sample, at what frequency, to detect what (derived from supplier formats).

**5. Report to management.** Write the answer to “when do we automate?” » in 10 lines: criteria for entry into automation by segment (type of document × field criticality), residual share in human review assumed, and monitoring system. Decision format, not course format.

### Success criteria

- [ ] The diagram distinguishes nullable/required with anti-hallucination justification
- [ ] Syntactic and semantic validation clearly separated (the schema guarantees the syntax, never the semantics)
- [ ] The test set is stratified and the 15 borderline cases each have an expected behavior
- [ ] No single global threshold; metrics by type × field
- [ ] Routing policy covers low trust, semantic failure and novelty
- [ ] The “unnecessary retry” case is correctly identified (information absent ≠ extraction failed)
- [ ] Restitution is a segmented decision, not a global yes/no

### Trap to avoid

Propose “we try 3 times then we escalate” as a universal policy. Retrying with error feedback corrects **poorly done** extractions; it will never reveal information **absent from the document**. A typical exam question sets this trap for you with a statement like “the due date is only in the general conditions, separate document”.

---

## Delivery

Drop `architecture-s1.md`, `comparaison-s3.md`, and `evaluation-s6.md` into the session's shared folder. Cross-correction at the opening of session 10: each pair rereads the production of another pair with the criteria grid above, in certification review format (what is reported, what is ignored, seriousness).