# Quiz — Advanced Level, Session 9 # “Certification scenarios” **Program:** Applied AI — Yann Isola **Format:** 15 MCQs (MCQ = Multiple Choice Questionnaire) in the exact format of the exam *Claude Certified Architect* — situation, question, 4 options, only 1 correct answer **Duration:** 22 minutes (exam pace: ~90 seconds/question) **Threshold:** 11/15 (≈ 72%, aligned with the certification threshold ⚠) **Breakdown by domain (proportional to exam weights):** D1 Architecture & orchestration ×4 · D2 Tools & MCP ×3 · D3 Claude Code ×3 · D4 Prompts & structured output ×3 · D5 Context & reliability ×2 > ⚠ The API behaviors, CLI flags, configuration paths and Batch API settings described reflect the documentation at the time of writing and are **volatile**: check the official Anthropic documentation. --- ## Domain 1 — Agent architecture & orchestration (4 questions) ### Q1 — Stopping the agentic loop (Scenario 8) **Situation:** An autonomous case processing agent loops: API call → execution of the requested tools → return of results. The developer implemented the stopping condition as follows: the loop ends when the helper text contains "task completed", with a guardrail of 10 iterations maximum as the main stopping mechanism. **What is the main problem with this design?** - A. The guardrail of 10 iterations is too low; it would take 50 iterations - B. The stopping condition should rely on `stop_reason` (`"end_turn"` vs `"tool_use"`), not on text analysis or an arbitrary iteration limit as the main mechanism - C. The text should be analyzed with a more robust regex than substring search - D. The loop should end on the first successful tool call **Answer: B.** The agentic loop contract is controlled by `stop_reason`: `"tool_use"` → execute and continue; `"end_turn"` → the model has finished. Parsing the helper text to detect completion and using an iteration limit as the **primary** stopping mechanism are the two canonical antipatterns (the limit is still useful as a secondary safeguard). A and C refine the anti-pattern instead of correcting it; D breaks the loop. --- ### Q2 — Partial coverage of a multi-agent report (Scenario 3) **Situation:** A coordinator + sub-agent system researches “the impact of AI on the healthcare sector”. The final report only covers medical imaging and diagnostic assistance. The logs show that the coordinator created three subtasks: “AI in Radiology”, “AI in Pathology Image Analysis”, “AI and Assisted Diagnosis”. Each sub-agent produced complete and correctly sourced work on its sub-task. **What is the root cause?** - A. The sub-research agents did not search widely enough in their fields - B. The summarizer did not detect coverage gaps before redaction - C. The coordinator broke down the subject too narrowly: all the subtasks relate to diagnosis, ignoring administration, clinical research, telemedicine - D. The coordinator's context window has been saturated by the results of the subagents **Answer: C.** The logs show subagents that did **correctly** what was asked of them — the problem is upstream: decomposition. “Health” has been reduced to just diagnosis.B is a desirable safety net but not the cause; Contradicted the newspapers; D is not supported by the situation. Examination reflex: partial coverage + correct subagents = suspect the **instructor**. --- ### Q3 — Passing context to subagents (Scenario 3) **Situation:** In a research pipeline, the coordinator conducted a phase 1 analysis that produced key findings. It then generates a sub-writing agent via the `Task` tool with the prompt: “Write the final report based on the conclusions of phase 1”. The subagent produces a generic report unrelated to the findings. **Why?** - A. The `Task` tool has a known context transmission bug that must be worked around with `fork_session` - B. The subagents operate with an isolated context: they do not inherit the history of the coordinator; phase 1 conclusions had to be included explicitly in the subagent prompt - C. The subagent did not have the web search tool in its `allowedTools` - D. The subagent prompt was too long and was truncated **Answer: B.** This is the most tested invariant of Domain 1: context isolation of subagents. “Based on the conclusions of phase 1” conveys nothing — the subagent has never seen these conclusions. The correction: include the complete outputs (or a structured summary) of the previous phases in the prompt, ideally with a structured format separating data and instructions. Invents a bug; C and D do not correspond to the symptom (generic report = lack of material, no tool or truncation). --- ### Q4 — Guaranteed execution order (Scenario 1) **Situation:** A banking support agent must verify the identity (`get_customer`) before any financial transaction (`process_refund`). The audit shows that in 7% of cases, the agent processes the reimbursement without prior verification, despite an all-caps instruction in the system prompt: “ALWAYS CHECK IDENTITY FIRST”. **Which correction provides a guarantee?** - A. Repeat the instruction at the beginning AND at the end of the system prompt to counter the “lost in the middle” effect - B. Add 4 few-shot examples showing the correct sequence - C. Implement a hook/programmatic precondition that blocks `process_refund` until no verified client ID is present in session state - D. Upgrade to a more capable model that follows instructions better **Answer: C.** The key word in the question is “**guarantee**”. Prompt (A, B) and pattern choice (D) improve the **probability** of compliance — they can reduce 7% to 1%, never to 0%. A critical business rule (financial operations) requires **deterministic** application: the software blocks the call, regardless of what the model “decides”. This is the structuring distinction of the entire examination: deterministic guarantees (hooks, preconditions) vs probabilistic conformity (prompts). --- ## Domain 2 — Tool design & MCP integration (3 questions) ### Q5 — Misrouting between similar tools (Scenario 1) **Situation:** An agent has `get_customer` (“Gets customer info”) and `lookup_order` (“Looks up orders”). For order questions, it calls `get_customer` 30% of the time. The team hesitates between four corrections. **Which one to apply first?** -A.Add an upstream classification layer that routes the request to the right tool - B. Merge the two tools into one tool `customer_and_orders` with a mode setting - C. Enrich each description: input formats with examples, what the tool returns, when to use it and when to use the other, borderline cases - D. Add few-shot examples of tool selection in the system prompt **Answer: C.** Tool descriptions are the **primary mechanism** for model selection; here they are minimal and almost identical — this is the root cause. Enriching it is the correction with the least effort and the greatest impact. D can help **then** (few-shots cost tokens and don't fix empty descriptions); A is over-engineering that adds a component to maintain; B degrades the design (catch-all tool with unclear contract). --- ### Q6 — Tool Error Taxonomy (Scenario 8) **Situation:** MCP tool `process_refund` fails in three ways: (1) payment gateway timeout, (2) amount exceeding the limit allowed by internal policy, (3) invalid transaction ID format. Currently all three return `{"isError": true, "message": "Operation failed"}`. The agent systematically retries the three cases in a loop. **Which error response recast is correct?** - A. (1) `transient`/`isRetryable: true`; (2) `business`/`retryable: false` + explanation of the policy to be transmitted to the user; (3) `validation`/retryable after correction, with the expected format in the message - B. Return the same message for all three but add a numeric HTTP code that the agent will learn to interpret - C. (1), (2) and (3): `isRetryable: true` with exponential backoff — the retry always ends up being successful or exhausting the budget - D. Delete `isError` and return an empty result: the agent will automatically conclude that it has failed **Answer: A.** The transient / business / validation (+permission) taxonomy with explicit `isRetryable` is what allows the agent to **decide intelligently**: retry the timeout, never retry a policy violation (it will always fail — inform or escalate), correct the input and then retry the validation. B leaves the interpretation to chance; C causes the agent to loop on non-retryable errors — exactly the symptom observed; D hides failure, worst anti-pattern: an empty "valid" result and a failure must be **distinguishable**. --- ### Q7 — Sharing an MCP server as a team (Scenario 4) **Situation:** A team of 12 developers wants the internal MCP “product catalog” server to be available to everyone from the repository clone, with authentication by individual token. A developer suggests adding the server with its token in `~/.claude.json` and sharing this file on the wiki. **What is the correct configuration?** - A. Each developer manually adds the server in `~/.claude.json` with their clear token - B. Declare the server in the project's (versioned) `.mcp.json` file, with the token referenced by environment variable (`${CATALOG_TOKEN}`) that each developer defines locally - C. Declare the server in the root CLAUDE.md with the token, since CLAUDE.md is versioned - D. Declare the server in `.mcp.json` with a clear shared team token in the file**Answer: B.** Two rules combine: **project** scope = versioned `.mcp.json` (available to clone for the entire team, unlike `~/.claude.json` which is individual and not shared); **secrets** = substitution of environment variables, never a clear token in a versioned file. A does not satisfy “available to clone” and duplicates the effort; C confuses instructions (CLAUDE.md) and server configuration; D versions a secret — eliminatory security fault. --- ## Domain 3 — Configuration & workflows of Claude Code (3 questions) ### Q8 — Conventions by file type (Scenario 2) **Situation:** A monorepo contains strict testing conventions that apply to all `**/*.test.ts` files, which are co-located with the source code in dozens of directories. Developers want Claude Code to load these conventions **only** when it edits test files, to save context. **Which configuration to choose?** - A. Put everything in the root CLAUDE.md: this is the only file always loaded - B. A CLAUDE.md in each directory containing tests - C. A `.claude/rules/testing.md` file with a YAML frontmatter `paths: ["**/*.test.ts"]` - D. A `.claude/skills/testing/` skill that developers invoke before writing tests **Answer: C.** Path-scoped rules (`.claude/rules/` + glob patterns) load **only** when editing matching files — exactly as needed (conditional + context saving). Permanently dependent on conventions, quite the opposite; B is unmaintainable when the tests are co-located in dozens of directories (and this is the switching criterion: conventions by **file type** dispersed → glob; conventions by **subtree** → CLAUDE.md of directory); D is manual and on demand — the situation calls for automatic. --- ### Q9 — Claude Code blocked in CI (Scenario 5) **Situation:** A GitLab CI job executes `claude "Génère les tests manquants pour ce module"` and remains suspended until the runner timeout. The same prompt works perfectly on the developer's workstation. **What is the documented fix?** - A. Define the environment variable `CLAUDE_HEADLESS=true` in the job - B. Use `claude -p "Génère les tests manquants pour ce module"` (non-interactive mode: processes the prompt, writes to stdout, exits) - C. Use flag `--batch` for non-terminal execution - D. Redirect standard input: `claude "..." < /dev/null` **Answer: B.** `-p` (or `--print`) is the documented headless mode of Claude Code ⚠: non-interactive, suitable for pipelines, combinable with `--output-format json` and `--json-schema` for structured output usable by the job. A and C are **invented features** — the most common distractor family in this scenario; D is a Unix workaround that does not match the documented execution mode. --- ### Q10 — Architectural restructuring (Scenario 2) **Situation:** You entrust Claude Code with the extraction of a billing service from a monolith: ~45 files affected, several possible divisions, choices of service boundaries to arbitrate. A colleague suggests: “launch it in direct execution with very detailed instructions, you will switch to plan mode if it goes badly”. **Which approach to recommend?** -A.Follow the suggestion: direct execution with detailed instructions is faster, plan mode as a backup - B. Planning mode from the outset: exploration of the code base and dependencies, design of the approach and service boundaries, validation of the plan, then execution - C. Direct incremental execution, one file at a time, to limit damage - D. Divide the work yourself into 45 independent prompts, one per file **Answer: B.** The three markers of plan mode are combined: significant change (45 files), **several viable approaches**, **architectural decisions** (service boundaries). Plan mode allows safe exploration before any modification. A is the classic “reactive” distractor: when “things go wrong”, recovery is already costly; C avoids local disasters but does not solve the central problem (service boundaries are a **global** decision); D destroys the inter-file consistency of the extraction. --- ## Domain 4 — Prompt engineering & structured exit (3 questions) ### Q11 — Strict schema, persistent errors (Scenario 6) **Situation:** An invoice extraction pipeline uses `tool_use` with strict JSON schema and forced `tool_choice`. The JSON produced is always syntactically valid and conforms to the schema. However, 4% of extractions contain errors: amount excluding tax placed in the including tax field, sum of lines different from the total extracted. **Which analysis is correct?** - A. The schema is poorly written: a correctly constrained JSON schema (types, formats, bounds) also eliminates these errors - B. The schema guarantees syntax and form, not semantics; add a programmatic reconciliation validation (Σ lines = total, excluding tax + VAT = including tax) and a retest loop with precise validation errors as feedback - C. Replace `tool_use` with an instruction “answer in valid JSON” with an example, more flexible for borderline cases - D. Increase the temperature so that the model explores other readings of the document **Answer: B.** This is THE distinction of Domain 4: strict JSON schemas eliminate **syntax** errors, not **semantic** errors (bad field, unreconciled totals). The solution is programmatic: consistency validations + retest with concrete error feedback (document + faulty extraction + specific errors). A is false — no type constraint prevents putting the right number in the wrong field; C regresses (reintroduces the syntax errors that `tool_use` eliminated); D increases variability, making the problem worse. --- ### Q12 — Missing fields and hallucination (Scenario 6) **Situation:** The contract extraction schema imposes `date_resiliation` as a **required** string type field. On indefinite-term contracts — which do not have a termination date — the model produces plausible but invented dates. The team adds to the prompt: “never invent a date”, without any notable improvement. **What basic correction?** - A. Make the field nullable/optional so that the absence of information has a legitimate representation in the schema, and document in the field description when to return `null` - B. Reinforce the instruction: “INTERDICTION ABSOLUTE to invent dates” at the top of the prompt - C. Add a retry step: if the extracted date seems suspicious, request the extraction again -D.Post-process: remove dates whose format is improbable **Answer: A.** A **required** field on sometimes absent information **structurally** forces the model to produce something — the contrary instruction creates a contradictory injunction that the schema wins. The fix is ​​in the schema design: nullable/optional gives "information does not exist" a legitimate output. Same family: enums with `"other"`/`"unclear"` + detail field. B has already shown its inefficiency (the diagram takes precedence); C tries again without changing the constraint that causes the invention; D doesn't detect **plausible** made-up dates — that's precisely the problem. --- ### Q13 — False positives in automated code review (Scenario 5) **Situation:** An automated PR review flags too many false positives, focused on style and naming remarks, which causes developers to also ignore security alerts — even though they are reliable. The current prompt asks: “do a thorough review and be thorough.” **Which prompt redesign is the most effective?** - A. Add “be more conservative in your reports and avoid false positives” - B. Define explicit categorical criteria — report: bugs, vulnerabilities, logic errors; ignore: style, naming, preferences — with one example per category and illustrated severity levels; temporarily disable the loudest categories - C. Run the review three times and only keep issues reported at least twice - D. Lower the temperature to 0 for more deterministic reviews **Answer: B.** Two exam insights combine: (1) **explicit, categorical criteria** override generic guidance — “be more conservative” (A) is precisely the documented anti-example; (2) false positives from a noisy category **undermine confidence in reliable categories** — hence temporarily disabling categories with high false positive rates. C (consensus by repetition) treats the noise without treating the cause and triples the cost; D reduces the variance between runs, not the over-reporting bias. --- ## Domain 5 — Context management & reliability (2 questions) ### Q14 — Progressive summary and transactional facts (Scenario 7) **Situation:** A claims assistant conducts 100+ round conversations. To contain the context, ancient tricks are gradually summarized. Customers complain: in the 60th round, the assistant gets the deductible amount announced in the 5th round and the file number wrong – successive summaries have transformed them into “the conditions have been discussed”. **What architectural correction?** - A. Summarize less aggressively: keep 30 verbatim turns instead of 10 - B. Extract the transactional facts (amounts, numbers, dates, commitments) in a persistent “case facts” block, never summarized, injected at each round — the summary only applying to the rest - C. Replace the summary with a strict sliding window of the last 20 rounds - D. Ask the model to “preserve important numbers” in its summaries **Answer: B.** The documented risk of progressive summarization is exactly this: the **numerical values, dates and identifiers** become diluted into vague formulations over successive summaries. The structural parade: a block of facts persisting **outside** the summary cycle.A and D only delay or attenuate the dilution (D remains probabilistic, and each summary recompresses); It's worse: the sliding window **deletes** purely the 5th round, franchise included. --- ### Q15 — Automate based on a global metric (Scenario 6) **Situation:** An extraction pipeline shows 97% overall accuracy on the validation set. Management wants to eliminate the human magazine. An audit by segments reveals: 99% on native invoices (85% of the volume), but 71% on degraded scans and 64% on the IBAN field (IBAN = International Bank Account Number) of delivery notes — field used for transfers. **What architectural decision?** - A. Automate: 97% exceeds the 95% threshold set by management - B. Maintain human review on 100% of documents until each segment reaches 97% - C. Automate by segment: native invoices go automatic with continuous stratified sampling; degraded scans and low critical fields (IBAN) remain under human review with field-level trust routing - D. Retrain the prompt on more degraded scans, then automate globally as soon as overall accuracy reaches 98% **Answer: C.** The aggregate metric **masks** the weak segments — 97% overall here is compatible with 64% in a critical field for transfers. The architect's response is segmented: automate where performance is demonstrated **by document type and by field**, keep the human touch where it is not, with continuous stratified random sampling to detect deviations (new supplier formats). Exactly reproduced the trap; B over-corrects and destroys the value on 85% of the volume without justification; D perhaps improves a segment but renews the faulty criterion (global metric). --- ## Scale and quick answer | Q | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | |---|---|---|---|---|---|---|---|---|---|----|----|----|----|----|----| | Answer | B | C | B | C | C | A | B | C | B | B | B | A | B | B | C | | Domain | D1 | D1 | D1 | D1 | D2 | D2 | D2 | D3 | D3 | D3 | D4 | D4 | D4 | D5 | D5 | **Recommended analysis after correction:** for each error, identify the **distractor family** that trapped you — magic prompt (Q4, Q12), over-engineering (Q5, Q13-C), invented feature (Q9), symptom instead of cause (Q2, Q15). Your dominant bias = your priority revision axis.