title: "Certification scenarios"
subtitle: "Applied AI — Advanced Level · Session 9"
author: "Yann Isola"
theme: "ink #1A2230 / teal #0F7A6C / copper #B4612A / light-teal #E9F6F3 / bg #F4F7F6"

Slide 1 — Title

Certification scenarios

Applied AI — Advanced Level · Session 9
Yann Isola · Preparation Claude Certified Architect

Slide 2 — Session Objectives

• Master the exam format and time strategy
• Map the 5 domains and their weightings
• Analyze the 8 scenarios: expected decisions, recurring pitfalls
• Apply the reading grid of the MCQs (MCQ = Multiple Choice Questionnaire)
• Self-assess and prioritize review by domain weight
• Train in timed conditions

Slide 3 — The format, black on white

Parameter	Value
Questions	60
Duration	90 min (~90 sec/question)
Type	MCQ 1/4 + questions based on scenarios
Threshold	720 / scale 100–1000 (≈ 72%)
Penalty for error	None
Scenarios	4 drawn from 8

Slide 4 — Three practical consequences

1. No penalty → never an empty question (25% expectation even at random)
2. 4 scenarios out of 8, random draw → no dead end possible
3. Threshold ≈ 72% → entitled to ~16 errors out of 60

67% of the exam = Domains 1 + 3 + 4. Solid on this = comfortable margin.

Slide 5 — The 5 areas and their weight

Domain	Weight	≈ Q/60
1. Agent architecture & orchestration	27%	~16
2. Tool design & MCP integration	18%	~11
3. Configuration & workflows Claude Code	20%	~12
4. Prompt engineering & structured output	20%	~12
5. Context management & reliability	15%	~9

Slide 6 — Anatomy of an exam question

1. Situation — system in production, symptom quantified (“in 12% of cases…”)
2. Question — “which change is most effective?” / “what is the first step?”
3. 4 options — 1 correct, 3 distractors built according to recipes

The words “most”, “first”, “at least effort” ⇒ several technically valid options — we decide based on cost/impact ratio.

Slide 7 — The 4 families of distractors

Family	Signature
Magic prompt	Solve by prompt which requires a deterministic guarantee
Over-engineered	ML classifier/routing layer for simple problem
Invented feature	CLAUDE_HEADLESS=true, --batch, .claude/config.json
Symptom, not the cause	Plausible fix… to another problem

Slide 8 — The reading grid in 3 questions

1. What is the root cause? (reread the figures of the situation)
2. Deterministic guarantee required, or probabilistic guarantee acceptable?
   → deterministic: hooks / preconditions · probabilistic: prompt / few-shot
3. Among the options that address the cause: best effort/impact ratio?

Slide 9 — Correspondence scenarios × domains

Slide 10 — Scenario 1: Customer Support Agent

SDK Agent + MCP Tools: get_customer, lookup_order, process_refund, escalate_to_human
Objective: first contact resolution > 80%, appropriate escalation

Expected decisions:

• Identity before reimbursement → programmatic precondition
• Bad tool routing → enrich descriptions first
• Poorly calibrated escalation → explicit criteria + few-shot

Slide 11 — Scenario 1: pitfalls

• Feeling ≠ complexity — feeling modulates the tone, never the escalation decision
• Self-assessed confidence ≠ reliable — the model is wrong with confidence
• Multiple customer matches → ask for credentials, never guess
• Explicit human request → immediate escalation, without investigation
• Handoff = structured summary (ID, reason, attempted, recommended), not the raw conversation

Slide 12 — Scenario 8: agentic AI tools

Selection · chaining · error recovery (recombination of D1 + D2)

• Too many tools (18 vs 4-5) reduces selection reliability → scope by role
• Overlapping descriptions → rewrite/rename (analyze_content → extract_web_results)
• tool_choice: "auto" (text possible) · "any" (mandatory tool) · forced ({"type":"tool","name":…})

Slide 13 — Taxonomy of tool errors

Category	Example	Retryable?
Transient	timeout
Validation	wrong input format	after correction
Profession	policy ceiling exceeded	+ explanation
Permission	access denied	→ climbing

Golden trap: access failure ≠ valid empty result. Masking a failure as a success = certain anti-pattern.

Slide 14 — Scenario 3: multi-agent search

Hub-and-spoke (star architecture) — the invariants:

1. The coordinator owns all the communication (observability)
2. Subagents = isolated context — nothing is inherited, everything happens in the prompt
3. Parallelism = several Task in one turn of the coordinator
4. Prompts from the coordinator in objectives + criteria, step by step

Slide 15 — Scenario 3: three classic failures

Slide 16 — Consensus and contradictions

Two credible sources: 40% (government report) vs 12% (sector study)

Choose heuristically · Block all · Transmit without marking
Keep both + annotate the conflict + attribution + dates → the coordinator reconciles

Provenance: require subagents to provide “claim → source” matches (URL, document, citation, date) — otherwise attribution dies at the abstract.

Slide 17 — Scenario 7: Conversational AI

Reminder #1 (most tested): the API is stateless.
No server-side session — history is your application responsibility.

Memory strategy	Risk
Complete history	saturation + cost
Sliding window	loss of old commitments
Progressive summary	dilution of figures, dates, amounts
Hybrid + “case facts”	complexity assumed

Slide 18 — The “case facts” pattern

Reflex response to Scenario 7:

• Transactional facts (amounts, numbers, dates, commitments) → persistent block never summarized, injected at each turn
• The progressive summary only applies to the rest
• “Lost in the middle” effect: reinject critical constraints near the end of the context
• Structure in XML (XML = eXtensible Markup Language): system / data / instructions

Slide 19 — Scenario 2: configuration Claude Code

Mechanism	Location	Shared (VCS)?
CLAUDE.md user	~/.claude/CLAUDE.md
CLAUDE.md project	root /.claude/
CLAUDE.md directory	subtree
Targeted rules	.claude/rules/ + paths: glob	loading conditional

Typical diagnosis: “the new one doesn't have the conventions” → user level **** instead of project.

Slide 20 — Slash commands, Skills, plan mode

• .claude/commands/ = team (versioned) · ~/.claude/commands/ = staff
• Skills: SKILL.md — context: fork (insulation), allowed-tools, argument-hint
• Plan mode: big change + multiple approaches + architectural decisions
• Direct execution: simple fix, clear stack trace, one file
• Distractor: “start straight away, plan when things get stuck” (reactive = recovery already paid)

Slide 21 — Scenario 4: integrated tools

Exploring an unknown database: Grep entry points → Read to trace → incremental

Slide 22 — Scenario 5: Claude Code in CI/CD

# Stuck in CI? The -p flag is missing:
claude -p "Analyse this PR for security" \
  --output-format json --json-schema review.json

• -p / --print = headless documented
• Invented: CLAUDE_HEADLESS=true, --batch
• CLAUDE.md = project context for CI (standards, fixtures, criteria)

Slide 23 — PR review: architecture

1. Independent instance — the session that generated the code reads poorly
2. False positives: explicit categorical criteria (report/ignore) ≫ “be conservative”
3. Noisy category → temporarily disable (it undermines trust in reliable categories)
4. Big PR: passes per file + integration pass — larger window doesn't fix attention
5. Re-run after commits: include previous review → only report new

Slide 24 — Batch vs. real-time

Load	API
Blocking verification before merge	Synchronous
Night report, weekly audit	Batches (−50% , ≤ 24 h , no SLA)

• custom_id to correlate requests/responses
• Partial failure → resubmit only failures
• No multi-turn tool calls in a batch query

Slide 25 — Scenario 6: The Reliable Mining Stack

1. tool_use + JSON schema = guaranteed syntax
2. Semantics is never true: wrong field, inconsistent totals
3. Fields nullable when the info may be absent (required = forced hallucination)
4. Enums with "other" / "unclear" + detail field
5. Self-verification: calculated_total vs stated_total
6. Retry with feedback of specific errors — useless if the info is missing from the source

Slide 26 — Scenario 6: calibration & human supervision

“97% overall accuracy” can hide 64% on the IBAN field of scans.

• Accuracy by document type × by field before automating
• Confidence at field level, calibrated on a labeled set
• Human review routing: low trust, semantic failure, ambiguous source
• Continuous stratified random sampling after automation

Slide 27 — Time strategy on the big day

1. Passage 1: respond to everything that comes out in < 60 s, mark the rest
2. Passage 2: the marked questions
3. Last minute: zero empty questions (no penalty)
4. Scenario questions: read the question before the situation
5. After each mistake in training: “Which distractor family got me?”

Slide 28 — Exam simulator (20 min)

Session web page → “Simulator” tab- 12 questions · 18 minutes · real conditions, without documentation

• Individual score + collective correction of the 3 most missed questions
• Objective: identify your dominant distractor bias

Slide 29 — Prioritized revision plan

Web page → “Domain Tracker” tab: self-assessment // by skill, score weighted by exam weights

Prioritization rule:

1. The of heavy domains (D1 27% · D3 20% · D4 20%)
2. Then the
3. A in D1 ≈ 4 questions lost · in D5 ≈ 1.5

Slide 30 — Summary: the 8 reflexes

1. Critical business rule → hook, never the prompt alone
2. Bad routing → tool descriptions first
3. Subagent = isolated context, everything goes in the prompt
4. Partial coverage → suspect the decomposition of the coordinator
5. -p in CI · project = versioned, user = staff
6. JSON Schema = syntax, never semantics · nullable anti-hallucination
7. Diluting numbers → persistent “case facts” block
8. Global metric → stratify type × field before automating

Exercises: S1 architecture · S3 comparison · S6 evaluation grid. Session 10: complete mock exam.