Exercises — Advanced Level, Session 10
« Mock exam & final project »
Program : Applied AI — Yann Isola Content : detailed evaluation grid for the final project (/100) + 3 mini architectural challenges for warm-up Usage: the mini-challenges are done at the opening of the session or in individual review (10 min each); The grid is used for defenses of block 3 and for upstream self-assessment.
⚠ Exam figures (60 questions, 90 min, threshold 72%, weightings) and API rates/limits quoted are volatile: check official Anthropic sources.
Part 1 — The final project: framework and evaluation grid
The expected deliverable
Each participant presents the complete architecture of an AI system responding to a real problem — ideally from its professional context — designed and refined over sessions 4 to 9. The deliverable includes:
- An architectural file (4–8 pages or slide equivalent) submitted before the session.
- A 4 minute defense + 2 minutes of questions.
- An architectural diagram mandatory (flows, components, human checkpoints).
The four families of projects worked on in the course are valid bases (to be personalized):
- Multi-agent customer support system (routing, specialists, human escalation)
- MCP server (MCP = Model Context Protocol) for a business domain with tools, resources and prompts
- Integration Claude Code in a team's CI/CD (CI/CD = continuous integration and deployment) pipeline
- Document processing pipeline with full assessment harness
Evaluation grid — /100
Criterion 1 — Problem & business framework (15 points)
| Level | Points | Descriptor |
|---|---|---|
| Excellent | 13–15 | Real, quantified problem (volume, current cost, issue); explicit perimeter with what is excluded and why; Measurable success criteria defined before the architecture. |
| Solid | 9–12 | Real and clear problem; defined perimeter; success criteria present but partially measurable. |
| Fragile | 5–8 | Plausible but unquantified problem; unclear perimeter; success defined as “it works”. |
| Insufficient | 0–4 | Artificial problem or pretext; no notion of perimeter or measurement. |
Related jury question: “How will you know, with supporting figures, that the system is worth its cost? »
Criterion 2 — Architecture & justification of choices (30 points)
The heart of the note. We evaluate the correctness of complexity level as much as the design itself.
| Sub-criterion | Points | What we check |
|---|---|---|
| 2a. Justified level of complexity | 10 | Each stage (single call → chain → agent → multi-agents) is justified by a demonstrated insufficiency of the lower stage. Over-engineering is penalized in the same way as under-engineering. |
| 2b. Cutting & responsibilities | 8 | Single responsibility components; clear boundaries; isolated contexts where useful; the diagram corresponds to the speech. |
| 2c. Choice of models & routing | 6 | Fast/economical model for closed tasks, powerful model where the reasoning justifies it; request routing is explicit. |
| 2d. State management & sessions | 6 | Sustainable state out of context (the API is stateless); recovery strategy; compaction planned for long sessions. |
Related jury questions: “Why not a simple workflow here? » · “What breaks if you double the volume?” »
Criterion 3 — Tool design & integrations (15 points)
| Level | Points | Descriptor |
|---|---|---|
| Excellent | 13–15 | Task-oriented tools (not modeled on the internal API); descriptions with scope AND contraindications; tool errors returned as actionable data; if MCP: primitives (tools/resources/prompts) correctly assigned, transport justified. |
| Solid | 9–12 | Consistent tools, correct diagrams; honest descriptions but without contraindications; error handling present. |
| Fragile | 5–8 | Granularity modeled on the existing API (too fine or too large); vague descriptions; errors = technical exceptions. |
| Insufficient | 0–4 | “Catch-all” tools or raw access (SQL/shell) without guardrails; no thought about tool selection. |
Criterion 4 — Reliability, security & human supervision (20 points)
| Sub-criterion | Points | What we check |
|---|---|---|
| 4a. Deposit vs. guarantee | 6 | Everything that is critical (exit validation, permissions, action limits) is applied by code , not just asked in a prompt. The deposit/guarantee confusion eliminates excellence. |
| 4b. Human in the loop | 5 | Approval point placed at the point of no return, proportionate to the risk (thresholds); neither approval everywhere (unusable system), nor total autonomy over the irreversible. |
| 4c. Failure modes | 5 | Answered questions: What happens if a tool fails? if the API returns 429/529? if the model is hallucinating? Graceful degradation described, not just “error handling”. |
| 4d. Attack surface | 4 | Prompt injection via external content planned; sensitive data (PII = personally identifiable data) minimized in contexts and logs. |
Red flag (to be sanctioned): “the model is intelligent, it will manage. »
Criterion 5 — Costs, evaluation & exploitation (10 points)
| Level | Points | Descriptor |
|---|---|---|
| Excellent | 9–10 | Estimated cost per request and per month at target volume ⚠; exploited prompt cache (stable content in head); defined evaluation harness (golden set, metrics, triggering); provenance logging (prompt version, template, tools called). |
| Solid | 6–8 | Orders of magnitude of cost present; planned but summary assessment; logging mentioned. |
| Fragile | 3–5 | Costs “to see later”; no automated evaluation; logs = generic application logs. |
| Insufficient | 0–2 | No concept of cost, evaluation or traceability. |
Criterion 6 — Clarity of the defense (10 points)
| Level | Points | Descriptor |
|---|---|---|
| Excellent | 9–10 | Lasts 4 minutes; readable and exploitable diagram; prioritize (the essentials first); answers questions by assuming boundaries (“not covered, and here’s why it’s acceptable”). |
| Solid | 6–8 | Clear structure, slight overhang; answers questions honestly. |
| Fragile | 3–5 | Clearly overflows; reads his slides; avoids difficult questions. |
| Insufficient | 0–2 | Confused, without a pattern, defensive or fabricated responses. |
Overall scale
| Total | Verdict |
|---|---|
| 85–100 | Confirmed architect level — the file could be presented to a real architecture committee. |
| 70–84 | Solid — a few blind spots identified, fixed in one iteration. |
| 50–69 | Foundations in place — rework the criteria < 60% before any use in production. |
| < 50 | Resume the design with the support of the guides from sessions 4 to 8. |
Scoring method during the session: cross-assessment (2 peers per project) + trainer’s note; indicative score = average of the three. The gap between prior self-assessment and grade received is in itself a lesson — the chronic under-estimator and the chronic over-confident both have a bias to correct before the exam.
Part 2 — Three mini architecture challenges (warm-up)
Format: 10 minutes per challenge, alone or in pairs. Produce: (1) an architectural choice in one sentence, (2) three justifications, (3) the main residual risk. Then compare to the answer key.
Challenge A — “Claims triage”
Statement. An insurer receives 30,000 claims reports per day (free form + photos). You must: classify the type of claim (7 categories), estimate the urgency (3 levels), extract the key fields in JSON, and transmit the cases “maximum urgency + estimated amount > €10k” to a human manager. Target latency: < 5 sec. The team offers a multi-agent system with an orchestrator and seven specialized agents (one per disaster category).
Question. Validate or counter-propose, justifying.
Corrected
Counter-proposal: no multi-agents. It's a fixed-stage pipeline — the path is known in advance (classify → estimate → extract → route), nothing depends on intermediate discoveries.
Target architecture: a single multimodal call (text + photos) with structured output forced by tool diagram (enumerated category, urgency, extracted fields, estimated amount) on a quick model; Then programmatic rule (not a prompt!) for climbing “max emergency + > 10 k€ → human queue”. Possibly a second verification call on escalated cases only (low volume, high stakes).
Justifications: (1) high volume + closed task → latency and cost dominate, one call is enough; (2) climbing is a critical threshold → guaranteed by code, never by deposit; (3) the scheme forces the structure → the downstream pipeline does not break.
Residual risk: ambiguous cases between categories — are handled by a few-shot of borderline cases + evaluation on golden set, not by additional agents.
Trap avoided: “7 categories = 7 agents” — the business taxonomy does not dictate the system topology.
Challenge B — “The Investigation Assistant”
Statement. An SRE team (SRE = Site Reliability Engineering) wants an assistant who, in the event of an incident, explores on its own: logs, metrics, recent changes (deployments, merged PRs), and offers a diagnosis with the evidence. The causes are by nature unpredictable. The assistant has read access to the observability systems and the Git repository. The team hesitates: scripted workflow or autonomous agent? And should we give it the right to restart the services?
Question. Cut the two points, with the guardrails.
Corrected
Point 1: agent. Case study — each clue determines the next investigation, the path is unknowable in advance. A scripted workflow would only cover previously seen failures. Agent safeguards: capped iteration budget, context compaction for long investigations, monitoring tools read only with descriptions including contraindications, obligation to cite the evidence (log extracts, deployment identifiers) — the verifiable citation is the anti-hallucination of the diagnosis.
Point 2: no to standalone reboot — but yes to reboot propose . Restarting is an action with side effects on production: point of no return → human approval. Design: the agent produces a structured proposal (action, target, justification, risk), a human approves in one click, execution is done by the system with complete logging. The read/write boundary is enforced by the permissions (guarantee), not by the prompt (deposit).
Residual risk: injection via the logs themselves (an attacker can write disguised instructions in the logs) → treat all log content as unreliable data, never as instructions.
Challenge C — “Team knowledge in Claude Code”
Statement. A team of 15 developers adopts Claude Code on a monorepo (monorepo = single repository containing several projects). Findings after 3 weeks: the agent offers false build commands, ignores team conventions, and an incident occurred — a developer let the agent run a basic migration on the staging environment without proofreading. Propose the complete governance system.
Question. What files, what mechanisms, what deposit/guarantee distribution?
Corrected
Three layers:
CLAUDE.mdversioned at the root (+ per sub-project of the monorepo) - there standing order : exact build/test commands per subproject, code conventions, folder architecture, known pitfalls. Short and dense: loaded at each session, each line costs tokens. Reviewed in code review like any file..claude/settings.jsonversioned - there shared guarantee : allowlist of commands always authorized (tests, lint, build), approval required for the rest, explicit ban migration commands and everything related to staging/production. This is what would have prevented the incident — aCLAUDE.mdsaying “do not migrate without proofreading” would not have not prevented (bypassable/forgettable instruction)..claude/commands/- THE rituals on demand :/revue-pr,/notes-version,/verif-migration(which prepares and verifies a migration without executing it). Charged only on call: zero permanent cost.
Add-ons: default plan mode recommended for any multi-file change; automatic headless review in CI (claude -p … --output-format json ⚠, read only); the incident becomes a post-mortem which fuels CLAUDE.md (known trap) AND settings.json (prohibition).
Distribution to remember: what the agent must know → CLAUDE.md ; what the agent should not power → settings.json ; what we ask of him Sometimes → slash commands.
After the challenges
Report your three self-diagnoses in the preparation dashboard (web page, “Preparation” tab): each challenge covers areas D1+D4 (challenge A), D1+D2+D5 (challenge B), D3 (challenge C) respectively. A failed challenge with the answer key misunderstood = red zone on the associated areas, regardless of your general impression.