# Exercises — Advanced Level, Session 10
# “Mock exam & final project”
**Program:** Applied AI — Yann Isola
**Content:** detailed evaluation grid for the final project (/100) + 3 mini architecture challenges for warm-up
**Usage:** the mini-challenges are done at the opening of the session or in individual review (10 min each); The grid is used for block 3 defenses and for upstream self-assessment.
> ⚠ Exam figures (60 questions, 90 min, threshold 72%, weightings) and API prices/limits quoted are volatile: check official Anthropic sources.
---
# Part 1 — The final project: framework and evaluation grid
## The expected deliverable
Each participant presents **the complete architecture of an AI system responding to a real problem** — ideally from their professional context — designed and refined over sessions 4 to 9. The deliverable includes:
1. **An architectural file** (4–8 pages or slide equivalent) submitted before the session.
2. **A 4-minute defense** + 2 minutes of questions.
3. **A mandatory architecture diagram** (flows, components, human control points).
The four families of projects worked on in the course are valid bases (to be personalized):
- Multi-agent customer support system (routing, specialists, human escalation)
- MCP server (MCP = Model Context Protocol) for a business domain with tools, resources and prompts
- Integration Claude Code into a team's CI/CD pipeline (CI/CD = continuous integration and deployment)
- Document processing pipeline with full evaluation harness
## Evaluation grid — /100
### Criterion 1 — Business problem & framing (15 points)
| Level | Points | Descriptor |
|---|---|---|
| Great | 13–15 | Real, quantified problem (volume, current cost, issue); explicit scope with what is *excluded* and why; Measurable success criteria defined before the architecture. |
| Solid | 9–12 | Real and clear problem; defined perimeter; success criteria present but partially measurable. |
| Fragile | 5–8 | Plausible but unquantified problem; unclear perimeter; success defined as “it works”. |
| Insufficient | 0–4 | Artificial problem or pretext; no notion of perimeter or measurement. |
**Related jury question:** “How will you know, based on figures, that the system is worth its cost? »
### Criterion 2 — Architecture & justification of choices (30 points)
The heart of the note. We evaluate the **correctness of the level of complexity** as much as the design itself.
| Sub-criterion | Points | What we check |
|---|---|---|
| 2a. Level of complexity justified | 10 | Each stage (single call → chain → agent → multi-agents) is justified by a demonstrated insufficiency of the lower stage. Over-engineering is penalized in the same way as under-engineering. |
| 2b. Division & responsibilities | 8 | Single responsibility components; clear boundaries; isolated contexts where useful; the diagram corresponds to the speech. |
| 2c. Choice of models & routing | 6 | Fast/economical model for closed tasks, powerful model where the reasoning justifies it; request routing is explicit. |
| 2d. State & Session Management | 6 | Sustainable state out of context (the API is stateless); recovery strategy; compaction planned for long sessions. |
**Related jury questions:** “Why not a simple workflow here? » · “What breaks if you double the volume?” »### Criterion 3 — Design of tools & integrations (15 points)
| Level | Points | Descriptor |
|---|---|---|
| Great | 13–15 | Task-oriented tools (not modeled on the internal API); descriptions with scope AND contraindications; tool errors returned as actionable data; if MCP: primitives (tools/resources/prompts) correctly assigned, transport justified. |
| Solid | 9–12 | Consistent tools, correct diagrams; honest descriptions but without contraindications; error handling present. |
| Fragile | 5–8 | Granularity modeled on the existing API (too fine or too large); vague descriptions; errors = technical exceptions. |
| Insufficient | 0–4 | “Catch-all” tools or raw access (SQL/shell) without guardrails; no thought about tool selection. |
### Criterion 4 — Reliability, security & human supervision (20 points)
| Sub-criterion | Points | What we check |
|---|---|---|
| 4a. Deposit vs. guarantee | 6 | Everything critical (exit validation, permissions, action limits) is enforced by **code**, not just requested in a prompt. The deposit/guarantee confusion eliminates excellence. |
| 4b. Human in the loop | 5 | Approval point placed at the point of no return, proportionate to the risk (thresholds); neither approval everywhere (unusable system), nor total autonomy over the irreversible. |
| 4c. Failure modes | 5 | Answered questions: What happens if a tool fails? if the API returns 429/529? if the model is hallucinating? Graceful degradation described, not just “error handling”. |
| 4d. Attack surface | 4 | Prompt injection via external content planned; sensitive data (PII = personally identifiable data) minimized in contexts and logs. |
**Red flag (to be sanctioned):** “the model is intelligent, it will manage. »
### Criterion 5 — Costs, evaluation & operation (10 points)
| Level | Points | Descriptor |
|---|---|---|
| Great | 9–10 | Estimated cost per request and per month at target volume ⚠; exploited prompt cache (stable content in head); defined evaluation harness (golden set, metrics, triggering); provenance logging (prompt version, template, tools called). |
| Solid | 6–8 | Orders of magnitude of cost present; planned but summary assessment; logging mentioned. |
| Fragile | 3–5 | Costs “to see later”; no automated evaluation; logs = generic application logs. |
| Insufficient | 0–2 | No concept of cost, evaluation or traceability. |
### Criterion 6 — Clarity of the defense (10 points)
| Level | Points | Descriptor |
|---|---|---|
| Great | 9–10 | Lasts 4 minutes; readable and exploitable diagram; prioritize (essentials first); answers questions by assuming boundaries (“not covered, and here’s why it’s acceptable”). |
| Solid | 6–8 | Clear structure, slight overhang; answers questions honestly. |
| Fragile | 3–5 | Clearly overflows; reads his slides; avoids difficult questions. |
| Insufficient | 0–2 | Confused, without a pattern, defensive or fabricated responses. |
## Overall scale
| Total | Verdict |
|---|---|
| 85–100 | Confirmed architect level — the file could be presented to a real architecture committee. |
| 70–84 | Solid — a few blind spots identified, fixed in one iteration. |
| 50–69 | Foundations in place — rework the criteria < 60% before any use in production.|
| <50 | Resume the design with the support of the guides from sessions 4 to 8. |
**Modality of marking during the session:** cross-assessment (2 peers per project) + trainer's note; indicative score = average of the three. The gap between prior self-assessment and grade received is in itself a lesson — the chronic under-estimator and the chronic over-confident both have a bias to correct before the exam.
---
# Part 2 — Three architecture mini-challenges (warm-up)
Format: 10 minutes per challenge, alone or in pairs. Produce: (1) an architectural choice in one sentence, (2) three justifications, (3) the main residual risk. Then compare to the answer key.
---
## Challenge A — “Claims triage”
**Statement.** An insurer receives 30,000 claims reports per day (free form + photos). You must: classify the type of claim (7 categories), estimate the urgency (3 levels), extract the key fields in JSON, and transmit the cases “maximum emergency + estimated amount > €10k” to a human manager. Target latency: <5s. The team offers a multi-agent system with an orchestrator and seven specialized agents (one per disaster category).
**Question.** Validate or counter-propose, justifying.
Fixed
**Counterproposal: no multi-agents.** It is a **pipeline with fixed stages** — the path is known in advance (classify → estimate → extract → route), nothing depends on intermediate discoveries.
Target architecture: **a single multimodal call** (text + photos) with structured output forced by tool diagram (enumerated category, urgency, extracted fields, estimated amount) on a fast model; then **programmatic rule** (not a prompt!) for escalation “max emergency + > 10 k€ → human queue”. Possibly a second verification call on escalated cases only (low volume, high stakes).
Justifications: (1) high volume + closed task → latency and cost dominate, one call is enough; (2) escalation is a **critical threshold** → guaranteed by code, never by instruction; (3) the scheme forces the structure → the downstream pipeline does not break.
Residual risk: ambiguous cases between categories — are handled by a few-shot of borderline cases + evaluation on golden set, not by additional agents.
**Trap avoided:** “7 categories = 7 agents” — the business taxonomy does not dictate the topology of the system.
---
## Challenge B — “The Investigation Assistant”
**Statement.** An SRE team (SRE = Site Reliability Engineering) wants an assistant who, in the event of an incident, explores on its own: logs, metrics, recent changes (deployments, merged PRs), and offers a diagnosis with the evidence. The causes are by nature unpredictable. The assistant has read access to the observability systems and the Git repository. The team hesitates: scripted workflow or autonomous agent? And should we give it the right to restart the services?
**Question.** Cut both points, with guardrails.
Fixed
**Point 1: agent.** Case study — each clue determines the next investigation, the path is unknowable in advance. A scripted workflow would only cover previously seen failures.Agent safeguards: capped iteration budget, context compaction for long investigations, **read-only** tools with descriptions including contraindications, obligation to cite evidence (log extracts, deployment identifiers) — verifiable citation is the anti-hallucination of the diagnosis.
**Point 2: no to the autonomous restart — but yes to the *proposed* restart.** The restart is an action with side effects on production: point of no return → human approval. Design: the agent produces a structured proposal (action, target, justification, risk), a human approves in one click, execution is done by the system with complete logging. The read/write boundary is enforced by the **permissions** (guarantee), not by the prompt (instruction).
Residual risk: injection via the logs themselves (an attacker can write disguised instructions in the logs) → treat all log content as unreliable data, never as instructions.
---
## Challenge C — “Team knowledge in Claude Code”
**Statement.** A team of 15 developers adopts Claude Code on a monorepo (monorepo = single repository containing several projects). Findings after 3 weeks: the agent offers false build commands, ignores team conventions, and an incident occurred — a developer let the agent run a basic migration on the staging environment without replay. Propose the complete governance system.
**Question.** What files, what mechanisms, what deposit/guarantee distribution?
Fixed
**Three layers:**
1. **`CLAUDE.md` versioned at the root (+ per sub-project of the monorepo)** — the *standing instructions*: exact build/test commands per sub-project, code conventions, folder architecture, known pitfalls. Short and dense: loaded at each session, each line costs tokens. Reviewed in code review like any file.
2. **`.claude/settings.json` versioned** — the *shared guarantee*: allowlist of commands always authorized (tests, lint, build), mandatory approval for the rest, **explicit ban** on migration commands and everything related to staging/production. This is what would have prevented the incident — a `CLAUDE.md` saying “do not migrate without rereading” would *not* have prevented it (workaround/forgettable instruction).
3. **`.claude/commands/`** — *on-demand rituals*: `/revue-pr`, `/notes-version`, `/verif-migration` (which prepares and verifies a migration without executing it). Charged only on call: zero permanent cost.
Add-ons: default plan mode recommended for any multi-file change; headless automatic review in CI (`claude -p … --output-format json` ⚠, read-only); the incident becomes a postmortem which feeds `CLAUDE.md` (known trap) AND `settings.json` (ban).
**Distribution to remember:** what the agent must *know* → `CLAUDE.md`; what the agent should not *be able to* → `settings.json` ; what is asked *sometimes* → slash commands.
---
## After the challenges
Report your three self-diagnoses in the preparation dashboard (web page, “Preparation” tab): each challenge covers areas D1+D4 (challenge A), D1+D2+D5 (challenge B), D3 (challenge C) respectively. A failed challenge with the answer key misunderstood = red zone on the associated areas, regardless of your general impression.