# Teacher guide — Session 3 (Intermediate level)
# Structured outing, temperature & assessments

**Program:** Applied AI — Yann Isola
**Duration:** 2 hours
**Prerequisites:** Sessions 1–2 (anatomy of a prompt, context, system/user roles, API concepts — Application Programming Interface, application programming interface)
**Source module:** Module 2, part 2

---

## 1. Educational objectives

At the end of the session, each participant should be able to:

1. **Explain** why a “generally valid” JSON (JavaScript Object Notation, structured data exchange format) output is unacceptable in production, and cite the mechanism that guarantees syntactic validity (function calling / schema-constrained structured output).
2. **Distinguish** *syntactic* error (malformed JSON) and *semantic* error (valid JSON but false content), and associate the correct solution to each.
3. **Demystify** temperature: temperature 0 = deterministic, NOT = correct. It controls variability, not veracity.
4. **Describe** the 4-level evaluation stack: assertions → golden set → LLM (Large Language Model) judge → human evaluation.
5. **Build** a robust extraction prompt that survives edge cases (letter amounts, injection attempts, empty strings) — this is the lab of the session.

**Central thesis of the session (to be hammered out):** *“The evaluation suite IS the specification. »* An AI product without evaluations is a product whose real behavior no one knows. “It looked good in the demo” is the #1 killer of AI products.

---

## 2. Material

- Video projector + slides (`slides/slides.md`)
- Interactive page (`webpage/index.html`) — works **offline**, to be opened in a browser; contains the JSON schema validator, the assessment dashboard simulator and the temperature comparator
- Printed or shared worksheets (`exercises/exercises.md`)
- Access to an LLM playground (Claude, ChatGPT, Mistral… whatever) for the lab — 1 station for 2 participants is enough
- Quiz (`quiz/quiz.md`) — to do at the end of the session or asynchronously

---

## 3. Timed course (120 min)

### Block A — Hook: the JSON that breaks everything (0:00 → 0:15, 15 min)

**Shock opening (5 min).** Tell the typical story: a team deploys an invoice extractor. In demo, the model returns perfect JSON 50 times in a row. In production, on the 3,412th invoice, it returns:```
Voici le JSON demandé :
```json
{"amount": 1250, "currency": "EUR"```

Virgule manquante, préambule verbeux, bloc de code non fermé. Le parseur crashe, le pipeline s'arrête à 2 h du matin. **Question à la salle :** « Qui a déjà vu un LLM ajouter du texte avant le JSON alors qu'on lui a dit "réponds UNIQUEMENT en JSON" ? » (Les mains se lèvent — c'est universel.)

**Message clé (10 min).** « Généralement valide » × volume = échec certain. Si le modèle produit du JSON valide 99 % du temps et que vous traitez 10 000 documents/jour, vous avez **~100 crashs par jour** ⚠ (chiffre illustratif — le taux réel varie selon modèle et prompt). La solution n'est pas un meilleur prompt, c'est une **contrainte structurelle**.

> **Note enseignant :** résistez à la tentation de montrer la solution tout de suite. Laissez les participants proposer leurs rustines habituelles (regex, retry aveugle, « je lui redemande gentiment »). Vous les démonterez au bloc B.

---

### Bloc B — Sortie structurée : function calling & schémas (0:15 → 0:45, 30 min)

**B1. Les rustines et pourquoi elles fuient (5 min).**
- Regex pour extraire le JSON du texte → fragile, casse sur les JSON imbriqués.
- « Réponds uniquement en JSON » dans le prompt → améliore, ne garantit pas.
- Retry aveugle → coûte des tokens, ne converge pas toujours.

**B2. La vraie solution : tool_use / function calling (15 min).**

Expliquez le mécanisme : au lieu de demander du texte « qui ressemble à du JSON », on déclare un **outil** (tool) avec un **schéma JSON** (JSON Schema : standard de description de la structure attendue — types, champs obligatoires, énumérations). Le fournisseur du modèle **contraint le décodage** : les tokens générés ne peuvent former qu'un document conforme au schéma. L'erreur syntaxique devient structurellement impossible.

Exemple à projeter (extraction de note de frais) :

```json
{
"name": "extract_expense_note",
"description": "Extracts fields from an expense report",
"input_schema": {
"type": "object",
"properties": {
"amount": { "type": "number", "description": "Amount including tax in numerical value" },
"currency": { "type": "string", "enum": ["EUR", "USD", "GBP", "CHF"] },
"date": { "type": "string", "description": "ISO 8601 format: YYYY-MM-DD" },
"category": { "type": "string", "enum": ["transport", "meals", "accommodation", "other"] }
},
"required": ["amount", "currency", "date", "category"]
}
}```

Points d'insistance :
- `enum` : le modèle ne peut PAS inventer une catégorie hors liste.
- `required` : les champs obligatoires seront présents (syntaxiquement).
- Tous les grands fournisseurs offrent une variante : *tool use* (Anthropic), *function calling* et *structured outputs* (OpenAI), *function calling* (Google, Mistral). Les noms diffèrent, le principe est identique.

**B3. La limite fondamentale (10 min) — transition vers le bloc C.**

Écrivez au tableau, en grand :

> **Le schéma garantit la FORME, jamais le FOND.**

Exemple concret : le reçu dit « quarante-deux euros ». Le modèle renvoie `{"montant": 402, "devise": "EUR", ...}` — JSON **parfaitement valide**, contenu **faux**. C'est une **erreur sémantique**. Le validateur de schéma ne la verra jamais.

**Démo interactive :** ouvrez `webpage/index.html`, onglet « Validateur de schéma ». Faites valider un JSON syntaxiquement correct mais sémantiquement faux — le validateur dit ✅. Effet garanti.

---

### Bloc C — Erreurs sémantiques : valider, réessayer, corriger (0:45 → 1:05, 20 min)

**C1. Taxonomie (5 min).** Tableau à deux colonnes :

| | Erreur syntaxique | Erreur sémantique |
|---|---|---|
| **Exemple** | Virgule manquante, champ absent | Montant faux, date hallucinée, catégorie plausible mais erronée |
| **Détection** | Parseur / validateur de schéma | Règles métier, recoupement, juge LLM, humain |
| **Parade** | Function calling (élimination structurelle) | Validation + retry avec feedback + garde-fous métier |

**C2. Le pattern « validate → retry with feedback » (10 min).**

Pseudo-code à projeter :

```result = call_llm(document, schema)
errors = validate_metier(result) # ex: amount > 0, date <= today,
# currency consistent with the country
if errors:
result = call_llm(document, schema,
feedback="Your previous response contained: " + errors
+ ". Correct and return.")
if still in error after N attempts:
climb_to_human(document)
```

Key points:
- The feedback must be **specific** (“the amount 402 does not correspond to any number in the document”) — a blind retry without feedback almost never converges.
- **Always** an attempt cap (2–3) and a human climbing route. Infinite loop = infinite API bill.
- Business validations are classic, deterministic, free code: they catch a huge share of semantic errors (negative amounts, future dates, inconsistent totals).

**C3. Mini-quick exercise (5 min).** Orally: “For an expense report extractor, cite 3 business validation rules. » Expected: amount > 0; date not in the future; date not earlier than N years; currency ∈ allowed list; VAT ≤ amount including tax; etc.

---

### BREAK (1:05 → 1:15, 10 min)

---

### Block D — Temperature: determinism ≠ accuracy (1:15 → 1:30, 15 min)

**D1. What the temperature really does (7 min).**

The model produces a probability distribution on the next token. The temperature **reshapes this distribution**:
- **T = 0**: we (almost) always take the most probable token → very reproducible outputs.
- **High T (0.8–1.2)**: less probable tokens are drawn more → variety, creativity, but also drift.

**The trap to destroy (to say word for word):** “Temperature 0 does not make the model more intelligent or more factual. If the most likely token is false, T = 0 gives you **the same error, every time, with perfect regularity**. You get a repeatable lie, not a truth. »

Temperature controls **variability**, not **accuracy**.

> **Teacher's note:** also specify that even at T = 0, absolute determinism is not guaranteed by all suppliers (GPU parallelism — Graphics Processing Unit, graphics processor —, model updates). “Quasi-deterministic” is the honest term.

**D2. Practical settings (3 min).**

| Use cases | Recommended temperature ⚠ (usual benchmarks, to be adjusted) |
|---|---|
| Extraction, classification, structured output | 0 – 0.2 |
| Assisted writing, reformulation | 0.5 – 0.8 |
| Brainstorming, creativity | 0.9 – 1.2 |

**D3. Demo (5 min).** Open the “Temperature comparator” tab of the interactive page: same prompt simulated at T = 0 (3 identical outputs, including one error reproduced 3 times) vs T = 1 (3 different outputs). Ask: “What do you notice about the error at T = 0? »

---

### Block E — Evaluations: the series of evaluations IS the spec (1:30 → 1:50, 20 min)

**E1. The thesis (5 min).**

In traditional software, the spec says what the code must do, and the tests verify it. With LLMs, the behavior is **non-deterministic and not specifiable by code**: the only operational definition of “it works” is **the evaluation suite**. Hence the thesis:

> **“The suite of evaluations IS the spec. »**
> If a behavior is not covered by an evaluation, you don't know if it works.“It looked good in the demo” = you tested 5 cases chosen by the person who wanted it to work.

**E2. The 4-story evaluation stack (10 min).**

1. **Assertions** (code, free, instant): does JSON parse? Are the required fields there? Is the amount > 0? → catches the coarse, rotates on each commit.
2. **Golden set** (50 to 500 examples ⚠ usual order of magnitude): expected input → output pairs, validated by a human. Deliberately includes **borderline cases** and **adverse cases**. We measure a success rate; any modification of prompt or model reverts the golden set.
3. **LLM judge (LLM-as-judge)**: a second model rates the outputs according to a grid (fidelity, completeness, tone). Scalable, but the judge has his own biases → you have to **calibrate the judge against human judgments** before trusting him.
4. **Human evaluation**: regular sampling in production, review of escalated cases. Expensive, slow, but it’s the ground truth that recalibrates everything else.

**E3. Metrics by system type (5 min).**

- **RAG (Retrieval-Augmented Generation, generation augmented by document recovery)**: **fidelity** (*faithfulness*) — is the response supported by the recovered documents, without invention?
- **Agents**: **task completion rate** — is the task actually completed, not just “a response was produced”?
- **Systems with humans in the loop**: **escalation rate** — what proportion goes to a human? Too high = useless system; too low = check that it does not swallow cases that it should escalate.

**Demo:** “Evals Dashboard” tab of the interactive page. Run the v1 (naive) prompt against the test set: edge cases fail. Switch to the v3 (hardened) prompt: the rate increases. The message: **we are not comparing impressions, we are comparing success rates on the same test set.**

---

### Block F — Lab: the indestructible expense report extractor (1:50 → 2:00, 10 min session + continuation of exercises)

Start exercise 3 (lab). In session: presentation of the trapped test game (amounts in letters, injection, empty string) and first iteration round. The rest is done independently or at the start of the next session.

**Closing (2 min).** The 3 sentences to remember:
1. The diagram guarantees the form, never the substance.
2. Temperature 0 = reproducible, not correct.
3. No review = no spec = no product.

Distribute the quiz (or link) + exit tickets.

---

## 4. Teaching notes & common pitfalls

- **Pitfall 1:** participants conclude “with function calling, no need to validate”. False — immediately reframe with the example “forty-two → 402”.
- **Trap 2:** temperature / top-p / top-k confusion. Stay on temperature; just mention that other sampling parameters exist and that we do not accumulate them randomly.
- **Pitfall 3:** “50–500 examples is too much work.” Answer: Start at 20, enrich with each production bug. A living golden set is better than a perfect golden set never built.
- **Trap 4:** the LLM judge presented as an oracle. Remember: an uncalibrated judge is an automated opinion.
- **Heterogeneous audience:** non-developers can do the entire lab in a playground without writing code — the diagram is pasted into the tool interface, business validations are done on paper.

---

## 5. Exit tickets (5 questions, 5 min at the end of the session)Each participant responds in writing (paper or form):

1. **Explain in one sentence why “the model returns valid JSON 99% of the time” is a problem in production.**
*Expected: at high volume, 1% failure = daily outages; only the schema constraint (function calling) eliminates the error class.*

2. **The model returns `{"montant": 402}` for a receipt of “forty-two euros”. Syntactic or semantic? What parade?**
*Expected: semantics; business validation / cross-checking + retry with specific feedback, human escalation if repeated failure.*

3. **True or false: “At temperature 0, the model no longer makes factual errors. » Justify.**
*Expected: False. T=0 makes the error reproducible, not absent. Temperature controls variability, not accuracy.*

4. **List the 4 levels of the evaluation stack, in ascending cost order.**
*Expected: assertions → golden set → LLM judge → human evaluation.*

5. **Your RAG assistant responds fluently but makes up references. Which metric to monitor, and by which level of the battery to measure it continuously?**
*Expected: faithfulness; calibrated LLM judge (with periodic human sampling).*

---

## 6. Links to the rest of the program

- **Session 2** (context & prompts): prompt techniques remain necessary — the diagram constrains the form, the prompt guides the substance.
- **Session 4** (RAG): the fidelity metric introduced here becomes central.
- **Agent sessions**: the task completion rate and escalation will be instrumented in practice.