Français
Applied AI · Intermediate 🟡 · Session 3
📝 Teacher's Guide
← Return to program 📄 Source .md

Teacher guide — Session 3 (Intermediate level)

Structured output, temperature & evaluations

Program : Applied AI — Yann Isola Duration : 2:00 a.m. Prerequisites: Sessions 1–2 (anatomy of a prompt, context, system/user roles, API concepts — Application Programming Interface) Source module: Module 2, part 2

1. Educational objectives

At the end of the session, each participant should be able to:

  1. Explain why a “generally valid” JSON (JavaScript Object Notation, structured data exchange format) output is unacceptable in production, and cite the mechanism that guarantees syntactic validity (function calling / structured output constrained by schema).
  2. Distinguish error syntactic (malformed JSON) and error semantics (valid JSON but false content), and associate the correct parade with each.
  3. Debunking temperature: temperature 0 = deterministic, PAS = correct. It controls variability, not veracity.
  4. Describe the 4-level evaluation stack: assertions → golden set → LLM (Large Language Model) judge → human evaluation.
  5. Build a robust extraction prompt that survives edge cases (amounts in letters, injection attempts, empty strings) — this is the lab of the session.

Central thesis of the session (to be hammered out): “The evaluation suite IS the specification. » An AI product without evaluations is a product whose real behavior no one knows. “It looked good in the demo” is the #1 killer of AI products.

2. Material

3. Timed course (120 min)

Block A — Hook: the JSON that breaks everything (0:00 → 0:15, 15 min)

Shock opening (5 min). Tell the typical story: A team deploys an invoice extractor. In demo, the model returns perfect JSON 50 times in a row. In production, on the 3,412th invoice, it returns:

Voici le JSON demandé :
```json
{"montant": 1250, "devise": "EUR"

Missing comma, wordy preamble, unclosed code block. The parser crashes, the pipeline stops at 2 a.m. Question to the room: “Who has ever seen an LLM add text before JSON when they were told “ONLY answer in JSON”? » (Hands go up — it’s universal.)

Key message (10 min). “Generally valid” × volume = certain failure. If the model produces JSON valid 99% of the time and you process 10,000 documents/day, you have ~100 crashes per day ⚠ (illustrative figure — the actual rate varies depending on model and prompt). The solution is not a better prompt, it’s a structural constraint .

Teacher note: Resist the temptation to show the solution right away. Let the participants suggest their usual fixes (regex, blind retry, “I’ll ask him again nicely”). You will dismantle them in Block B.

Block B — Structured output: function calling & diagrams (0:15 → 0:45, 30 min)

B1. Patches and why they leak (5 min).

B2. The real solution: tool_use / function calling (15 min).

Explain the mechanism: instead of requesting text “that looks like JSON”, we declare a tool (tool) with a JSON schema (JSON Schema: standard for describing the expected structure — types, mandatory fields, enumerations). The model supplier constrains decoding : the generated tokens can only form a document conforming to the schema. The syntactic error becomes structurally impossible.

Example to project (expense report extraction):

{
  "name": "extraire_note_de_frais",
  "description": "Extrait les champs d'une note de frais",
  "input_schema": {
    "type": "object",
    "properties": {
      "montant": { "type": "number", "description": "Montant TTC en valeur numérique" },
      "devise":  { "type": "string", "enum": ["EUR", "USD", "GBP", "CHF"] },
      "date":    { "type": "string", "description": "Format ISO 8601 : AAAA-MM-JJ" },
      "categorie": { "type": "string", "enum": ["transport", "repas", "hebergement", "autre"] }
    },
    "required": ["montant", "devise", "date", "categorie"]
  }
}

Points of emphasis:

B3. The fundamental limit (10 min) — transition to block C.

Write on the board, large:

The diagram guarantees the FORM, never the SUBSTANCE.

Concrete example: the receipt says “forty-two euros”. The model returns {"montant": 402, "devise": "EUR", ...} — JSON perfectly valid , content fake . It's a semantic error . The schema validator will never see it.

Interactive demo: open webpage/index.html , “Schema Validator” tab. Validate a syntactically correct but semantically wrong JSON — the validator says . Guaranteed effect.

Block C — Semantic errors: validate, retry, correct (0:45 → 1:05, 20 min)

C1. Taxonomy (5 min). Two-column table:

Syntactic error Semantic error
Example Missing comma, missing field False amount, hallucinated date, plausible but erroneous category
Detection Schema parser/validator Business rules, cross-checking, LLM judge, human
Parade Function calling (structural elimination) Validation + retry with feedback + business safeguards

C2. The “validate → retry with feedback” pattern (10 min).

Pseudo-code to project:

resultat = appel_llm(document, schema)
erreurs = valider_metier(resultat)   # ex : montant > 0, date <= aujourd'hui,
                                     #      devise cohérente avec le pays
si erreurs:
    resultat = appel_llm(document, schema,
                         feedback="Ta réponse précédente contenait : " + erreurs
                                 + ". Corrige et renvoie.")
    si toujours en erreur après N tentatives:
        escalader_vers_humain(document)

Key points:

C3. Quick mini-exercise (5 min). Orally: “For an expense report extractor, cite 3 business validation rules. » Expected: amount > 0 ; date not in the future; date not earlier than N years; currency ∈ allowed list; VAT ≤ amount including tax; etc.

BREAK (1:05 ​​→ 1:15, 10 mins)

Block D — Temperature: determinism ≠ accuracy (1:15 → 1:30, 15 min)

D1. What Temperature Really Does (7 min).

The model produces a probability distribution on the next token. The temperature reshapes this distribution :

The trap to destroy (to say word for word): “Temperature 0 does not make the model smarter or more factual. If the most likely token is false, T = 0 gives you the same mistake, every time, with perfect regularity . You get a repeatable lie, not a truth. »

The temperature controls the variability , not the exactness .

Teacher note: also specify that even at T = 0, absolute determinism is not guaranteed by all suppliers (GPU parallelism — Graphics Processing Unit, graphics processor —, model updates). “Quasi-deterministic” is the honest term.

D2. Practical settings (3 min).

Use cases Recommended temperature ⚠ (usual references, to be adjusted)
Extraction, classification, structured output 0 – 0.2
Assisted writing, reformulation 0.5 – 0.8
Brainstorming, creativity 0.9 – 1.2

D3. Demo (5 mins). Open the “Temperature comparator” tab of the interactive page: same prompt simulated at T = 0 (3 identical outputs, including one error reproduced 3 times) vs T = 1 (3 different outputs). Ask: “What do you notice about the error at T = 0? »

Block E — Evaluations: the series of evaluations IS the spec (1:30 → 1:50, 20 min)

E1. The thesis (5 min).

In traditional software, the spec says what the code must do, and the tests verify it. With LLMs, behavior is non-deterministic and non-specifiable by code : the only operational definition of “it works” is the evaluation suite . Hence the thesis:

“The suite of tests IS the spec. » If a behavior isn't covered by an evaluation, you don't know if it works. “It looked good in the demo” = you tested 5 cases chosen by the person who wanted it to work.

E2. The 4-story evaluation stack (10 min).

  1. Assertions (code, free, instant): does JSON parse? Are the required fields there? Is the amount > 0? → catches the coarse, rotates on each commit.
  2. Golden set (50 to 500 examples ⚠ usual order of magnitude): expected input → output pairs, validated by a human. Deliberately includes borderline cases and the adverse cases . We measure a success rate; any modification of prompt or model reverts the golden set.
  3. LLM Judge (LLM-as-judge) : a second model rates the outputs according to a grid (fidelity, completeness, tone). Scalable, but the judge has his own biases → it is necessary calibrate the judge against human judgments before trusting him.
  4. Human evaluation : regular sampling in production, review of escalated cases. Expensive, slow, but it’s the ground truth that recalibrates everything else.

E3. Metrics according to system type (5 min).

Demo: “Evals Dashboard” tab of the interactive page. Run the v1 (naive) prompt against the test set: edge cases fail. Switch to the v3 (hardened) prompt: the rate increases. The message: we don't compare impressions, we compare success rates on the same test set.

Block F — Lab: the indestructible expense report extractor (1:50 → 2:00, 10 min session + continuation of exercises)

Start exercise 3 (lab). In session: presentation of the trapped test game (amounts in letters, injection, empty string) and first round of iteration. The rest is done independently or at the start of the next session.

Closing (2 mins). The 3 sentences to remember:

  1. The diagram guarantees the form, never the substance.
  2. Temperature 0 = reproducible, not correct.
  3. No review = no spec = no product.

Distribute the quiz (or link) + exit tickets.

4. Teaching notes & common pitfalls

5. Exit tickets (5 questions, 5 min at the end of the session)

Each participant responds in writing (paper or form):

  1. Explain in one sentence why “the model returns valid JSON 99% of the time” is a problem in production. Expected: at high volume, 1% failure = daily outages; only the schema constraint (function calling) eliminates the error class.

  2. The model returns {"montant": 402} for a receipt of “forty-two euros”. Syntactic or semantic? What parade? Expected: semantics; business validation / cross-checking + retry with specific feedback, human escalation if repeated failure.

  3. True or false: “At temperature 0, the model no longer makes factual errors. » Justify. Expected: False. T = 0 makes the error reproducible, not absent. Temperature controls variability, not accuracy.

  4. List the 4 levels of the evaluation stack, in ascending cost order. Expected: assertions → golden set → LLM judge → human evaluation.

  5. Your RAG assistant responds fluently but invents references. Which metric to monitor, and by which stage of the battery to measure it continuously? Expected: faithfulness; calibrated LLM judge (with periodic human sampling).

6. Links with the rest of the program