Quiz — Session 3 (Intermediate level)

Structured output, temperature & evaluations

Program : Applied AI — Yann Isola 10 questions — only one correct answer per question, unless otherwise noted. Suggested scale: 1 point per question. Validation threshold: 7/10.

Q1. In production, your pipeline processes 20,000 documents/day. The model returns syntactically valid JSON (JavaScript Object Notation) “99.5% of the time”. What is the correct reading?

HAS It’s excellent: 99.5% is a very good score, the pipeline is reliable. B That's around 100 parsing failures per day ⚠: we must structurally eliminate this class of error, not accept it. C Just add “reply only in JSON” to the prompt to reach 100%. D The temperature must be lowered to 0 to guarantee always valid JSON.

Q2. What mechanism makes the error syntactic structurally impossible in the model output?

HAS An automatic retry as long as the JSON is not parsed. B A regex that extracts the JSON block from the response. C Function calling / tool use: decoding is constrained by a declared JSON schema. D An example of well-formed JSON in the prompt (few-shot).

Q3. The receipt says “forty-two euros”. The model returns `{"montant": 402, "devise": "EUR"}` — JSON perfectly conforms to the schema. What type of error is it, and who can detect it?

HAS Syntactic — the schema validator will reject it. B Semantic — the schema validator lets it pass; you need business rules, cross-checking, a judge or a human. C Syntactic — function calling would have prevented it. D This is not an error: 402 is a valid number.

Q4. In the “validation → retry with feedback” pattern, what element is essential for the retry to have a chance of converging?

HAS Increase the temperature with each attempt to explore other answers. B Resending the exact same prompt, the model will eventually get it right. C Include feedback specific describing the error detected (e.g. “amount 402 does not appear anywhere in the document”), with a ceiling on attempts and human escalation. D Delete the diagram to let the model express itself freely.

Q5. Which of these statements about temperature 0 is correct?

HAS It ensures that the model no longer makes factual errors. B It makes the outputs (quasi) reproducible: if the most probable token is false, the error will be reproduced identically. C It forces the model to respond more slowly but more accurately. D It only has an effect on JSON output, not free text.

Q6. For a structured extraction task in production, which temperature setting is best?

HAS 1.2 — for the model to find creative solutions to difficult documents. B 0 to 0.2 ⚠ (usual benchmark) — variability brings nothing to an extraction task, reproducibility facilitates debugging and evaluations. C 0.7 — the recommended “golden mean” for all uses. D Temperature has no impact on extraction tasks.

Q7. What does the thesis “the sequence of evaluations IS the spec” mean?

HAS The specification must be written before the evaluations, as in a V-cycle. B Since the behavior of an LLM (Large Language Model) system is not specifiable by code, the only operational definition of “it works” is the set of evaluation cases that pass. C Evals replace user documentation. D A product with 100% successful evaluations no longer needs human supervision.

Q8. Order the 4 stages of the evaluation stack from least expensive/fastest to most expensive/slowest.

HAS LLM judge → assertions → golden set → human evaluation. B Golden set → LLM judge → assertions → human evaluation. C Assertions (code) → golden set (50–500 examples ⚠) → LLM judge → human evaluation. D Human evaluation → LLM judge → golden set → assertions.

Q9. What precaution is essential before trusting an LLM judge (LLM-as-judge)?

HAS Use the same model as generator and judge, for consistency. B Calibrate it against human judgments on a sample, measure the agreement, and recalibrate periodically. C Give it a high temperature so that it is more nuanced. D None: an LLM is by nature more objective than a human.

Q10. After a prompt update, your assistant's human escalation rate drops from 18% to 4% ⚠. What is the professional reaction?

HAS Celebrate: fewer escalations = more autonomous system, this is mechanically an improvement. B Return immediately to the old prompt: any sudden variation is a bug. C Suspend judgment and audit a sample of cases that are no longer escalated: does the model really resolve better, or does it respond with confidence on cases that it misses? D Remove the escalation metric, which became useless at 4%.