# Quiz — Session 3 (Intermediate level) # Structured outing, temperature & assessments **Program:** Applied AI — Yann Isola **10 questions — only one correct answer per question, unless otherwise noted.** **Suggested scale: 1 point per question. Validation threshold: 7/10.** --- ### Q1. In production, your pipeline processes 20,000 documents/day. The model returns syntactically valid JSON (JavaScript Object Notation) “99.5% of the time”. What is the correct reading? - A. It's excellent: 99.5% is a very good score, the pipeline is reliable. - B. That's around 100 parsing failures per day ⚠: we must structurally eliminate this class of error, not accept it. - C. Just add “reply only in JSON” to the prompt to reach 100%. - D. We must lower the temperature to 0 to guarantee always valid JSON. **Answer: B.** At high volume, a small percentage of failures becomes a daily stream of failures. Prompt (C) instructions improve without guaranteeing; temperature (D) controls variability, not syntactic validity. The correct answer is structural constraint (function calling / schema constrained output). --- ### Q2. What mechanism makes the *syntactic* error structurally impossible in the model output? - A. An automatic retry as long as the JSON is not parsed. - B. A regex that extracts the JSON block from the response. - C. Function calling / tool use: decoding is constrained by a declared JSON schema. - D. An example of well-formed JSON in the prompt (few-shot). **Answer: C.** With function calling, the provider constrains the generation: the tokens can only form a document conforming to the schema. A and B are healing patches; D improves the probability without guarantee. --- ### Q3. The receipt says “forty-two euros”. The template returns `{"montant": 402, "devise": "EUR"}` — JSON that conforms perfectly to the schema. What type of error is it, and who can detect it? - A. Syntactic — the schema validator will reject it. - B. Semantics — the schema validator lets it pass; you need business rules, cross-checking, a judge or a human. - C. Syntactic — function calling would have prevented it. - D. This is not an error: 402 is a valid number. **Answer: B.** The diagram guarantees the *form*, never the *content*. A valid JSON can be factually false: this is the definition of a semantic error, invisible to the validator. --- ### Q4. In the “validation → retry with feedback” pattern, what element is essential for the retry to have a chance of converging? - A. Increase the temperature with each attempt to explore other answers. - B. Return the exact same prompt, the model will eventually be correct. - C. Include **specific** feedback describing the error detected (e.g. “amount 402 does not appear anywhere in the document”), with a ceiling of attempts and human escalation. - D. Delete the diagram to let the model express itself freely. **Answer: C.** A blind retry (B) almost never converges; without a ceiling, we pay for infinite loops. Targeted feedback gives the model the information needed to correct. --- ### Q5. Which of these statements about temperature 0 is correct? - A. It guarantees that the model no longer makes factual errors. -B.It makes the outputs (quasi) reproducible: if the most probable token is false, the error will be reproduced identically. - C. It forces the model to respond more slowly but more precisely. - D. It only has an effect on JSON output, not free text. **Answer: B.** Temperature controls the **variability** of sampling, not the **accuracy**. T = 0 produces quasi-deterministic behavior — including perfectly reproducible errors. --- ### Q6. For a structured extraction task in production, which temperature setting is best? - A. 1.2 — for the model to find creative solutions to difficult documents. - B. 0 to 0.2 ⚠ (usual benchmark) — variability brings nothing to an extraction task, reproducibility facilitates debugging and evaluations. - C. 0.7 — the “golden mean” recommended for all uses. - D. Temperature has no impact on extraction tasks. **Answer: B.** Extraction and classification benefit from minimal variability. Creativity (A) is useful for brainstorming, not for reading a receipt. --- ### Q7. What does the thesis “the sequence of evaluations IS the spec” mean? - A. The specification must be written before the evaluations, as in the V cycle. - B. The behavior of an LLM (Large Language Model) system is not specifiable by the code, the only operational definition of “it works” is the set of evaluation cases that pass. - C. Evals replace user documentation. - D. A product with 100% successful evaluations no longer needs human supervision. **Answer: B.** Behavior not covered by an evaluation is unknown behavior. “It looked good in the demo” = 5 cases chosen by the person who wanted it to work. D is false: human evaluation remains the stage that recalibrates everything else. --- ### Q8. Order the 4 stages of the evaluation stack from least expensive/fastest to most expensive/slowest. - A. LLM Judge → assertions → golden set → human evaluation. - B. Golden set → LLM judge → assertions → human evaluation. - C. Assertions (code) → golden set (50–500 examples ⚠) → LLM judge → human evaluation. - D. Human evaluation → LLM judge → golden set → assertions. **Answer: C.** Assertions are deterministic, free code, executed on each change; the golden set measures a success rate on validated cases; the LLM judge notes on the scale (after calibration); the human is the ground truth, expensive and slow. --- ### Q9. What precaution is essential before trusting an LLM judge (LLM-as-judge)? - A. Use the same model as generator and judge, for consistency. - B. Calibrate it against human judgments on a sample, measure the agreement, and recalibrate periodically. - C. Give it a high temperature so that it is more nuanced. - D. None: an LLM is by nature more objective than a human. **Answer: B.** An uncalibrated judge is an automated opinion, not a metric. We measure the judge/human agreement before deploying, then we re-sample regularly. --- ### Q10. After a prompt update, your assistant's human escalation rate drops from 18% to 4% ⚠. What is the professional reaction? - A. Celebrate: fewer escalations = more autonomous system, this is mechanically an improvement. -B.Return immediately to the old prompt: any sudden variation is a bug. - C. Suspend judgment and audit a sample of cases that are no longer escalated: does the model really resolve better, or does it respond with confidence on cases that it misses? - D. Remove the escalation metric, which became useless at 4%. **Answer: C.** An isolated metric that suddenly “improves” is a red flag. The classic risk: the model becomes too confident and sends its errors to clients instead of sending them to humans. Only a human audit on a sample decides. --- ## Quick correction grid | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | |----|----|----|----|----|----|----|----|----|-----| | B | C | B | C | B | B | B | C | B | C |