# Quiz — Advanced Level, Session 7 # “Prompt advanced engineering” **Program:** Applied AI — Yann Isola **Format:** 10 MCQs (MCQ = Multiple Choice Questionnaire), certification level *Claude Certified Architect* **Duration:** 20 minutes — only one correct answer per question unless otherwise stated **Threshold:** 7/10 > ⚠ The API behaviors described (prefilling, extended thinking, stop_sequences) reflect the documentation at the time of writing and are volatile: check the official Anthropic documentation. --- ### Q1 — When the Chain of Thought is night A team adds "Think step by step" to a prompt that **copies verbatim** an IBAN number (IBAN = International Bank Account Number) from a document. Accuracy drops. What is the most likely explanation? - A. CoT increases the effective temperature of the model - B. The model “over-reasons” and paraphrases instead of copying; the CoT is counterproductive on verbatim extraction - C. The CoT has no possible effect on accuracy, the problem comes from elsewhere - D. The CoT exceeds the budget `max_tokens`, systematically truncating the IBAN **Answer: B.** The CoT helps with multi-step reasoning tasks; on a verbatim copy, the reasoning stage introduces reformulations and errors. A is false: CoT does not modify the temperature. D is possible but not “most likely” without indication of truncation. Architect's reflex: the CoT is measured, it cannot be assumed. --- ### Q2 — Design of a few-shot game You have a budget of 6 few-shot examples for a classification. What composition maximizes reliability? - A. 6 examples of the most common case in production, to stick to real distribution - B. 6 examples each covering a different class, all perfectly nominal - C. 1-2 nominal cases, 2-3 ambiguous borderline cases, 1 out-of-scope/refusal case, varying the surface attributes - D. The maximum number of examples possible: reliability increases linearly with their number **Answer: C.** The model already handles nominal cases well; it is on the borderline cases that he needs to be guided, and the case of refusal defines the boundary of the perimeter. Varying the surface attributes avoids spurious biases (e.g. “transfer → P1”). Overlearns the easy case; B ignores ambiguities; D is false — redundancy adds nothing and inflates the cost of prompt. --- ### Q3 — Prefilling: exact API behavior You send `{"role": "assistant", "content": ""}` as the last message with `stop_sequences: [""]`. The template generates `42…`. What does `response.content` contain and what is `stop_reason` worth? - A. `42` — `stop_reason: "end_turn"` - B. `42` — `stop_reason: "stop_sequence"` - C. `42` — `stop_reason: "stop_sequence"` - D. `42` — `stop_reason: "stop_sequence"` **Answer: B.** The prefill text **is not repeated** in the output (the response starts after it), and the triggered stop_sequence **is not included** in the returned text; `stop_reason` is `"stop_sequence"` and the `stop_sequence` field in the response indicates which one triggered. This is the canonical extraction pattern: the output contains only the useful value. --- ### Q4 — Prefill trap Which of these prefills causes an **API error**? - A.`{"role": "assistant", "content": "{"}` -B.`{"role": "assistant", "content": "Voici l'analyse : "}` - C.`{"role": "assistant", "content": "[ANALYSTE]"}` - D.`{"role": "assistant", "content": ""}` **Answer: B.** A prefill cannot end with a trailing whitespace: the API rejects the request. The other three are valid. Recurring certification pitfall — and a sneaky bug in production, because the space is invisible on proofreading. --- ### Q5 — Long context: placement of instructions Prompt of 80,000 tokens: three contracts attached and a critical analysis instruction. What placement maximizes compliance with the instructions? - A. Instructions at the beginning only — the model reads in order - B. Instructions in the middle, between the documents, as close as possible to the data - C. Documents tagged in XML at the beginning; critical instruction stated at the beginning **and** repeated at the end - D. It doesn't matter: the attention is uniform across the entire context window **Answer: C.** Attention favors the beginning (primacy) and end (recency) of the prompt; the middle is the weak zone (“lost in the middle”). Placing large documents at the top (bonus: maximizes prompt caching hits on stable blocks) and doubling the start + end critical setpoint is the recommended practice. B places the setpoint in the weakest zone; D is factually false. --- ### Q6 — Examples vs instructions A system prompt requires "reply in JSON only", but two of the five few-shot examples contain an explanation sentence before the JSON. In production, the model often adds a phrase before its JSON. What diagnosis? - A. Model bug: system instructions always take precedence over examples - B. Temperature too high: lowering it to 0 will be enough - C. Examples/instructions conflict: the model imitates the examples, which often override the opposite instruction — correct the examples - D. You must add “VERY IMPORTANT” in front of the JSON instruction **Answer: C.** Examples show, instructions describe — and in cases of conflict, demonstration frequently trumps description. First reflex when auditing a faulty prompt: check the consistency of examples/instructions. A is false; B will reduce the variance but not the bias learned from the examples; D is the anti-patron of “incantation”. --- ### Q7 — Canonical production pattern In pattern `classification → routage → génération spécialisée → validation`, which statement is correct? - A. Routing should be handled by a dedicated LLM call for flexibility - B. The routing is deterministic code (switch on the structured output of the classification); you don't pay for a model for a `if` - C. Validation must always be a judge LLM, never programmatic rules - D. Classification and generation must use the same model to guarantee consistency **Answer: B.** Classification produces structured output; the routing that consumes it is a simple software connection — deterministic, free, testable. This is false: validation combines programmatic rules (JSON schema, regex) for the mechanically verifiable and LLM judges for the qualitative (tone, contractual commitment). D is false: adapting the model by link (quick to classify, capable of generating) is precisely an advantage of chaining. ⚠ Volatile model names. --- ### Q8 — Negative promptingWhy is “If information is missing, use the value “unknown”” more reliable than “Never make up information” alone? - A. The models do not include French grammatical negation - B. The negative statement describes a void without defining the replacement behavior; the positive statement specifies exactly what to do in the problem case - C. Negative instructions are ignored by the system prompt, they only work as a user message - D. No measurable difference: it is a stylistic preference **Answer: B.** Negative prompting is not useless, but it is considered **less reliable** than the positive equivalent: prohibiting without proposing an alternative leaves the model to choose the fallback behavior itself. Practical rule: every “don’t do X” is accompanied by a “do Y instead”. A and C are false; D contradicts the evaluation observations. --- ### Q9 — Temperature and Chain of Thought For an extraction pipeline with structured CoT, what temperature setting and why? And in which case would we voluntarily do the opposite? - A. High temperature always: the CoT needs creativity to explore - B. Low temperature (often 0) for reproducible reasoning paths; the opposite — sampling several CoTs at higher temperature then majority vote (self-consistency) — is reserved for critical decisions because it is costly - C. Temperature has no effect when a CoT is present in the prompt - D. Temperature 0 makes the CoT useless since the model becomes deterministic **Answer: B.** The temperature also diversifies the reasoning paths: in extraction/classification we want it low for reproducibility. Self-consistency deliberately exploits this diversity (N samples, vote), at the cost of N times the cost. C and D confuse sampling determinism and utility of the reasoning space. Classic certification trap. --- ### Q10 — The “contract” system prompt A prompt production system for a ticket sorting agent should, at a minimum, contain four clauses. Which ones? - A. Detailed persona, company history, sample tickets, signature - B. Explicit rules (scope, sources), exact output format, refusal policy with imposed wording, triggers for escalation towards humans - C. Temperature, max_tokens, stop_sequences, model name - D. Positive instructions only — any mention of refusal degrades performance **Answer: B.** The “system prompt as contract” specifies behavior like specifying an API: what the component does (rules), what it produces (format), what it refuses (with the exact wording — an impromptu refusal is a legal and brand risk), and when it hands over (escalation). C confuses prompt and API parameters. D is false: the explicit refusal policy is precisely a mandatory clause in the contract. --- ## Quick correction grid | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | |----|----|----|----|----|----|----|----|----|-----| | B | C | B | B | C | C | B | B | B | B |**Analysis of common errors:** - Q3/Q4 failed together → redo the prefill + stop_sequences demo (block 3 of the guide); - Q1/Q9 missed → review “when the CoT helps / when it harms” and the temperature interaction (blocks 1 and 8); - Q6/Q8 failed → review the hierarchy of examples/instructions and positive > negative (blocks 2 and 6).