# Teacher Guide — Session 8: Infrastructure & Deployment **Program:** Applied AI — Intermediate Level **Instructor:** Yann Isola **Duration:** 2 hours (120 minutes) **Module covered:** Module 6 — Infrastructure & Deployment --- ## 1. Session overview ### Educational objectives At the end of this session, each participant should be able to: 1. **Describe the notebook → production divide**: what works “on my laptop” for 1 user does not hold up to 10,000 users — and list the 6 projects that separate the two worlds (authentication, latency, costs, observability, security, scalability). 2. **Properly integrate an API (Application Programming Interface) of LLM (Large Language Model)**: API key management, OAuth (Open Authorization, delegated authorization protocol), rate limits, retry strategies with exponential backoff (wait that doubles with each failure). 3. **Reasoning in latency budget**: how long can the user wait? Explain why streaming via SSE (Server-Sent Events) transforms the perceived experience. 4. **Manage costs**: count tokens, exploit prompt caching (prompt caching — up to 90% savings ⚠), route intelligently (economic model for simple tasks, powerful model for difficult tasks). 5. **Implement observability**: log each request/response, track token consumption, latency percentiles (p50, p95, p99), error rates. 6. **Treat prompts like code**: versioning, testing, review — and integrate a suite of evaluations into CI/CD (Continuous Integration / Continuous Deployment). 7. **Choose a deployment pattern**: serverless (Lambda, Cloud Functions), container (Docker), dedicated GPU (Graphics Processing Unit, graphics processor) instance — and justify the choice according to the use case. 8. **Understand quantization**: reduce the precision of weights (FP32 → FP16 → INT8 → INT4), know when it makes self-hosting possible, and know the quality/cost/material trade-offs. 9. **Secure production**: never an API key on the client side, input sanitization, output filtering. ### Prerequisites - Sessions 1 to 7 followed (notably Session 5 — tool calling, and sessions on prompts and RAG, Retrieval-Augmented Generation, generation augmented by retrieval). - Understand what an HTTP (HyperText Transfer Protocol) request and a JSON (JavaScript Object Notation, data exchange format) response are. - No prior DevOps skills (development/operations fusion) required: this session **is** the DevOps initiation applied to AI. ### Materials needed - Video projector + slides (`slides/slides.md`). - Interactive web page (`webpage/index.html`) — works **offline**: token cost calculator, deployment architecture builder, monitoring dashboard mockup with animated metrics. - Worksheets (`exercises/exercises.md`). - End of session quiz (`quiz/quiz.md`). - Ideally: one laptop for two participants (the cost calculator is the key activity). ### Central message of the session> “A working AI prototype is 20% of the work. The remaining 80% is called production: authentication, latency, costs, observability, security, scalability. No one will ever see that 80% — except when they're missing. » Repeat this idea in several forms throughout the session. A participant who only remembers this leaves with the essential: **production is a discipline, not a detail**. ### Narrative thread The entire session follows **a single common thread application**: “SupportBot”, an internal customer support assistant that answers questions from employees of a 5,000-person company. We follow it from the developer's notebook to a service that receives 10,000 requests/day. Each part of the session solves a problem that SupportBot encountered growing up: - 📓 Part A: SupportBot works in the notebook... and that's it. - 🔑 Part B: SupportBot gets cut off by rate limits. - ⏱️ Part C: Users find SupportBot “slow”. - 💸 Part D: the monthly bill explodes. - 🧊 Part D-bis: SupportBot must run locally for sensitive data. - 🔭 Part E: “It doesn’t work” — but no one knows why. - 🚀 Part F: where and how to deploy, secure, scale up. Narrative continuity is your best educational tool: each concept arrives as the **solution to a problem experienced** by SupportBot, never as an abstract notion. --- ## 2. Unfolded minute by minute | Schedule | Duration | Sequence | Support | |---|---|---|---| | 0:00 – 0:05 | 5 mins | Home, reminder Session 7, objectives, presentation of SupportBot | Slides 1–3 | | 0:05 – 0:18 | 13 mins | **Part A — The notebook → production gap** | Slides 4–6 | | 0:18 – 0:33 | 15 mins | **Part B — API integration: auth, rate limits, retry & backoff** | Slides 7–11 | | 0:33 – 0:45 | 12 mins | **Part C — Latency & streaming budget (SSE)** | Slides 12–14 + web demo (dashboard, latency curve) | | 0:45 – 1:00 | 15 mins | **Exercise 1: Cost calculator** (on the web page) | Worksheet + webpage | | 1:00 – 1:05 | 5 mins | ☕ Short break | — | | 1:05 – 1:18 | 13 mins | **Part D — Cost management: tokens, caching, model routing** | Slides 15–18 + web calculator | | 1:18 – 1:27 | 9 mins | **Part D-bis — Quantization: fitting a large model onto a small machine** | Slides 18a–18c | | 1:27 – 1:34 | 7 mins | **Part D-ter — Sovereignty & corporate architecture** | Palantir diagrams + insurance table | | 1:34 – 1:41 | 7 mins | **Part E — Observability, prompt versioning, CI/CD** | Slides 19–22 + web dashboard mockup | | 1:41 – 1:49 | 8 mins | **Exercise 2: Design a deployment architecture** | Worksheet + web builder | | 1:49 – 1:53 | 4 mins | **Part F — Deployment patterns, security, scalability** | Slides 23–27 | | 1:53 – 2:00 | 7 mins | Quick quiz + Exit Tickets + announcement Session 9 | Slides 28–30 | **Flexibility note:** Exercise 3 (design of a monitoring dashboard) is intended as **homework** or bonus activity. If you fall behind, compress Part F to 5 minutes by projecting only the serverless/container/dedicated GPU comparison table — but **never** sacrifice Part D (costs) or the “never client-side API key” security passage: these are the two most cost-effective protections for your participants. --- ## 3.Detailed teaching notes by sequence ### 0:00 – 0:05 | Reception and framing **What to say:** “Last session, you built systems that reason and act. Today, blunt question: how many of you have ever seen an awesome prototype... that never made it out of the notebook? » (Raised hands guaranteed.) “This session is the instructions for crossing this divide. » Introduce SupportBot in 30 seconds: a RAG assistant + tools that answers internal questions. It works perfectly... for its developer, alone, on a Tuesday afternoon. **Trap to avoid:** Don't let non-developers drop out from the first minute. Explicitly announce: “Zero lines of code to write today. We think like an architect, not a developer. The decisions that we will learn to make are **project and budget management** decisions as well as technical decisions. » --- ### 0:05 – 0:18 | Part A — The notebook → production gap **Major idea:** “it works on my computer” and “it serves 10,000 users” are two different professions. **Unfolded:** 1. Project the difference table (slide 5). Bring it to life with SupportBot: - *Notebook:* 1 user (the developer), API key in plain text in the code, no error management, invisible cost, “who cares” latency. - *Production:* 10,000 requests/day, secrets protected, inevitable network outages, monthly bill scrutinized by management, users who give up after 5 seconds. 2. Introduce the **6 projects** (slide 6): authentication & API integration, latency, costs, observability, security, scalability. Announce that the session is processing them in this order. **Analogy that works well:** the food truck vs the restaurant. The recipe (the prompt, the model) is the same. But the restaurant needs reservations (rate limits), a dimensioned kitchen (scaling), accounting (costs), controlled hygiene (safety) and a chef who knows what is happening in his room (observability). **Question to ask the audience:** “What do you think is the most often forgotten project? » Expected and correct answer: observability — because it is not seen as long as everything is going well. --- ### 0:18 – 0:33 | Part B — API integration: authentication, rate limits, retry **Concepts to cover, in order:** 1. **API Keys** (slide 8): an API key is a service password. Golden rules: - Never in the source code (nor in Git, the version management tool). - Never client side (browser, mobile application) — anyone who opens the browser developer tools can read and steal it. - Always in environment variables or a secrets manager (AWS Secrets Manager, HashiCorp Vault, etc.). - Regular rotation: a key must be able to be revoked without breaking everything. 2. **OAuth** (slide 9): when the application acts **on behalf of a user** (access their Google calendar, their CRM - Customer Relationship Management, customer relationship management), we do not use a global API key but OAuth: the user explicitly authorizes, the application receives a limited and revocable token (access token). Simplify: “API key = company badge; OAuth = nominative visitor badge, with expiration date”. 3.**Rate limits** (slide 10): Providers limit throughput — typically in RPM (Requests Per Minute) and TPM (Tokens Per Minute). ⚠ The exact figures vary depending on the provider and the account level: give an order of magnitude (e.g. a few hundred to a few thousand RPM) and insist on the **reflex**: read the HTTP response `429 Too Many Requests` and the header `retry-after`. 4. **Retry & exponential backoff** (slide 11): when a call fails (error 429 or 5xx), we try again — but not immediately and not indefinitely: - Attempt 1 fails → wait 1 s → attempt 2 → wait 2 s → attempt 3 → wait 4 s → attempt 4 → clean abort. - Add **jitter** (random variation) to prevent 1,000 clients from trying again at exactly the same time (“herd” effect — thundering herd). - **Never** retry 400/401/403 errors (invalid request, missing authentication, forbidden): retrying an invalid request will give the same invalid result. **SupportBot trivia:** “Monday morning, 9:00 a.m.: 800 employees open SupportBot at the same time. Without retry or backoff, 30% of requests fail with a 429 error and users see "Server Error". With backoff + jitter: everyone is served, with a 2–3 second wait for the less fortunate. Same business code, radically different experience. » **Frequent mistake made by participants:** confusing rate limit (voluntary limitation from the supplier) and outage (involuntary unavailability). Both are managed by retry, but the rate limit is also **prevented**: smooth traffic, put in a queue, request a quota increase. --- ### 0:33 – 0:45 | Part C — Latency & streaming budget **Concepts:** 1. **The latency budget** (slide 12): start from the user, not from the technique. Classic UX (User eXperience) benchmarks: - < 1 s: perceived as instantaneous or fluid. - 1–3 s: acceptable with a wait indicator. - 3–10 s: frustrating; requires streaming or an honest progress bar. - \> 10 s: to be processed asynchronously (“we will notify you when it is ready”). 2. **Breaking down latency** (slide 13): forward network (+ ~50–100 ms) + TTFT (Time To First Token, delay before the first token — often 200 ms to 1 s ⚠) + generation (proportional to the number of output tokens) + return network. The dominant position is almost always **the generation of output tokens**. Direct consequence: a response twice as short is ~twice as fast — controlling the output length is a latency optimization. 3. **Streaming via SSE** (slide 14): instead of waiting for the complete response, the server sends the tokens **as the generation progresses**. SSE = Server-Sent Events, a standard HTTP mechanism where the connection remains open and the server pushes events. The total latency hardly changes, but the **perceived latency** collapses: the user sees the first word in less than a second. **Web demo:** open the interactive page, “Monitoring” tab, show the TTFT vs total latency line. Then ask the question: “Which do you prefer: a complete answer in 8 seconds of frozen screen, or a first word at 0.8 s and the rest scrolling? » This is exactly why all AI chat interfaces stream.**Subtle point to mention:** streaming slightly complicates the code (management of partial events, error management during the flow) and output filtering (what do we filter, if the response arrives in pieces?). Nothing insurmountable, but it's not free. --- ### 0:45 – 1:00 | Exercise 1 — Cost calculator (15 min) **Organization:** pairs, one computer per pair, web page “Cost calculator” tab. The worksheet guides you step by step. **Hidden educational objective:** to discover **through manipulation** three truths: 1. Output tokens cost more than input tokens (often 3 to 5× ⚠). 2. Prompt caching radically changes the equation when the system prompt is long and repeated. 3. Model routing (small model for 80% of cases) is often the biggest savings available. **Circulation in the room:** locate the pairs who finished early and give them the bonus challenge (“find the configuration that divides the bill by 10 without changing the volume of requests”). **Restitution (last 3 minutes):** Ask two pairs for their numbers. Write the “naive” invoice vs the “optimized” invoice on the board. The gap (often 5–15×) is the memorable moment of the session. --- ### 1:00 – 1:05 | ☕ Break During the break, let the model of the monitoring dashboard (animated metrics) be projected: it intrigues and prepares Part E. --- ### 1:05 – 1:20 | Part D — Cost management **Concepts, based on what Exercise 1 has just brought to life:** 1. **Count the tokens** (slide 15): a token ≈ ¾ of a word in English, a little less in French (≈ 1.3–2 tokens per word ⚠ depending on the tokenizer). The invoice = entry tokens × entry price + exit tokens × exit price. Orders of magnitude ⚠ (July 2026, to be checked before each session): from ~0.1–1 $ par million de tokens d'entrée pour les petits modèles à ~2–15 $ for large ones; output typically costs 3–5× input. 2. **Prompt caching** (slide 16): if the start of the prompt is identical from one request to another (system prompt, reference documents, tool definitions), the provider can cache it and charge for the cached tokens at a fraction of the price — **up to 90% savings on the hidden part ⚠**. Practical rule: put all stable content **at the beginning** of the prompt, variable content (user question) **at the end**. 3. **Model routing** (slide 17): not all queries are equal. “What are the support hours? does not need the same template as "Analyze this 40 page contract." Architecture: a classifier (or a small model, or simple rules) routes each request to the cheapest model capable of processing it. In practice, 70–90% ⚠ of a support assistant's traffic falls under the business model. 4. **Caps and alerts** (slide 18): monthly token budget, alert at 50%/80%/100%, graceful cutoff or degradation (switch to the small model) in the event of exceeding. “A surprise bill of €20,000 kills more AI projects than a bug. » **Teaching trap:** Don't turn this part into a pricing lesson — prices change quarterly. Teach the **calculation method** and the reflex “I check today’s price list”, not the numbers. --- ### 1:18 – 1:27 | Part D-bis — Quantization: fitting a large model onto a small machine (9 min)> 🎯 **To be inserted between cost management and observability.** This part answers the question that always comes up: “What if I want to host the model myself?” » It wraps up with the message from **Session 1**: **memory bandwidth** (the speed at which we move model weights from memory) is the real bottleneck of inference, not just the number of calculations. **The problem in one sentence.** A model is billions of **weights** (numbers). In standard training precision, each weight occupies **32 bits** (FP32, 32-bit floating point) or **16 bits** (FP16). A model of **70 billion parameters** in FP16 therefore weighs ≈ **140 GB** — impossible to load on a consumer card (24 GB of VRAM), and slow to serve even on data center hardware, because these 140 GB must be reread for each token generated. **What is quantization?** Reduce the **number of bits by weight**. We go from 16 bits to 8 bits (INT8, 8-bit integer), or even 4 bits (INT4). Each time the number of bits is divided by two, we (approximately) halve the size of the model **and** the amount of memory to be reread per token — so we gain both footprint and speed. | Accuracy | Bits/weight | Size of a 70B model ≈ | Analogy | |---|---|---|---| | FP32 | 32 | ≈ 280 GB | Uncompressed RAW photo | | FP16 | 16 | ≈ 140 GB | High definition photo | | INT8 (Q8) | 8 | ≈ 70 GB | JPEG quality 90% | | INT4 (Q4) | 4 | ≈ 40 GB | JPEG quality 70% | **The analogy to hammer home:** quantization, **it's the JPEG of models**. We accept a slight loss of quality (to the eye, often invisible) in exchange for a much smaller file that is quick to handle. As for JPEG, there is a slider: compressing too much (beyond INT4, e.g. 2 bits) visibly degrades the responses (the model becomes “fuzzy”: more errors, less nuance). **The quality/size compromise (key figure to be given):** in practice, **INT8 is almost indistinguishable** from FP16 on most tasks; **INT4 (Q4) loses very little** (often 1–3 points on benchmarks ⚠) for **~4× less memory** than FP16. This is the default setting for general public self-hosting: a 70B in Q4 (≈ 40 GB) fits on two 24 GB cards, or on a Mac with unified memory. **How ​​we do it, concretely.** We don't re-quantify ourselves: we download a model **already quantified**, distributed in **GGUF** format (the file format of quantized models for local execution), via a tool like **Ollama** or **llama.cpp**. Copyable example to show:```bash # Installer Ollama puis récupérer un modèle open-weights quantifié en Q4 ollama run llama3:8b-instruct-q4_K_M "Explique la quantization en 2 phrases." # Pour un 70B open-weights : ollama pull :70b-q4_K_M # ~40 Go ```The suffix `q4_K_M` reads: **4 bits**, variant “K” (block quantization, finer), size “M” (medium, good compromise). You will also see `q5_K_M`, `q8_0`, etc. : the higher the number, the more faithful and heavy it is. **When to use it (decision tree to draw):** - **API from a provider** (default) → you do not manage the quantization, that's its problem. 80% of projects end here. - **Self-hosting** (data sovereignty, data which must not go out, enormous volume, large-scale cost) → quantization becomes **essential**: it is this which makes self-hosting economically viable. Choose **Q4** by default, go up to Q5/Q8 if the measured quality is not enough, rarely go below Q4. **Pedagogical pitfall:** quantization does not make a small model "as good" as a large one — it makes a **large model executable** on modest hardware. A 70B in Q4 remains much better than an 8B in FP16, for a comparable footprint. The real lever remains: **the right model for the right task** (Part D, routing). **Session 1 link to verbalize:** “We said that inference is limited by **memory bandwidth**. Quantization attacks exactly this bottleneck: fewer bits to reread per token = more tokens per second. That's why it accelerates **and** lightens. » --- ### 1:27 – 1:34 | Part D-ter — Sovereignty & enterprise architecture (7 min) **Why add it here.** Quantization answers the question “can I host?” ". Sovereignty answers the following question: “**what must I master so that the company agrees to depend on this AI?**” The right framing comes from Palantir (2026): institutional sovereignty is not limited to a confidentiality contract. It is designed in layers – data, models, calculation, control – and each layer must be chosen according to the real sensitivity of the workload. **1. ZDR is not a marketing slogan.** Zero Data Retention (ZDR) means: the supplier does not retain prompts, responses, attachments, nor ideally usable **metadata** (identifiers, times, volumes, fine traces). It's better than a standard API, but it's not equivalent to an owned environment. Formula to give: “ZDR reduces exposure; it does not give you complete sovereignty. » Zero Data Retention and standard API comparison _Source: Palantir — Institutional Sovereignty in the Age of AI (2026)_ **2. Decide by workload sensitivity.** Draw a simple tree: public data → standard API possible; non-critical internal data → enterprise cloud with ZDR; sensitive/regulated customer data → ZDR strict or confidential computing; industrial secret, defense, critical health → owned or isolated environment. The professional reflex is not “all on-prem” nor “all cloud”: it is **classification by workload**. AI decision tree by workload sensitivity _Source: Palantir — Institutional Sovereignty in the Age of AI (2026)_ **3.Structural assurance is better than contractual insurance.** Present the scale of trust, from the most sovereign to the least sovereign: **owned / air-gapped** (owned, isolated hardware), then **attested TEE** (Trusted Execution Environment, trusted execution environment with cryptographic attestation), then **cloud ZDR**, then **standard API**. A contract promises; an architecture technically limits what can happen. Confidential computing is the modern compromise: if you rent computing, you require a certificate proving that the code and environment executed are those intended. Structural Assurance Scale for AI Computing _Source: Palantir — Institutional Sovereignty in the Age of AI (2026)_ Concretely, the attestation workflow takes place in four stages: encrypted submission of the workload, execution in the TEE, signature of the attestation by the hardware, then verification of the proof before accepting the result. Show the diagram: this is the circuit that replaces contractual trust with cryptographic proof. Hardware attestation workflow: encrypted submission, TEE execution, signing, verification _Source: Palantir — Institutional Sovereignty in the Age of AI (2026)_ **4. Direct link to quantization.** Self-hosting is only viable if the model fits on the hardware you control. Quantization (Q4/Q5/Q8) therefore transforms a sovereignty constraint into a possible architecture: a fine-tuned, quantized open-weights model, served on owned GPU or Mac with unified memory, with internal logs and permissions. It's not always the right choice — fixed cost, expertise, operations — but for certain workloads, it's the only defensible answer. Three-layer sovereign AI architecture _Source: Palantir — Institutional Sovereignty in the Age of AI (2026)_ For participants who want the complete view, show the end-to-end on-prem stack: owned GPUs → self-hosted models → control layer (permissions, logs) → business ontology → users and agents. Each layer answers a different sovereignty question. On-prem architecture stack: from owned hardware to users and agents _Source: Palantir — Institutional Sovereignty in the Age of AI (2026)_ **Summary sentence:** “The higher the business risk, the more you must replace declarative confidence with technical control: zero retention, certified calculation, equipment owned, and ability to change model. » --- ### 1:34 – 1:41 | Part E — Observability, prompt versioning, CI/CD **Concepts:** 1. **Log each request/response** (slide 19): timestamp, request ID, prompt version, model used, input/output tokens, latency, status. Without a log, a user’s first “it’s not working” is undiagnosable. Please note GDPR (General Data Protection Regulation): define the retention period and anonymize/pseudonymize if the prompts contain personal data. 2. **Latency percentiles** (slide 20): the average lies. p50 (median) = typical experience; p95 = the experience of 1 user in 20; p99 = worst cases.A service can average 1.2s and 9s on p99 — and it's the p99 users who complain and leave. SLAs (Service Level Agreement) are written in percentiles: “p95 < 3 s”, never “average < 2 s”. 3. **Error rate and alerts** (slide 21): alert thresholds on the error rate (e.g. > 2% over 5 minutes), p95 latency, token budget. An alert should be **actionable**: if no one knows what to do when it sounds, delete it or document it. 4. **Prompts are code** (slide 22): versioned (Git), reviewed (peer review), tested (evaluation suite). CI/CD for AI: with each PR (Pull Request, code merge request) that modifies a prompt, the suite of evaluations runs automatically — prompt regression tests. “Changing a word in a prompt can break 15% of responses. Without CI evaluations, you discover it in production, via your users. » **Web demo:** dashboard mockup — show the metrics that are moving, the p95 that is dropping, the budget alert that is turning orange. Ask: “What metric would you look at first thing Monday morning?” » **Link with previous sessions:** remember that the suite of evaluations was conceptually built in the sessions devoted to the quality of prompts — here, we **automate** it. --- ### 1:41 – 1:49 | Exercise 2 — Deployment architecture (8 min) **Organization:** same pairs. Specifications in the worksheet (three scenarios to choose from: internal HR chatbot, high-volume document summary API, client-side real-time assistant). The web page architecture builder serves as a visual aid: drag and drop the components (API gateway, LLM provider, cache, queue, database) and read the latency estimate. **What you evaluate while driving:** - Is the API key on the server side (never in the browser)? - Is there an observability (logging) component? - Does the asynchronous scenario use a queue? - Is the serverless/container/GPU choice **justified** (and not just “Docker because it’s cool”)? **Restitution (3 min):** a pair presents their architecture; the challenge room. You play devil's advocate: "What happens if the LLM provider goes down 10 minutes?" » (Expected responses: retry, degraded message, absorbing queue, possibly fallback provider.) --- ### 1:49 – 1:53 | Part F — Deployment patterns, security, scalability This part is deliberately dense and fast - Exercise 2 has already manipulated the concepts. 1. **Three deployment patterns** (slide 24): - **Serverless** (Lambda, Cloud Functions): zero servers to manage, runtime billing, automatic scaling. Limitations: capped execution time, cold start. Ideal for irregular traffic and small teams. - **Container** (Docker, orchestrated for example with Kubernetes): portable, reproducible, fine control. Higher skill cost. Ideal for sustained and regular traffic. - **Dedicated GPU instance**: only if you host the model yourself (open-weights) — data sovereignty, high fixed cost, in-depth expertise. The majority of projects don't need it: the API call to a provider is enough. 2.**Security** (slide 25): the three non-negotiable rules — API key never on the client side (always an intermediate backend), input sanitation (maximum length, prompt injection detection), output filtering (sensitive data, inappropriate content) before display. 3. **Load ramp** (slide 26): horizontal rise (several instances behind a load balancer) rather than vertical (a larger machine). For long or massive processing: **queue architecture** (queue) — the client submits a task, workers process it at the rate limits, the client is notified. This is THE boss for batch processing. **Shortcut if late:** only project the comparative table of the three bosses and the safety slide. The queue has already been seen in Exercise 2. --- ### 1:53 – 2:00 | Quiz, Exit Tickets, closing - Quiz: 12 MCQs (multiple-choice questionnaire), 5–6 minutes, quick collective self-correction if time permits (otherwise corrected online). - Exit tickets (below): 1 minute, one post-it or form per participant. - Teaser Session 9: “You now know how to deploy. Next time: How to know if what you've deployed is **good** — advanced evaluation and continuous improvement. » --- ## 4. The 5 Exit Tickets To be distributed (paper or form) in the last 5 minutes. Each participant answers **one** question (distribute them alternately) or all five for a short version. 1. **“List two differences between a notebook prototype and a production service, and explain why they matter. »** *Expected: two of — secrets management, retry/rate limits, controlled latency, cost tracking, observability, security, scalability — with a one-sentence justification.* 2. **“Your application receives an HTTP 429 error. What does it mean and what should your code do?” »** *Expected: rate limit exceeded (Too Many Requests) → wait then retry with exponential backoff (+ jitter), respecting the `retry-after` header if present. Don't hammer the API.* 3. **“Explain in two sentences why SSE streaming improves user experience without reducing total latency. »** *Expected: tokens are displayed over generation; the first token arrives in < 1 s, so the **perceived** latency drops, even if the complete generation takes the same time.* 4. **“Give two levers to reduce the cost of an LLM application without reducing the volume of requests. »** *Expected: two of — prompt caching (stable content at the start of the prompt), routing to a cheaper model for simple tasks, reduction of output length, reduction of injected context (better RAG).* 5. **“Why do we say that “prompts are code”? Name a concrete practice that results from this. »** *Expected: a change in prompt changes the behavior of the system like a code change → Git versioning, peer review, regression tests (sequence of evaluations) executed in CI/CD for each modification.* 6. **“What is the use of quantization when you self-host a model? »** *Expected: it reduces the number of bits per weight (FP16 → INT8 → INT4), therefore the memory footprint and the bandwidth to be reread per token. Q4 allows for example to hold a 70B around 40 GB, with a limited loss of quality.* **Operation:** read the tickets before Session 9.If more than 30% of responses to ticket 2 confuse 429 and server failure, allow 5 minutes of reminder at the opening of the next session. --- ## 5. Frequently asked questions from participants (and ready answers) **“Why not call the provider's API directly from the browser? It would be simpler. »** Because the API key would be visible to anyone who opens the browser's developer tools (F12 → Network tab). It would be stolen within hours and used at your expense. An intermediate backend — even tiny, even serverless — is **mandatory**. This is the #1 safety rule of the session. **“Prompt caching, do we manage it or the supplier? »** The provider hosts the cache, but **you** structure the prompt to take advantage of it: stable content at the beginning, variable content at the end, and depending on the provider an explicit cache point marker. If your system prompt changes on every request (dynamic timestamp, for example), the cache is useless. **“Do you need a GPU to deploy an AI application? »** No, in the vast majority of cases. If you call a vendor's API (Anthropic, OpenAI, Mistral…), the GPU is at home. You only need GPUs if you're hosting an open-weights model yourself — a choice that is justified by data sovereignty or huge volume, not by default. **“How much does an app like SupportBot really cost? »** ⚠ It depends on the daily rates, but the method is stable: daily volume × (entry tokens × entry price + exit tokens × exit price), then apply caching and routing. Exercise 1 gives the method; redo the calculation with the current price list. Order of magnitude: from a few tens to a few thousand euros/month depending on volume and optimization. **“The average latency of our service is good, why are users complaining? »** Because the mean hides the tail of the distribution. Look at p95 and p99: if p99 is 9 s, 1 request in 100 is unbearable — and these users are noisy. Commit to percentiles, not averages. **“We already have a CI/CD for our code. What's changing with AI? »** Two additions: (1) the evaluation suite runs at each PR that hits a prompt or model parameter — these are the unit tests of the model behavior; (2) the outputs being non-deterministic, the tests relate to criteria (respected format, presence of key information, score of a judge) rather than to strict equality. --- ## 6. Emergency equipment (if the technique fails) - The web page does not open → Exercise 1 can be done **by hand**: the exercise sheet contains the fictitious price list and the formulas. Bring calculators (or telephones). - No video projector → the comparative table of deployment patterns and the retry/backoff diagram are drawn on the whiteboard in 2 minutes each; the templates are in the notes to slides 11 and 24. - Remote session → share the web page upstream (single file, works offline); pairs become 15-minute breakout rooms.