Français

Slides — Session 8: Infrastructure & Deployment

Program: Applied AI — Intermediate Level — Instructor: Yann Isola
Format: 33 slides. Each slide includes the projected content then the speaker notes.
Palette: #1A2230 ink, #0F7A6C teal, #B4612A copper, #E9F6F3 light teal, #F4F7F6 background.

Slide 1 — Title

Infrastructure & Deployment

From notebook to 10,000 users

Applied AI — Session 8 — Yann Isola

Slide 2 — Quick reminder: Session 7

  • You know how to build AI systems that read (RAG — Retrieval-Augmented Generation) and act (tools)
  • You know how to evaluate the quality of their answers
  • One thing is missing: making them run for real, for real users

Slide 3 — Session Objectives

At the end of these 2 hours, you will know:

  1. Name the 6 projects between notebook and production
  2. Integrate an API properly: keys, OAuth, rate limits, retry with backoff
  3. Think about latency budget and explain streaming (SSE)
  4. Reduce the bill: tokens, caching, model routing
  5. Set up observability and alerts; treat prompts as code
  6. Choose a deployment pattern and secure production

Slide 4 — Here is SupportBot 📓

  • Internal support assistant: RAG + tools, answers employee questions
  • In the developer's notebook: it works perfectly
  • 1 user · clear API key · no error handling · invisible cost · ignored latency
  • Tomorrow: 5,000 employees, 10,000 requests/day

Slide 5 — Notebook vs. Production

📓 Notebook 🏭 Production
Users 1 (you) 10,000/day
API Key clearly in the code secrets manager
Network error we restart the cell automatic retry
Cost invisible scrutinized by management
Latency “we don’t care” abort after 5 s
“It doesn’t work” you debug live we need newspapers

Slide 6 — The 6 production sites

  1. 🔑 API integration — auth, rate limits, retry
  2. ⏱️ Latency — budget, streaming
  3. 💸 Costs — tokens, caching, routing
  4. 🔭 Observability — logs, percentiles, alerts
  5. 🛡️ Security — secrets, entrances, exits
  6. 📈 Scaling — scaling, queues

Slide 7 — Project 1: talking to the supplier's API

  • Your application calls the API (Application Programming Interface) of an LLM (Large Language Model) provider
  • Three questions to resolve:
  1. Who has the right to call? → authentication
  2. At what rate? → rate limits
  3. And when it fails? → retry & backoff

Slide 8 — API keys: the company badge

  • An API key = a service password
  • Golden rules:
  • ❌ never in the source code (nor in Git)
  • ❌ never client side — browser, mobile app
  • ✅ environment variables or secrets manager
  • ✅ rotation: revocable without breaking everything

Slide 9 — OAuth: the nominative visitor badge

  • OAuth (Open Authorization, delegated authorization protocol)
  • When the application acts on behalf of a user: their calendar, their CRM (Customer Relationship Management), their files
  • The user explicitly authorizes → the application receives a limited, expirable, revocable token
  • API key = company badge · OAuth = nominative visitor badge with expiration date

Slide 10 — Rate limits: the supplier queue

  • Suppliers limit the flow:
  • RPM (Requests Per Minute, requests/minute)
  • TPM (Tokens Per Minute, tokens/minute)
  • ⚠ Values depend on provider and account level
  • Overflow → response HTTP 429 “Too Many Requests” + often a retry-after header
  • The 429 is not a breakdown: it is an instruction — slow down

Slide 11 — Retry & exponential backoff

Attempt 1 → failure (429/5xx)
   attendre 1 s (+ jitter)
Attempt 2 → failure
   attendre 2 s (+ jitter)
Attempt 3 → failure
   attendre 4 s (+ jitter)
Attempt 4 → failure → CLEAN abort (clear message to the user)
  • Jitter = random variation → prevents 1,000 clients from trying again at the same time
  • ❌ Never retry 400/401/403: an invalid request remains invalid

Slide 12 — Project 2: the latency budget

Starting from the user, not from the technique:

Waiting Perception Technical response
< 1 sec instant nothing to do
1–3 sec acceptable wait indicator
3–10 sec frustrating streaming required
> 10 sec unacceptable live switch to asynchronous

Slide 13 — Where does time go?

Latence totale =
   outbound network (~50–100 ms)
 + TTFT (Time To First Token, delay before the 1st token: ~0.2–1 s ⚠)
 + generation (proportional to OUTPUT tokens)
 + return network
  • The dominant position: the generation of output tokens
  • Consequence: a response 2× shorter ≈ 2× faster

Slide 14 — Streaming: SSE changes everything (except latency)

  • SSE (Server-Sent Events, events sent by the server): the HTTP connection remains open, the server pushes the tokens as the generation progresses
  • total latency: almost unchanged
  • Perceived latency: collapsed — first word < 1 s
  • This is why all AI chat interfaces stream

Slide 15 — Project 3: the invoice is counted in tokens

Cost = input_tokens × input_price + output_tokens × output_price
  • 1 token ≈ ¾ word in English; in French ≈ 1.3–2 tokens/word ⚠
  • The exit costs 3 to 5 times more expensive than the entry ⚠
  • Orders of magnitude ⚠ (July 2026): ~0.1–1 $/M input tokens (small models) to ~2–15 $/M (large) — check today's grid

Slide 16 — Lever 1: prompt caching

  • The start of the prompt is often identical for each request: system prompt, documents, tool definitions
  • The provider puts it in cache → hidden tokens charged at a fraction of the price
  • Up to 90% savings ⚠ on the hidden part
  • Golden rule: stable at the beginning, variable at the end

Slide 17 — Lever 2: model routing

  • Not all requests are equal:
  • “What are the support hours?” → small model
  • “Analyze this 40-page contract” → large model
  • A router (simple rules or small classifier) directs each request to the cheapest model capable of processing it
  • In practice: 70–90% ⚠ of a support assistant's traffic falls under the small model

Slide 18 — Lever 3: budgetary ceilings and alerts

  • Monthly token budget defined in advance
  • Alerts at 50% / 80% / 100% of the budget
  • At 100%: cutoff or graceful degradation (switches to the small model)

“A surprise bill of €20,000 kills more AI projects than a bug.”

Slide 18a — Lever 4: quantization — JPEG of models

One model = billions of weights (numbers). How many bits per weight?

  • FP16 (16 bits): original precision
  • INT8 / INT4: we reduce the bits → we reduce the size and we accelerate

Quantization is the JPEG of models: a little less quality, a lot less weight.

Slide 18b — Compromise in numbers

Accuracy Bits Model 70B ≈ Quality
FP16 16 140 GB reference
INT8 (Q8) 8 70 GB almost identical
INT4 (Q4) 4 ~40 GB −1 to −3 pts ⚠

Q4 = ~4× lighter than FP16, for minimal loss → a 70B fits on consumer hardware.

Slide 18c — In practice: GGUF + Ollama, and WHEN to use it

ollama run llama3:8b-instruct-q4_K_M "Bonjour !"
# 70B : ollama pull <modele>:70b-q4_K_M  # ~40 Go, GGUF
  • Supplier API (default) → quantization does not concern you
  • Self-hosting (sovereignty, sensitive data, large volume) → Q4 by default, upgrade if necessary

Slide 19 — Project 4: observability — log EVERYTHING

For each request, record:

  • timestamp · request id · prompt version
  • model used · input/output tokens
  • latency · status (success/error)

⚖️ GDPR (General Data Protection Regulation): defined retention period, pseudonymization if personal data

Slide 20 — The average lies: think percentiles

  • p50 (median): typical experience
  • p95: the experience of 1 user in 20
  • p99: the worst cases — those who complain and leave

Example: average 1.2 s… and p99 at 9 s 😱

SLA (Service Level Agreement) are written in percentiles: “p95 < 3 s” — never “average < 2 s”

Slide 21 — Alerts: each alarm has instructions for use

Format of an alert rule:

IF error rate > 2 % WHILE 5 min THEN on-call notified DONE check provider status, enable degraded mode

The 3 basic alerts:

  1. Error rate (threshold on sliding window)
  2. p95 latency (SLA)
  3. Token budget (50/80/100%)

Slide 22 — Prompts are code

  • A word changed in a prompt can break 15% of responses
  • So the prompts are:
  • versioned (Git, the version management tool)
  • relus (peer review, like a PR — Pull Request, merge request)
  • tested: the evaluation suite runs at each PR → prompt regression tests
  • This is CI/CD (Continuous Integration / Continuous Deployment) applied to AI

Slide 23 — Project 5 & 6: deploy, secure, collect

Three final questions:

  1. Where to run the backend? → deployment patterns
  2. How to protect it? → security
    3.How to collect the charge? → scaling & queues

Slide 24 — Three deployment patterns

☁️Serverless 📦 Container 🖥️ Dedicated GPU
Examples Lambda, Cloud Functions Docker (+ Kubernetes) cloud/on-prem GPU instance
Server management zero average total
Billing at execution to the instance fixed, high
Scaling up automatic to configure manual
Limits max duration, cold start required skills cost + expertise
Ideal for irregular traffic, small team sustained and regular traffic self-hosting of an open-weights model

Slide 25 — Safety: the 3 non-negotiable rules

  1. 🔑 Never client-side API key → always an intermediate backend, even tiny
  2. 🧹 Sanitize inputs (input sanitization): max length, prompt injection detection
  3. 🚿 Filter output (output filtering): sensitive data, inappropriate content — before display

Slide 26 — Ramp up: horizontal + queue

  • Horizontal: several identical instances behind a load balancer — rather than a giant machine
  • For massive asynchronous: architecture with queue (queue)
Producers → [ Queue ] → Workers (at the rate limit's pace) → Results

Example: 50,000 documents/night, rate limit 500 req/min ⚠ → the queue absorbs, the workers smooth, everything is ready in the morning

Slide 27 — SupportBot architecture in production

Employee → API Gateway → Backend (serverless)
              │                 │  ├── cache de prompt (fournisseur)
              │                 │  ├── small/large model router
              │                 │  └── retry + backoff
              │                 ▼
              │          Fournisseur LLM (streaming SSE)
              └── Journalisation → Dashboard + alertes

10,000 req/day · p95 < 3 sec · ~$1,100/month ⚠ · zero keys exposed

Slide 28 — What to remember

  1. Production = the invisible 80% of work
  2. 429 → exponential backoff + jitter; never retry on 400/401/403
  3. Streaming SSE: perceived latency collapses, not total latency
  4. Costs: caching (stable at the beginning) + routing (good model, good task) + ceilings
  5. Self-hosting: quantization Q4/GGUF to hold a large model locally
  6. Observability: comprehensive logs, percentiles, actionable alerts
  7. Prompts are code: versioned, proofread, tested in CI/CD
  8. API key never on the client side

Slide 29 — Quiz & Exit Tickets

  • 📝 Quiz: 12 multiple choice questions — 6 minutes
  • 🎫 Exit ticket: 1 question, 1 minute, before leaving

Slide 30 — Next: Session 9

You know how to deploy.

But how do you know if what you have deployed is good?

Session 9: advanced evaluation & continuous improvement
— production evaluations, user feedback, improvement loops

Speaker notes: Welcome. Opening question: “Who has ever seen a great AI prototype… that never came out of the notebook?” Hands are raised. “Today, we are learning to cross this divide.” Zero code to write: we think like an architect.

Speaker notes: 90 seconds max. Transition: “A brilliant system that no one can use is worth zero.”

Speaker notes: Clear contract. Specify the common thread: “We will follow a single application, SupportBot, from notebook to production. Each concept solves a problem she encountered growing up.”

Speaker Notes: Set the scene in 60 seconds. Emphasize: nothing is “wrong” in the prototype — it is just designed for a world that does not exist (one user, perfect network, infinite budget).

Speaker notes: Food-truck vs restaurant analogy: same recipe, but the restaurant needs reservations, a sized kitchen, accounting, hygiene and a chef who sees his room. The recipe (the prompt) is the easy part.

Speaker Notes: This is the session outline. Question to the room: “Which is most often forgotten?” Answer: observability — invisible as long as everything goes well, essential from the first incident. Central message to hammer home: “a prototype that works = 20% of the work”.

Speaker Notes: Reframe: In 90% of projects, you are NOT hosting the model — you are calling an API. The entire reliability of your product therefore depends on the quality of this integration.

Speaker Notes: Mental demonstration: “Open F12 in your browser, Network tab. Everything the browser sends, YOU see. A key in the front-end is de facto public — it will be stolen and used at your expense.” This is the #1 safety rule of the session, it will come back on slide 25.

Speaker notes: Do not go into detail about the protocol (flows, scopes) — remember the WHEN: global service identity → API key; action on behalf of a specific user → OAuth. SupportBot example: consult the leave balance of *this* employee → OAuth to the HRIS (HR information system).

Speaker notes: SupportBot anecdote: Monday 9 a.m., 800 employees connect at the same time → burst of 429. Distinguish: rate limit = voluntary, predictable, negotiable limitation (quota increase); breakdown = unintentional. Both are managed by retry, but the rate limit is also prevented (smoothing, queue).

Speaker notes: Diagram to reproduce on the board if necessary. Explain the “thundering herd” effect: without jitter, all clients try again at the same second and recreate the peak. With backoff + jitter, the Monday 9 a.m. experience goes from “30% errors” to “2–3 s wait for the less lucky”.

Speaker Notes: “How long can the user wait?” is a product question, not a technical question. Set the budget BEFORE choosing model and architecture. A chat user doesn't wait like a nightly batch service.

Speaker notes: Counter-intuitive insight: controlling the length of responses (concision guideline, `max_tokens`) is an optimization of LATENCY as well as cost. Participants often believe that latency comes from the network — no, it comes from generation.

Speaker notes: Rhetorical question: “Do you prefer 8 seconds of frozen screen, or a first word at 0.8 s and the rest scrolling?” Mention the hidden cost: error management during the flow and more delicate output filtering (what do we filter, if the response arrives in pieces?). Demo: Monitoring tab of the web page, TTFT line.

Speaker notes: Do not teach prices (they change every quarter), teach METHOD and the reflex “I check the price list before projecting a budget”. Transition: “Let’s look at the three levers to reduce this bill.” Exercise 1 just brought it to life — rely on their figures.

Speaker Notes: Figures from Exercise 1: SupportBot goes from $9,000 to ~$4,100/month just with cache. Classic trap: insert a dynamic timestamp at the top of the system prompt → cache invalidated on each request, zero savings. The cache is managed by the provider, but it is YOUR prompt structure that makes it possible.

Speaker notes: This is often THE biggest lever: in Exercise 1, 80/20 routing increases the bill from ~4,100 to ~$1,100/month (−88% vs. naive). The risk: misdirecting a difficult question to the small model → planning an escalation (the small model can say “I’m handing over”).

Speaker Notes: Generic true story: the agent loop running all night, or the test script running on the wrong loop. Without a ceiling, we discover it on the invoice. With ceiling + alert, we discover it at 50%. Graceful degradation is better than dead cut: the service continues, in economical mode.

Speaker Notes: Connect to Session 1: Inference is limited by **memory bandwidth**. Fewer bits to reread per token = faster AND lighter. The JPEG analogy is the key: everyone has already compressed a photo.

Speaker notes: Bring the figure to life: 140 GB = impossible on a 24 GB card. 40 GB = two 24 GB cards, or a unified memory Mac. Slider: beyond INT4 (e.g. 2 bits), the quality visibly drops — the model becomes “fuzzy”.

Speaker notes: `q4_K_M` = 4 bits, “K” blocks, medium size. Emphasize: quantization does not make a small model as good as a large one — it makes a **large model executable** locally. A 70B-Q4 > an 8B-FP16 with a comparable footprint. Transition to observability.

Speaker notes: Scenario: a user says “yesterday, SupportBot answered me nonsense”. Without log: impossible to diagnose. With log: you find the request, the exact prompt, the version, the model — and you reproduce. The “prompt version” in the newspaper prepares slide 22.

Speaker notes: Web demo: the dashboard displays p50/p95/p99 live — show the p95 dropping while the average remains reasonable. Shocking sentence: “Your users do not live average lives; everyone lives HIS request.”

Speaker notes: Absolute rule: an alert without documented action is noise — it will be ignored from the 3rd time, including when it is serious. If no one knows what to do when it rings: delete it or write down the instructions.

Speaker Notes: Bridge to previous sessions: “You learned how to construct evals. Here, we plug them into the tap: they turn automatically with each change.” AI specificity: non-deterministic outputs → test criteria (format, presence of key information, judge's score), not strict equality. Without that, regressions are discovered in production, via users.

Speaker Notes: Quick transition — Exercise 2 has already manipulated these choices. This part consolidates and names the patterns.

Speaker Notes: Key Point: Dedicated GPU (Graphics Processing Unit) is ONLY necessary if you are hosting the model yourself — data sovereignty or extreme volume. If you call an API, the GPU is with the provider. 90% of projects: serverless or container is enough. Cold start: first request after slower inactivity — acceptable or not depending on the latency budget (loop with slide 12).

Speaker notes: Rule 1 loops with slide 8 — this is intentional, spaced repetition. Rule 2: session callback on prompt injection — in production, the attacker really exists. Rule 3: the model can regurgitate data from the context (e.g. data from another client if the RAG is poorly partitioned) — last trickle before the user's screen.

Speaker notes: The queue reconciles two incompatible rhythms: the massive arrival of tasks and the throughput authorized by the supplier. Bonus: free resilience — if the supplier falls for 10 minutes, the line grows then empties, nothing is lost. This was scenario B of Exercise 2.

Speaker notes: Summary slide — the loop is closed: each component responds to a problem seen in the session. Have the room name each brick: “What is the gateway for? The router? Why is the newspaper plugged in everywhere?” 2–3 minutes of active recapitulation.

Speaker Notes: Read slowly. Each line corresponds to a project on slide 6. If a participant only remembers one sentence: “production is a discipline, not a detail”.

Speaker notes: Distribute the 5 exit tickets alternately in the room. Take note of them: they calibrate the opening of Session 9.

Speaker notes: Teaser in 30 seconds. Remember the trick question from Exercise 3 (the dashboard does not see the quality): “This is exactly the hole that Session 9 will fill.” Thank you, remain available for individual questions.