Français
Applied AI · Intermediate 🟡 · Session 8
📝 Teacher's Guide
← Return to program 📄 Source .md

Teacher Guide — Session 8: Infrastructure & Deployment

Program : Applied AI — Intermediate Level Instructor: Yann Isola Duration : 2 hours (120 minutes) Module covered: Module 6 — Infrastructure & Deployment

1. Session overview

Educational objectives

At the end of this session, each participant should be able to:

  1. Describe the notebook → production gap : what works “on my laptop” for 1 user does not hold up to 10,000 users — and list the 6 projects that separate the two worlds (authentication, latency, costs, observability, security, scalability).
  2. Properly integrate an API (Application Programming Interface) of LLM (Large Language Model) : API key management, OAuth (Open Authorization, delegated authorization protocol), rate limits, retry strategies with exponential backoff (wait that doubles with each failure).
  3. Thinking about latency budget : how long can the user wait? Explain why streaming via SSE (Server-Sent Events) transforms the perceived experience.
  4. Manage costs : count tokens, exploit prompt caching (prompt caching — up to 90% savings ⚠), route intelligently (economic model for simple tasks, powerful model for difficult tasks).
  5. Implement observability : log each request/response, track token consumption, latency percentiles (p50, p95, p99), error rates.
  6. Treat prompts like code : versioning, testing, review — and integrating a suite of evaluations into CI/CD (Continuous Integration / Continuous Deployment).
  7. Choose a deployment pattern : serverless (Lambda, Cloud Functions), container (Docker), dedicated GPU (Graphics Processing Unit, graphics processor) instance — and justify the choice according to the use case.
  8. Understanding quantization : reduce the precision of the weights (FP32 → FP16 → INT8 → INT4), know when it makes self-hosting possible, and know the quality/cost/material trade-offs.
  9. Secure production : never an API key on the client side, input sanitization, output filtering.

Prerequisites

Necessary equipment

Central message of the session

“A working AI prototype is 20% of the work. The remaining 80% is called production: authentication, latency, costs, observability, security, scalability. No one will ever see that 80% — except when they're missing. »

Repeat this idea in several forms throughout the session. A participant who only remembers this leaves with the essential: production is a discipline, not a detail .

Narrative thread

The entire session follows a single red thread application : “SupportBot”, an internal customer support assistant that answers questions from employees of a 5,000-person company. We follow it from the developer's notebook to a service that receives 10,000 requests/day. Each part of the session solves a problem that SupportBot encountered growing up:

Narrative continuity is your best teaching tool: each concept arrives as the solution to a problem experienced by SupportBot, never as an abstract notion.

2. Rolled out minute by minute

Hourly Duration Sequence Support
0:00 – 0:05 5 mins Home, reminder Session 7, objectives, presentation of SupportBot Slides 1–3
0:05 – 0:18 13 mins Part A — The notebook → production gap Slides 4–6
0:18 – 0:33 15 mins Part B — API integration: auth, rate limits, retry & backoff Slides 7–11
0:33 – 0:45 12 mins Part C — Latency budget & streaming (SSE) Slides 12–14 + web demo (dashboard, latency curve)
0:45 – 1:00 15 mins Exercise 1: Cost calculator (on the web page) Worksheet + webpage
1:00 – 1:05 5 mins ☕ Short break —
1:05 – 1:18 13 mins Part D — Cost management: tokens, caching, model routing Slides 15–18 + web calculator
1:18 – 1:27 9 mins Part D-bis — Quantization: fitting a large model onto a small machine Slides 18a–18c
1:27 – 1:34 7 mins Part D-ter — Sovereignty & enterprise architecture Palantir diagrams + insurance table
1:34 – 1:41 7 mins Part E — Observability, prompt versioning, CI/CD Slides 19–22 + web dashboard mockup
1:41 – 1:49 8 mins Exercise 2: Design a deployment architecture Worksheet + web builder
1:49 – 1:53 4 mins Part F — Deployment patterns, security, scalability Slides 23–27
1:53 – 2:00 7 mins Quick quiz + Exit Tickets + announcement Session 9 Slides 28–30

Flexibility rating: Exercise 3 (design of a monitoring dashboard) is planned as homework or bonus activity. If you're falling behind, compress Part F to 5 minutes by projecting only the serverless/container/dedicated GPU comparison chart — but don't sacrifice Never Part D (costs) nor the “never client-side API key” security transition: these are the two most cost-effective protections for your participants.

3. Detailed teaching notes by sequence

0:00 – 0:05 | Reception and framing

What to say: “Last session, you built systems that reason and act. Today, brutal question: how many of you have already seen an awesome prototype... that never came out of the notebook? » (Raised hands guaranteed.) “This session is the instructions for crossing this divide. »

Introduce SupportBot in 30 seconds: a RAG assistant + tools that answers internal questions. It works perfectly... for its developer, alone, on a Tuesday afternoon.

Trap to avoid: don't let non-developers off the hook from minute one. Explicitly announce: “Zero lines of code to write today. We think like an architect, not a developer. The decisions that we will learn to make are decisions of project and budget management as much as technical decisions. »

0:05 – 0:18 | Part A — The notebook → production gap

Main idea: “it works on my computer” and “it serves 10,000 users” are two different professions.

Unfolded:

  1. Project the difference table (slide 5). Bring it to life with SupportBot:
    • Notebook: 1 user (the developer), clear API key in the code, no error management, invisible cost, “who cares” latency.
    • Production : 10,000 requests/day, protected secrets, inevitable network outages, monthly bill scrutinized by management, users who give up after 5 seconds.
  2. Introduce them 6 construction sites (slide 6): authentication & API integration, latency, costs, observability, security, scalability. Announce that the session is processing them in this order.

Analogy that works well: the food truck vs the restaurant. The recipe (the prompt, the model) is the same. But the restaurant needs reservations (rate limits), a dimensioned kitchen (scaling), accounting (costs), controlled hygiene (safety) and a chef who knows what is happening in his room (observability).

Question to ask the room: “What do you think is the most often forgotten project? » Expected and correct answer: observability — because it is not seen as long as everything is going well.

0:18 – 0:33 | Part B — API integration: authentication, rate limits, retry

Concepts to cover, in order:

  1. API keys (slide 8): An API key is a service password. Golden rules:
    • Never in the source code (nor in Git, the version management tool).
    • Never on the client side (browser, mobile application) — anyone who opens the browser developer tools can read it and steal it.
    • Always in environment variables or a secrets manager (AWS Secrets Manager, HashiCorp Vault, etc.).
    • Regular rotation: a key must be able to be revoked without breaking everything.
  2. OAuth (slide 9): when the application acts on behalf of a user (access your Google calendar, your CRM - Customer Relationship Management), we do not use a global API key but OAuth: the user explicitly authorizes, the application receives a limited and revocable token (access token). Simplify: “API key = company badge; OAuth = nominative visitor badge, with expiration date.”
  3. Rate limits (slide 10): Providers limit throughput — typically in RPM (Requests Per Minute) and TPM (Tokens Per Minute). ⚠ The exact figures vary depending on the supplier and the account level: give an order of magnitude (e.g. a few hundred to a few thousand RPM) and emphasize the reflex : read the HTTP response 429 Too Many Requests and the header retry-after .
  4. Retry & exponential backoff (slide 11): when a call fails (error 429 or 5xx), we try again — but not immediately and not indefinitely:
    • Attempt 1 fails → wait 1 s → attempt 2 → wait 2 s → attempt 3 → wait 4 s → attempt 4 → clean abort.
    • Add jitter (random variation) to prevent 1,000 clients from trying again at exactly the same time (“herd” effect — thundering herd).
    • Born Never retry 400/401/403 errors (invalid request, missing authentication, forbidden): retrying an invalid request will give the same invalid result.

SupportBot story: “Monday morning, 9:00 a.m.: 800 employees open SupportBot at the same time. Without retry or backoff, 30% of requests fail with a 429 error and users see “Server Error”. With backoff + jitter: everyone is served, with a 2–3 second wait for the less fortunate. Same business code, radically different experience. »

Common mistake made by participants: confuse rate limit (voluntary limitation by the supplier) and breakdown (involuntary unavailability). Both are managed by retry, but the rate limit is warns also: smooth traffic, queue, request a quota increase.

0:33 – 0:45 | Part C — Latency budget & streaming

Concepts:

  1. The latency budget (slide 12): start from the user, not from the technique. Classic UX (User eXperience) benchmarks:
    • < 1 s: perceived as instantaneous or smooth.
    • 1–3 s: acceptable with a wait indicator.
    • 3–10 sec: frustrating; requires streaming or an honest progress bar.
    • > 10 s: to be processed asynchronously (“we will notify you when it is ready”).
  2. Breaking down latency (slide 13): forward network (+ ~50–100 ms) + TTFT (Time To First Token, delay before the first token — often 200 ms to 1 s ⚠) + generation (proportional to the number of output tokens) + return network. The dominant position is almost always generation of output tokens . Direct consequence: a response twice as short is ~twice as fast — controlling the output length is a latency optimization.
  3. Streaming via SSE (slide 14): instead of waiting for the complete response, the server sends the tokens over the generation . SSE = Server-Sent Events, a standard HTTP mechanism where the connection remains open and the server pushes events. The total latency hardly changes, but the perceived latency collapses: the user sees the first word in less than a second.

Web demo: open the interactive page, “Monitoring” tab, show the TTFT vs total latency line. Then ask the question: “Which do you prefer: a complete answer in 8 seconds of frozen screen, or a first word at 0.8 s and the rest scrolling? » This is exactly why all AI chat interfaces stream.

Subtle point to mention: streaming slightly complicates the code (handling partial events, handling errors during the flow) and the output filtering (what do we filter if the response arrives in pieces?). Nothing insurmountable, but it’s not free.

0:45 – 1:00 | Exercise 1 — Cost calculator (15 min)

Organization : pairs, one computer per pair, web page “Cost calculator” tab. The worksheet guides you step by step.

Hidden educational objective: to discover by manipulation three truths:

  1. Exit tokens cost more than entry tokens (often 3 to 5× ⚠).
  2. Prompt caching radically changes the equation when the system prompt is long and repeated.
  3. Model routing (small model for 80% of cases) is often the biggest savings available.

Circulation in the room: spot the pairs who finished early and give them the bonus challenge (“find the configuration that divides the bill by 10 without changing the volume of requests”).

Restitution (last 3 minutes): ask two pairs for their numbers. Write the “naive” invoice vs the “optimized” invoice on the board. The gap (often 5–15×) is the memorable moment of the session.

1:00 – 1:05 | ☕ Break

During the break, let the model of the monitoring dashboard (animated metrics) be projected: it intrigues and prepares Part E.

1:05 – 1:20 | Part D — Cost management

Concepts, based on what Exercise 1 has just brought to life:

  1. Count tokens (slide 15): a token ≈ ¾ of a word in English, a little less in French (≈ 1.3–2 tokens per word ⚠ depending on the tokenizer). The invoice = entry tokens × entry price + exit tokens × exit price. Orders of magnitude ⚠ (July 2026, to be checked before each session): from ~0.1–1 $ par million de tokens d’entrée pour les petits modèles à ~2–15 $ for large ones; output typically costs 3–5x input.
  2. Prompt caching (slide 16): if the start of the prompt is identical from one request to another (system prompt, reference documents, tool definitions), the provider can cache it and charge for the cached tokens at a fraction of the price — up to 90% savings on the hidden part ⚠. Practical rule: put all content stable at first of the prompt, variable content (user question) at the end .
  3. Model routing (slide 17): not all requests are equal. “What are the support hours? does not need the same template as "Analyze this 40-page contract." Architecture: a classifier (or a small model, or simple rules) routes each request to the cheapest model capable of processing it. In practice, 70–90% ⚠ of a support assistant's traffic falls under the business model.
  4. Ceilings and alerts (slide 18): monthly token budget, alert at 50%/80%/100%, graceful cut-off or degradation (switch to the small model) in case of overrun. “A surprise bill of €20,000 kills more AI projects than a bug. »

Educational trap: don't turn this part into a pricing course — prices change quarterly. Teach it calculation method and the reflex “I check the price list of the day”, not the figures.

1:18 – 1:27 | Part D-bis — Quantization: fitting a large model onto a small machine (9 min)

🎯 To be inserted between cost management and observability. This part answers the question that always comes up: “What if I want to host the model myself?” » It ends with the message of Session 1 : there memory bandwidth (the speed at which we move the model weights from memory) is the real bottleneck of inference, not just the number of calculations.

The problem in one sentence. A model is billions of weight (numbers). In standard training precision, each weight occupies 32 bit (FP32, 32-bit floating point) or 16 bit (FP16). A model of 70 billion parameters in FP16 therefore weighs ≈ 140 GB — impossible to load on a consumer card (24 GB of VRAM), and slow to serve even on data center hardware, because these 140 GB must be reread for each token generated.

What is quantification? Reduce the number of bits by weight . We go from 16 bits to 8 bits (INT8, 8-bit integer), or even 4 bits (INT4). Each time the number of bits is divided by two, the size of the model is (approximately) halved. And the amount of memory to be reread per token — so we gain both footprint and speed.

Precision Bits/weight Size of a 70B model ≈ Analogy
FP32 32 ≈ 280 GB Uncompressed RAW photo
FP16 16 ≈ 140 GB High definition photo
INT8 (Q8) 8 ≈ 70 GB JPEG quality 90%
INT4 (Q4) 4 ≈ 40 GB JPEG quality 70%

The analogy to hammer home: quantization, this is the JPEG of the models . We accept a slight loss of quality (to the eye, often invisible) in exchange for a much smaller file that is quick to handle. As with JPEG, there is a slider: compressing too much (beyond INT4, e.g. 2 bits) visibly degrades the responses (the model becomes “fuzzy”: more errors, less nuance).

The quality/size compromise (key figure to be given): in practice, INT8 is almost indistinguishable of FP16 on most tasks; INT4 (Q4) loses very little (often 1–3 points on the test benches ⚠) for ~4× less memory as FP16. This is the default setting for general public self-hosting: a 70B in Q4 (≈ 40 GB) fits on two 24 GB cards, or on a Mac with unified memory.

How we do it, concretely. We do not re-quantify ourselves: we download a model already quantified , distributed in the format GGUF (the file format of quantized models for local execution), via a tool like Ollama Or llama.cpp . Copyable example to show:

# Installer Ollama puis récupérer un modèle open-weights quantifié en Q4
ollama run llama3:8b-instruct-q4_K_M "Explique la quantization en 2 phrases."
# Pour un 70B open-weights : ollama pull <modele>:70b-q4_K_M  # ~40 Go

The suffix q4_K_M reads: 4 bits , “K” variant (quantization by blocks, finer), “M” size (medium, good compromise). You will also see q5_K_M , q8_0, etc. : the higher the number, the more faithful and heavy it is.

When to use it (decision tree to draw):

Educational trap: quantization does not make a small model “as good” as a large one — it makes a large executable model on modest equipment. A 70B in Q4 remains much better than an 8B in FP16, for a comparable footprint. The real lever remains: the right model for the right task (Part D, routing).

Session 1 link to verbalize: “It was said that inference is limited by the memory bandwidth . Quantization attacks exactly this bottleneck: fewer bits to reread per token = more tokens per second. That's why she accelerates And lightens. »

1:27 – 1:34 | Part D-ter — Sovereignty & enterprise architecture (7 min)

Why add it here. Quantization answers the question “can I host?” ". Sovereignty answers the following question: “ what do I need to master for the company to agree to depend on this AI? » The right framing comes from Palantir (2026): institutional sovereignty is not just a confidentiality contract. It is designed in layers – data, models, calculation, control – and each layer must be chosen according to the real sensitivity of the workload.

1. ZDR is not a marketing slogan. Zero Data Retention (ZDR) means: the supplier does not retain prompts, responses, attachments, or ideally metadata exploitable (identifiers, times, volumes, fine traces). It's better than a standard API, but it's not equivalent to an owned environment. Formula to give: “ZDR reduces exposure; it does not give you complete sovereignty. »

Comparison of Zero Data Retention and Standard API

Source: Palantir — Institutional Sovereignty in the Age of AI (2026)

2. Decide by workload sensitivity. Draw a simple tree: public data → possible standard API; non-critical internal data → enterprise cloud with ZDR; sensitive/regulated customer data → ZDR strict or confidential computing; industrial secret, defense, critical health → owned or isolated environment. The professional reflex is not “all on-prem” nor “all cloud”: it is classification by workload .

AI decision tree by workload sensitivity

Source: Palantir — Institutional Sovereignty in the Age of AI (2026)

3. Structural insurance is better than contractual insurance. Present the scale of trust, from the most sovereign to the least sovereign: owned / air-gapped (equipment owned, isolated), then attested TEE (Trusted Execution Environment, trusted execution environment with cryptographic attestation), then ZDR cloud , Then Standard API . A contract promises; an architecture technically limits what can happen. Confidential computing is the modern compromise: if you rent computing, you require a certificate proving that the code and environment executed are those intended.

Structural Assurance Scale for AI Calculation

Source: Palantir — Institutional Sovereignty in the Age of AI (2026)

Concretely, the attestation workflow takes place in four stages: encrypted submission of the workload, execution in the TEE, signature of the attestation by the hardware, then verification of the proof before accepting the result. Show the diagram: it is this circuit that replaces contractual trust with cryptographic proof.

Hardware attestation workflow: encrypted submission, TEE execution, signature, verification

Source: Palantir — Institutional Sovereignty in the Age of AI (2026)

4. Direct link with quantization. Self-hosting is only viable if the model fits on the hardware you control. Quantization (Q4/Q5/Q8) therefore transforms a sovereignty constraint into a possible architecture: a fine-tuned, quantized open-weights model, served on owned GPU or Mac with unified memory, with internal logs and permissions. This is not always the right choice – fixed cost, expertise, operations – but for certain workloads, it is the only defensible answer.

Sovereign AI architecture in three layers

Source: Palantir — Institutional Sovereignty in the Age of AI (2026)

For participants who want the complete view, show the end-to-end on-prem stack: owned GPUs → self-hosted models → control layer (permissions, logs) → business ontology → users and agents. Each layer answers a different sovereignty question.

On-prem architecture stack: from owned hardware to users and agents

Source: Palantir — Institutional Sovereignty in the Age of AI (2026)

Summary sentence: “The higher the business risk, the more you must replace declarative confidence with technical control: zero retention, certified calculation, equipment owned, and ability to change model. »

1:34 – 1:41 | Part E — Observability, prompt versioning, CI/CD

Concepts:

  1. Log each request/response (slide 19): timestamp, request identifier, prompt version, model used, input/output tokens, latency, status. Without a log, a user’s first “it’s not working” is undiagnosable. Please note GDPR (General Data Protection Regulation): define the retention period and anonymize/pseudonymize if the prompts contain personal data.
  2. Latency percentiles (slide 20): the average lies. p50 (median) = typical experience; p95 = the experience of 1 user in 20; p99 = worst cases. A service can average 1.2s and 9s on p99 — and it's the p99 users who complain and leave. SLAs (Service Level Agreement) are written in percentiles: “p95 < 3 s", never "average < 2 sec.
  3. Error rates and alerts (slide 21): alert thresholds on the error rate (e.g. > 2% over 5 minutes), p95 latency, token budget. An alert must be actionable : If no one knows what to do when it rings, delete it or document it.
  4. Prompts are code (slide 22): versioned (Git), reviewed (peer review), tested (suite of evaluations). CI/CD for AI: with each PR (Pull Request, code merge request) that modifies a prompt, the suite of evaluations runs automatically — prompt regression tests. “Changing a word in a prompt can break 15% of responses. Without CI evaluations, you discover it in production, via your users. »

Web demo: dashboard mockup — show the metrics that are moving, the p95 that is stalling, the budget alert that is turning orange. Ask: “What metric would you look at first thing Monday morning?” »

Link to previous sessions: recall that the suite of evaluations was conceptually constructed in the sessions devoted to the quality of prompts — here, we automates .

1:41 – 1:49 | Exercise 2 — Deployment architecture (8 min)

Organization : same pairs. Specifications in the worksheet (three scenarios to choose from: internal HR chatbot, high-volume document summary API, real-time client-side assistant). The web page architecture builder serves as a visual support: drag and drop the components (API gateway, LLM provider, cache, queue, database) and read the latency estimate.

What you evaluate while driving:

Restitution (3 min): a pair presents their architecture; the challenge room. You play devil’s advocate: “What happens if the LLM provider goes down 10 minutes?” » (Expected responses: retry, degraded message, absorbing queue, possibly backup provider.)

1:49 – 1:53 | Part F — Deployment patterns, security, scalability

This part is deliberately dense and fast - Exercise 2 has already manipulated the concepts.

  1. Three deployment patterns (slide 24):
    • Serverless (Lambda, Cloud Functions): zero servers to manage, runtime billing, automatic scaling. Limitations: capped execution time, cold start. Ideal for irregular traffic and small teams.
    • Container (Docker, orchestrated for example with Kubernetes): portable, reproducible, fine control. Higher skill cost. Ideal for sustained and regular traffic.
    • Dedicated GPU instance : only if you host the model yourself (open-weights) — data sovereignty, high fixed cost, in-depth expertise. The majority of projects do not have not need: the API call to a provider is sufficient.
  2. Security (slide 25): the three non-negotiable rules — API key never on the client side (always an intermediate backend), input sanitation (maximum length, prompt injection detection), output filtering (sensitive data, inappropriate content) before display.
  3. Scaling up (slide 26): horizontal rise (several instances behind a load balancer) rather than vertical (a larger machine). For long or massive treatments: queue architecture (queue) — the client submits a task, workers process it at the rate of rate limits, the client is notified. This is THE boss for batch processing.

Shortcut if late: only project the comparative table of the three bosses and the safety slide. The queue has already been seen in Exercise 2.

1:53 – 2:00 | Quiz, Exit Tickets, closing

4. The 5 Exit Tickets

To be distributed (paper or form) in the last 5 minutes. Each participant responds to a question (distribute them alternately) or to all five in short version.

  1. “List two differences between a notebook prototype and a production service, and explain why they matter. » Expected: two of — secrets management, retry/rate limits, controlled latency, cost tracking, observability, security, scalability — with a one-sentence justification.

  2. “Your application receives an HTTP 429 error. What does it mean and what should your code do? » Expected: rate limit exceeded (Too Many Requests) → wait then retry with exponential backoff (+ jitter), respecting the header retry-after if it is present. Don't hammer the API.

  3. “Explain in two sentences why SSE streaming improves user experience without reducing total latency. » Expected: tokens are displayed over generation; the first token arrives in < 1 s, therefore the latency perceived falls, even if the complete generation takes the same time.

  4. “Give two levers to reduce the cost of an LLM application without reducing the volume of requests. » Expected: two among — prompt caching (stable content at the start of the prompt), routing to a cheaper model for simple tasks, reduction of output length, reduction of injected context (better RAG).

  5. “Why do we say “prompts are code”? Name a concrete practice that results from this. » Expected: a change in prompt changes the behavior of the system like a code change → Git versioning, peer review, regression tests (sequence of evaluations) executed in CI/CD for each modification.

  6. “What is the use of quantization when you self-host a model? » Expected: it reduces the number of bits per weight (FP16 → INT8 → INT4), therefore the memory footprint and the bandwidth to be reread per token. Q4 allows for example to fit a 70B around 40 GB, with a limited loss of quality.

Operation: read the tickets before Session 9. If more than 30% of the responses to ticket 2 confuse 429 and server failure, allow 5 minutes of reminder at the opening of the next session.

5. Frequently asked questions from participants (and ready answers)

“Why not call the provider's API directly from the browser? It would be simpler. » Because the API key would be visible to anyone who opens the browser's developer tools (F12 → Network tab). It would be stolen within hours and used at your expense. An intermediate backend — even tiny, even serverless — is OBLIGATORY . This is the #1 safety rule of the session.

“Do we manage prompt caching or the supplier? » The provider hosts the cache, but YOU structure the prompt to take advantage of it: stable content at the beginning, variable content at the end, and depending on the provider, an explicit cache point marker. If your system prompt changes on every request (dynamic timestamp, for example), the cache is useless.

“Do you need a GPU to deploy an AI application? » No, in the vast majority of cases. If you call a provider's API (Anthropic, OpenAI, Mistral...), the GPU is at home. You only need GPUs if you're hosting an open-weights model yourself — a choice that is justified by data sovereignty or huge volume, not by default.

“How much does an app like SupportBot really cost? » ⚠ It depends on the daily rates, but the method is stable: daily volume × (entry tokens × entry price + exit tokens × exit price), then apply caching and routing. Exercise 1 gives the method; redo the calculation with the current price list. Order of magnitude: from a few tens to a few thousand euros/month depending on volume and optimization.

“The average latency of our service is good, why are users complaining? » Because the mean hides the tail of the distribution. Look at p95 and p99: if p99 is 9 s, 1 request in 100 is unbearable — and these users are noisy. Commit to percentiles, not averages.

“We already have a CI/CD for our code. What's changing with AI? » Two additions: (1) the evaluation suite runs at each PR that touches a prompt or a model parameter — these are the unit tests of the model behavior; (2) the outputs being non-deterministic, the tests relate to criteria (respected format, presence of key information, score of a judge) rather than to strict equality.

6. Emergency equipment (if the technique fails)