Français
Applied AI · Intermediate 🟡 · Session 8
✏️ Exercises
← Return to program 📄 Source .md

Exercises — Session 8: Infrastructure & Deployment

Program : Applied AI — Intermediate Level — Instructor: Yann Isola Reminder of the central message: a working prototype is 20% of the work. The remaining 80% is called production: authentication, latency, costs, observability, security, scalability.

Exercise 1 — The cost calculator (15 min, in pairs)

Context

You resume “SupportBot”, the internal support assistant of the session. It has been running as a pilot for a month and management is requesting a cost projection before deployment to the entire company (5,000 employees).

Usage profile measured during the pilot:

Setting Value
Queries per day (full scale projection) 10 000
System prompt + tool definitions + reference documents (identical for each request) 6,000 tokens
User question + variable RAG context 1,500 tokens
Response generated (average) 500 tokens

Fictitious price list (⚠ inspired by the actual orders of magnitude of July 2026 — in production, always check the daily grid):

Model Entry ($/million tokens) Output ($/million tokens) Cache entry ($/million)
Large model (complex reasoning) 3,00 15,00 0,30
Intermediate model 1,00 5,00 0,10
Small model (simple tasks) 0,25 1,25 0,025

Your work

Use the web page calculator (“Cost Calculator” tab) or do the calculations by hand using the formulas below.

Basic formula:

Coût journalier = requêtes/jour × [ (tokens_entrée / 1 000 000) × prix_entrée
                                  + (tokens_sortie / 1 000 000) × prix_sortie ]
Coût mensuel ≈ coût journalier × 30

Step 1 — The Naive Scenario. All traffic passes on the large model , without caching. Input tokens per request = 6000 + 1500 = 7500. → Calculate the daily then monthly cost.

Step 2 — Enable prompt caching. The 6,000 stable tokens are billed at the “cache” rate (we neglect the first request which writes the cache). Only the 1,500 variable tokens remain at the full entry price. → Recalculate. What percentage of savings on the total bill?

Step 3 — Add Model Routing. Analysis of the pilot shows that 80% of questions are simple (schedules, standard procedures) and can be served by the small model ; 20% require large model . Caching remains active on both. → Recalculate the total monthly bill.

Step 4 — Summary for management. Complete the table:

Scenario Monthly cost Economy vs. naive
Naive (large model, without cover) —
+ Prompt caching
+ 80/20 routing

Bonus question 🏆: without changing the volume of requests or the price list, what additional lever could further reduce the bill? (Hint: look at the “output” column of the grid and the average length of responses.)

Indicative answer key (for the trainer)

Exercise 2 — Design a deployment architecture (13 min, in pairs)

Context

Choose A of the three scenarios (or the one that the trainer assigns to you). For each, you must design the deployment architecture by assembling the available components, then justify your choices. THE architectural builder of the web page (“Architecture” tab) allows you to visualize the assembly and latency estimation.

Available components: client (browser/mobile) · API gateway · application backend · response cache · queue + workers · database · LLM provider (external API) · logging/monitoring.

Scenario A — Internal HR Chatbot

Scenario B — High Volume Document Summarization API

Scenario C — Real-time assistant integrated into a public site

Your work

  1. Draw the architecture (on the web page or on paper): which components, in what order, who speaks to whom.
  2. Choose the deployment pattern of the backend — serverless, container, or dedicated GPU instance — and justify in 2 sentences.
  3. Answer the three robustness questions:
    • Where is the LLM provider API key stored? (Any response involving the customer is eliminatory.)
    • What happens if the LLM provider returns 429 errors for 5 minutes?
    • What is the first metric you put on alert?
  4. Streaming or not? State whether your scenario benefits from Server-Sent Events (SSE) streaming, and why.

Self-assessment grid

Criteria /❌
The API key is server side, never in the client
A logging/monitoring component is present
The strategy for 429 errors is explicit (backoff, file, degraded message)
The serverless/container/GPU choice is justified by the traffic profile
Scenario B uses a queue (if scenario B chosen)
The streaming decision is consistent with the use case

Indicative answer key (for the trainer)

Exercise 3 — Design a monitoring dashboard (homework or bonus, ~30 min)

Context

SupportBot has been in production for two weeks. Yesterday, a user reported "it's slow and sometimes it responds nonsense." You currently do not have none dashboard. Your mission: design (on paper or with the tool of your choice) the monitoring dashboard that the team will consult every morning, taking inspiration from the model of the web page (“Monitoring” tab).

Your work

  1. Choose 6 to 8 metrics , no more. For each, specify:

    • name and unit;
    • the source (request log, provider API, user feedback, etc.);
    • the observation window (real time, hourly, daily);
    • THE alert threshold and the associated action (an alert without documented action is prohibited).

  2. Organize them into three areas on your model:

    • 🟢 Health — is the service working? (availability, error rate, etc.)
    • ⏱️ Performance — is it fast enough? (latency p50/p95/p99, TTFT…)
    • 💰 Costs — how much do we burn? (tokens/day, cumulative cost vs budget, cache hit rate, etc.)

  3. Write the 3 priority alert rules in format:

    IF <métrique> <condition> DURING <durée> SO <qui est prévenu> DO <quoi>.

  4. Trick question: Does your dashboard detect the “sometimes it responds anything” problem? If no (probable), what should be added? (Hint: Response quality isn't measured in milliseconds or tokens — think about continuous evaluations and user feedback 👍/👎.)

Evaluation grid (for the trainer)

Criteria Points
6–8 relevant metrics, with unit and source /4
Latency expressed in percentiles (not just average) /2
Each alert has a threshold, duration AND documented action /3
Cost zone: token budget + present cache rate /2
Trick question: identifies the limit (quality ≠ technical metrics) and offers a quality signal (continuous evaluations, user feedback, manual sampling) /4
General clarity of the model /1
Total /16

Answer keys (for the trainer)

Expected metrics (any consistent selection is acceptable): availability (%), error rate (% over 5 min), 429 error rate specifically, p50/p95/p99 latency (s), p95 TTFT (ms), tokens consumed/day vs. budget (%), monthly cumulative cost ($), cache hit rate (%), small/large model routing distribution (%), queue depth (if queue), average score of continuous evaluations, rate of 👍 user.

Example of a well-formed alert rule:

IF taux d'erreur > 2 % DURING 5 minutes SO l'ingénieur d'astreinte reçoit une notification DO vérifier le statut du fournisseur LLM, activer le mode dégradé si panne confirmée .

Answer to trick question: no — technical metrics don't see quality. Add: (1) a 👍/👎 ​​button in the interface, aggregated daily; (2) an automatic evaluation on a sample of X% of responses in production (LLM judge or rules); (3) a weekly human review of a random sample. This is the bridge to Session 9.

Exercise 4 — Self-hosting: sizing a quantization (10 min, in pairs)

Context

A clinic refuses to send its patient data to an external provider (data sovereignty — the data must never leave the building). You must therefore self-host an assistant on a server equipped with 2 x 24 GB GPU cards of video memory (VRAM), or 48 GB in total. The open-weights model chosen takes into account 70 billion parameters .

Course reminder (footprint formula):

Taille mémoire des poids ≈ nombre_de_paramètres × (bits_par_poids / 8)  [en octets]

(We neglect the context memory/KV-cache here; the order of magnitude is enough to decide.)

Your work

Step 1 — Calculate the footprint of model 70B to each precision and complete the table. Indicate if this holds within the 48 GB available.

Precision Bits/weight Footprint ≈ Fits in 48 GB?
FP32 32
FP16 16
INT8 (Q8) 8
INT4 (Q4) 4

Step 2 — Choose the precision that you would deploy and justify in 2 sentences (imprint And quality).

Step 3 — Translate your choice in a concrete command: which tool (Ollama / llama.cpp), which file format, which quantization suffix? Write the line ollama pull ... corresponding.

Step 3-bis — Local test if possible. If your machine has Ollama, run a small model already quantified and note: (1) the order, (2) the time before the first token, (3) your qualitative impression on a short answer. Copyable example:

ollama run llama3:8b-instruct-q4_K_M "Explique la quantization en 2 phrases."

If Ollama is not installed, just write down what you would test and why this Q4 model is less risky than a 70B for a training position.

Step 4 — Judgment question. A colleague suggests “let’s instead take a small 8B model in FP16, so there’s no need to quantify”. What do you answer? (Hint: compare the quality expected of a 70B-Q4 compared to an 8B-FP16.)

Bonus question 🏆: why quantization does she accelerate inference, not just footprint reduction? (Hint: Session 1, the inference bottleneck.)

Indicative answer key (for the trainer)