# Exercises — Session 8: Infrastructure & Deployment

**Program:** Applied AI — Intermediate Level — Instructor: Yann Isola
**Reminder of the central message:** *a working prototype is 20% of the work. The remaining 80% is called production: authentication, latency, costs, observability, security, scalability.*

---

## Exercise 1 — The cost calculator (15 min, in pairs)

### Context

You resume “SupportBot”, the internal support assistant of the session. It has been running as a pilot for a month and management is requesting a **cost projection** before deployment to the entire company (5,000 employees).

**Usage profile measured during the pilot:**

| Parameter | Value |
|---|---|
| Queries per day (full scale projection) | 10,000 |
| System prompt + tool definitions + reference documents (same for each query) | 6,000 tokens |
| User question + variable RAG context | 1,500 tokens |
| Response generated (average) | 500 tokens |

**Fictitious price list** (⚠ inspired by the real orders of magnitude of July 2026 — in production, always check the current table):

| Model | Entry ($/million de tokens) | Sortie ($/million tokens) | Cache entry ($/million) |
|---|---|---|---|
| **Large model** (complex reasoning) | 3.00 | 15.00 | 0.30 |
| **Intermediate model** | 1.00 | 5.00 | 0.10 |
| **Small model** (simple tasks) | 0.25 | 1.25 | 0.025 |

### Your work

Use the **web page calculator** (“Cost Calculator” tab) or do the calculations by hand with the formulas below.

**Basic formula:**```
Coût journalier = requêtes/jour × [ (tokens_entrée / 1 000 000) × prix_entrée
                                  + (tokens_sortie / 1 000 000) × prix_sortie ]
Coût mensuel ≈ coût journalier × 30
```**Step 1 — The naive scenario.** All traffic goes to the **large model**, without caching. Input tokens per request = 6000 + 1500 = 7500.
→ Calculate the daily then monthly cost.

**Step 2 — Enable prompt caching.** The 6,000 stable tokens are billed at the “cached” rate (we neglect the first request that writes the cache). Only the 1,500 variable tokens remain at the full entry price.
→ Recalculate. What percentage of savings on the total bill?

**Step 3 — Add Model Routing.** Pilot analysis shows that **80% of questions are simple** (timetables, standard procedures) and can be served by the **small model**; 20% require the **large model**. Caching remains active on both.
→ Recalculate the total monthly bill.

**Step 4 — Summary for management.** Complete the table:

| Scenario | Monthly cost | Economy vs. naive |
|---|---|---|
| Naive (large model, without cover) | | — |
| + Prompt caching | | |
| + 80/20 routing | | |

**Bonus question 🏆:** without changing the volume of requests or the price list, what additional lever could further reduce the bill? (Hint: look at the “output” column of the grid and the average length of responses.)

### Indicative answer (for the trainer)

- **Step 1:** input: 10,000 × 7,500 / 1M × 3.00 $ = 225 $/d; output: 10,000 × 500 / 1M × 15.00 $ = 75 $/d. **Total: 300 $/j ≈ 9 000 $/month.**
- **Step 2:** hidden entry: 10,000 × 6,000 / 1M × 0.30 $ = 18 $/d; live input: 10,000 × 1,500 / 1M × 3.00 $ = 45 $/d; output unchanged 75 $/j. **Total : 138 $/d ≈ $4,140/month** (−54%).
- **Step 3:** 8,000 req/d on the small model: cache 8,000 × 6,000/1M × 0.025 = 1.20 $ ; entrée vive 8 000 × 1 500/1M × 0,25 = 3,00 $; output 8,000 × 500/1M × 1.25 = 5.00 $ → 9,20 $/d. 2,000 req/d on the large model: cache 3.60 $ ; entrée vive 9,00 $; exit 15.00 $ → 27,60 $/d. **Total: 36.80 $/j ≈ 1 104 $/month** (−88% vs. naive).
- **Bonus:** reduce the length of the outputs (brevity instruction in the prompt, `max_tokens` capped) — the output is the most expensive token; going from 500 to 300 output tokens further reduces the dominant position. Other acceptable answers: reduce the injected RAG context (1500 → 800 tokens), use the intermediate model instead of the large one for part of the 20%.

---

## Exercise 2 — Design a deployment architecture (13 min, in pairs)

### Context

Choose **one** of the three scenarios (or the one the trainer assigns to you). For each, you must design the deployment architecture by assembling the available components, then justify your choices. The **architecture builder** of the web page (“Architecture” tab) allows you to visualize the assembly and latency estimation.

**Available components:** client (browser/mobile) · API gateway · application backend · response cache · queue + workers · database · LLM provider (external API) · logging/monitoring.

### Scenario A — Internal HR chatbot
- 2,000 employees, ~1,500 requests/day, peaks at 9 a.m. and 2 p.m.
- Users expect a quick conversational response.
- Sensitive HR data in context.### Scenario B — High Volume Document Summarization API
- An internal service submits 50,000 documents every night to be summarized.
- No human waits in front of the screen: the result must be ready the next morning.
- The LLM provider imposes a rate limit of 500 requests/minute ⚠.

### Scenario C — Real-time assistant integrated into a public site
- Chat widget on an e-commerce site, unpredictable public traffic (10× on sales days).
- Critical perceived latency: first token < 1 s expected.
- High risk of malicious input (prompt injection, spam).

### Your work

1. **Draw the architecture** (on the web page or on paper): which components, in what order, who speaks to whom.
2. **Choose the deployment pattern** of the backend — serverless, container, or dedicated GPU instance — and justify it in 2 sentences.
3. **Answer the three robustness questions:**
- Where is the LLM provider API key stored? (Any response involving the customer is eliminatory.)
- What happens if the LLM provider returns 429 errors for 5 minutes?
- What is the first metric that you put under alert?
4. **Streaming or not?** Tell us if your scenario benefits from SSE (Server-Sent Events) streaming, and why.

### Self-assessment grid

| Criterion | ✅/❌ |
|---|---|
| The API key is server side, never in the client | |
| A logging/monitoring component is present | |
| The strategy for 429 errors is explicit (backoff, file, degraded message) | |
| The serverless/container/GPU choice is justified by the traffic profile | |
| Scenario B uses a queue (if scenario B chosen) | |
| The streaming decision is consistent with the use case | |

### Indicative answer (for the trainer)

- **Scenario A:** client → API gateway → backend (serverless is suitable: moderate traffic with peaks, automatic rise) → LLM provider, + logging + possible cache on frequently asked questions. Streaming: **yes** (conversational). Please note HR data: pseudonymized logging, limited retention.
- **Scenario B:** document submission → **queue** → workers in containers which consume at the rate limit (500 req/min → ~100 minutes for 50,000 documents, comfortable margin over one night) → results database. Streaming: **no** (no humans waiting). Pattern: containers (regular load, long execution time — serverless is duration limited). Key metric: queue depth + error rate.
- **Scenario C:** client → API gateway (with IP rate limiting against spam) → serverless backend (unpredictable traffic ×10 → automatic rise) → LLM provider in **SSE streaming**. Mandatory sanitation of inputs (max length, injection filtering), filtering of outputs before display. Key metric: TTFT (Time To First Token) p95 and error rate.
- **Dedicated GPU:** none of the three scenarios justifies it — point to emphasize in restitution. The GPU is only justified for the self-hosting of an open-weights model (sovereignty, extreme volume).

---

## Exercise 3 — Design a monitoring dashboard (homework or bonus, ~30 min)

### Context

SupportBot has been in production for two weeks. Yesterday, a user reported "it's slow and sometimes responds nonsense." You currently have **no** dashboards.Your mission: design (on paper or with the tool of your choice) the monitoring dashboard that the team will consult every morning, taking inspiration from the model of the web page (“Monitoring” tab).

### Your work

1. **Choose 6 to 8 metrics**, no more. For each, specify:
- the name and the unit;
- the source (request log, provider API, user feedback, etc.);
- the observation window (real time, hourly, daily);
- the **alert threshold** and the associated action (an alert without documented action is prohibited).

2. **Organize them into three areas** on your mockup:
- 🟢 **Health** — is the service working? (availability, error rate, etc.)
- ⏱️ **Performance** — is it fast enough? (latency p50/p95/p99, TTFT…)
- 💰 **Costs** — how much do we burn? (tokens/day, cumulative cost vs budget, cache hit rate, etc.)

3. **Write the 3 priority alert rules** in the format:
> IF `<métrique>` `<condition>` DURING `<durée>` THEN `<qui est prévenu>` DOES `<quoi>`.

4. **Trick question:** Does your dashboard detect the “sometimes it responds whatever” problem? If no (probable), what should be added? (Hint: Response quality isn't measured in milliseconds or tokens — think about continuous evaluations and user feedback 👍/👎.)

### Evaluation grid (for the trainer)

| Criterion | Points |
|---|---|
| 6–8 relevant metrics, with unit and source | /4 |
| Latency expressed in percentiles (not just average) | /2 |
| Each alert has a threshold, duration AND documented action | /3 |
| Cost zone: token budget + cache rate present | /2 |
| Trick question: identifies the limit (quality ≠ technical metrics) and proposes a quality signal (continuous evaluations, user feedback, manual sampling) | /4 |
| General clarity of the model | /1 |
| **Total** | **/16** |

### Answer key elements (for the trainer)

Expected metrics (any consistent selection is acceptable): availability (%), error rate (% over 5 min), 429 error rate specifically, p50/p95/p99 latency (s), p95 TTFT (ms), tokens consumed/day vs. budget (%), monthly cumulative cost ($), cache hit rate (%), small/large model routing distribution (%), queue depth (if queue), average continuous eval score, 👍 user.

Example of a well-formed alert rule:
> IF `taux d'erreur` `> 2 %` DURING `5 minutes` THEN `l'ingénieur d'astreinte reçoit une notification` DOES `vérifier le statut du fournisseur LLM, activer le mode dégradé si panne confirmée`.

Answer to trick question: no — technical metrics don't see quality. Add: (1) a 👍/👎 ​​button in the interface, aggregated daily; (2) an automatic evaluation on a sample of X% of responses in production (LLM judge or rules); (3) a weekly human review of a random sample. This is the bridge to Session 9.

---

## Exercise 4 — Self-hosting: sizing a quantization (10 min, in pairs)

### Context

A clinic refuses to send its patient data to an external provider (**data sovereignty** — data must never leave the building). You must therefore **self-host** an assistant on a server equipped with **2 GPU cards of 24 GB** of video memory (VRAM), for **48 GB** in total. The open-weights model chosen has **70 billion parameters**.**Course reminder (imprint formula):**```
Taille mémoire des poids ≈ nombre_de_paramètres × (bits_par_poids / 8)  [en octets]
```(We neglect the context memory/KV-cache here; the order of magnitude is enough to decide.)

### Your work

**Step 1 — Calculate the footprint** of Model 70B at each precision and complete the table. Indicate if it **fits** within the 48 GB available.

| Accuracy | Bits/weight | Footprint ≈ | Fits in 48 GB? |
|---|---|---|---|
| FP32 | 32 | | |
| FP16 | 16 | | |
| INT8 (Q8) | 8 | | |
| INT4 (Q4) | 4 | | |

**Step 2 — Choose** the precision you would deploy and justify it in 2 sentences (footprint **and** quality).

**Step 3 — Translate** your choice into a concrete command: which tool (Ollama / llama.cpp), which file format, which quantization suffix? Write the corresponding `ollama pull ...` line.

**Step 3-bis — Local test if possible.** If your machine has Ollama, run a small model already quantified and note: (1) the order, (2) the time before the first token, (3) your qualitative impression on a short answer. Copyable example:```bash
ollama run llama3:8b-instruct-q4_K_M "Explique la quantization en 2 phrases."
```If Ollama is not installed, just write down what you would test and why this Q4 model is less risky than a 70B for a training position.

**Step 4 — Question of judgment.** A colleague suggests "let's instead take a small model of 8B in FP16, so there's no need to quantify." What do you answer? (Hint: compare the expected **quality** of a 70B-Q4 versus an 8B-FP16.)

**Bonus question 🏆:** why does quantization **accelerate** inference, and not just footprint reduction? (Hint: Session 1, the inference bottleneck.)

### Indicative answer (for the trainer)

- **Step 1:** FP32 = 70 × 4 = **280 GB** (does not fit); FP16 = 70 × 2 = **140 GB** (does not fit); INT8 = 70 × 1 = **70 GB** (does not fit in 48 GB); INT4 = 70 × 0.5 = **~35–40 GB** (✅ fits, with a margin for context).
- **Step 2:** **INT4 (Q4)** — only precision that fits in 48 GB, and the quality loss against FP16 is generally in the order of 1–3 points ⚠, acceptable for an internal assistant. (Q5_K_M ≈ 45–48 GB can also be attempted if the measured quality justifies it, but the margin is slim.)
- **Step 3:** `ollama pull <modele>:70b-q4_K_M` (format **GGUF**, 4 bits, “K” blocks, “M” size medium), then `ollama run`.
- **Step 3-bis:** expected: a short test of the `ollama run llama3:8b-instruct-q4_K_M "Explique la quantization en 2 phrases."` type, with observation of the time before the first token and the fluidity. If not tested, the learner must justify: installation / material / time constraints, and specify that the exercise mainly verifies the sizing reasoning.
- **Step 4:** a **70B-Q4 remains significantly better** than an 8B-FP16 for a comparable footprint. Quantization does not make a small model "as good" as a large one — it makes a **large model executable** on modest hardware. The “small model” reflex is only justified if the tasks are really simple (see routing, Exercise 1).
- **Bonus:** inference is limited by **memory bandwidth** — you have to reread all the weights from memory for each generated token. Fewer bits per weight = fewer bytes to reread per token = more tokens per second. Quantization makes **and** faster for the same reason.