The key-value cache stores attention state for tokens already processed. It speeds decoding but grows with context and concurrent sequences.
Sizing model
A common decoder-only estimate is 2 × layers × KV heads × head dimension × cached tokens × bytes per element. Multiply by active sequences when caches are independent. Architectures and runtimes vary.
Optimization levers
Bound context, batch requests deliberately, use grouped-query or multi-query attention when the model supports it, page cache blocks, reuse verified shared prefixes, and evaluate lower cache precision separately from weight quantization.
Measure the service
Track time to first token, inter-token latency, output throughput, cache occupancy, eviction/recompute rate, and p50/p95 latency at realistic context distributions and concurrency.
Quick check
What happens to KV memory when active context doubles?
Show answer
Under the simplified linear estimate, it approximately doubles.
Is provider prompt caching identical to a request’s runtime KV cache?
Show answer
No. Provider prefix reuse has provider-specific persistence and eligibility rules.