KV-cache sizing and optimization

The key-value cache stores attention state for tokens already processed. It speeds decoding but grows with context and concurrent sequences.

Sizing model

A common decoder-only estimate is 2 × layers × KV heads × head dimension × cached tokens × bytes per element. Multiply by active sequences when caches are independent. Architectures and runtimes vary.

Optimization levers

Bound context, batch requests deliberately, use grouped-query or multi-query attention when the model supports it, page cache blocks, reuse verified shared prefixes, and evaluate lower cache precision separately from weight quantization.

Measure the service

Track time to first token, inter-token latency, output throughput, cache occupancy, eviction/recompute rate, and p50/p95 latency at realistic context distributions and concurrency.