Hardware fit depends on workload shape. Quantization can reduce model-weight memory, but compatibility, latency, and task quality still require measurement.
CPU and GPU roles
CPUs excel at general control flow, preprocessing, and modest workloads. GPUs expose many parallel arithmetic units and high memory bandwidth for dense tensor operations. Prefill is highly parallel; autoregressive decode remains sequential across token positions.
Memory estimate
A first weight-memory estimate is parameters × bits per weight ÷ 8. It excludes KV cache, activations, allocator overhead, runtime buffers, metadata, and sometimes unquantized layers.
Quantization release gate
Compare the quantized candidate with a higher-precision baseline on representative tasks. Record quality, time to first token, inter-token latency, throughput, peak memory, hardware, runtime, and artifact hash.
Approximation boundary
Keep workload, prompts, runtime, and quality rubric fixed while comparing hardware or precision. Separate estimates from measured resident memory.