This is why quantization matters
Quantization stores weights at lower precision — 8-bit or 4-bit instead of 16. The obvious win is that the model fits. The less obvious and often larger win is that inference is memory-bandwidth bound, so moving fewer bytes per token makes it faster too.
- 2026-03-18
- Programmatic DIB
- Quantization · LLM inference · Model efficiency
Two wins, not one
The first is capacity: halving precision roughly halves the weight memory, which decides whether a model fits on the hardware you have at all.
The second is speed, and it surprises people. Generating a token requires reading essentially every weight. That makes decoding memory-bandwidth bound rather than compute bound, so halving the bytes read per token can nearly halve the time per token — even though the arithmetic is unchanged.
What it costs
Precision loss is real but unevenly distributed. Most weights tolerate aggressive quantization; a small number of outlier channels do not, and naive uniform quantization on those is where quality collapses. Modern schemes handle outliers separately, which is most of why 4-bit is usable at all.
The honest framing is the same as for pruning: this is a priced trade, and the price should be measured on your workload rather than assumed from a benchmark.
- Lower precision buys capacity and speed, because decoding is bandwidth bound.
- Quality loss concentrates in a few outlier channels, not uniformly.
- Measure the cost on your own workload rather than trusting a benchmark number.