Why Fewer Bits Can Change the Answers

What is quantization in an AI model?

Quantization represents numbers in a model with fewer bits. Weight quantization reduces the space needed for its weights. The effect on speed and answers depends on the model, method, and task.

After compression, a model may fit on a smaller card. That solves the memory problem. Whether it still solves the tasks equally well is a separate question. An answer that sounds similar is not enough evidence.

Less Space for the Same Weights

Weights are numbers a model learned during training. They help determine how an input becomes the next answer. Those numbers take up memory.

The smaller representation has fewer possible values. One simple method rounds each original value to the nearest value in that smaller set. Several nearby numbers can then become the same number. A distinction the model could previously represent has been lost.

This approximation using fewer bits has a name: quantization.

What Four Bits Instead of Sixteen Means

Weight-only quantization from FP16 or BF16 to INT4 reduces the bit-width per weight from 16 to 4. The raw weight bits then take a quarter of the space. Scales and unquantized parts need additional memory. The KV cache does not shrink along with them.

A worked example with simplified numbers: 16 values are to cover the range from -0.8 to 0.7. The step size is 0.1. Rounding changes the weight 0.26 to 0.3. This is stored as the integer 3 with a scale of 0.1. The integer therefore represents a value in the range of the weights.

INT8 offers 256 values instead of 16. Smaller groups of weights can each receive their own scales to better cover their ranges. FP8 also uses eight bits but represents floating-point numbers. NVIDIA Hopper GPUs, for example, support this format.

GPTQ and AWQ are methods for quantizing weights after training. GPTQ seeks a layer-by-layer approximation with a small error at the layer output. AWQ uses activation data to protect important weight channels through scaling. Both need sample data for calibration. How well that data matches the later task can affect the result.

The KV cache, by contrast, retains computation results for the current conversation. Quantizing it is a separate decision, especially with long contexts.

Test the Task the Model Will Actually Do

Fewer bits do not affect every model and task equally. Li et al. found greater sensitivity on multi-step mathematical reasoning among the tasks they tested. ParoQuant describes cases where deviations compound over long generations. Liu et al., however, report nearly lossless four-bit weight quantization on the reasoning models they studied.

The test therefore needs to include the intended work. For an agent run, a handful of short chat questions is not enough evidence. Context length, answer length, and the tasks themselves need to match. But differences can also appear in short answers. A similar average score can hide changes in how individual tasks are solved.

The speed gain is also an open question. The QSPEC authors show that quantized models can run faster or slower than FP16, depending on implementation and batch size. Memory use, runtime, and answer quality therefore need separate comparisons.

For self-hosted inference, the next question is whether the setup reduces costs while meeting the quality requirement. Using less memory alone does not automatically make a rented GPU or the entire operation cheaper.

The model’s tasks determine how many bits are enough.

Next term: speculative decoding. A small model proposes tokens and the large one checks them. This can make generation faster.

Frequently Asked Questions

What is quantization in an AI model? Quantization represents numbers in a model with fewer bits. Weight quantization reduces the space needed for its weights. The effect on speed and answers depends on the model, method, and task.

What is the difference between INT8 and INT4? INT8 represents 256 values, while INT4 represents only 16. The raw weight bits take half as much space in INT4 as in INT8. Scales map the integers to the range of the weights. Bit-width alone does not establish whether the answers remain good enough.

Does quantization always make a model faster? No. Storing weights in less space can speed up inference. Whether it does also depends on hardware, kernels, and concurrent requests. Memory use and runtime must be measured separately.

Is a short test enough for a quantized model? The test must cover the intended tasks, including typical context and answer lengths. Differences can appear even in short answers. Passing a short test therefore does not establish that a long run will work well too.

Sources

• Frantar et al., GPTQ, 2022/2023: weight quantization and layer-by-layer reconstruction.
• Lin et al., AWQ, 2023/2024: scaling important channels and calibration data.
• Micikevicius et al., FP8 Formats for Deep Learning, 2022; NVIDIA, Hopper Architecture.
• Li et al., Evaluating Quantized Large Language Models, 2024: task- and model-dependent quality, including long contexts.
• Liu et al., Quantization Hurts Reasoning?, 2025: nearly lossless four-bit weight quantization on the reasoning models studied.
• ParoQuant, 2025: deviations across long reasoning chains.
• Dutta et al., Accuracy is Not All You Need, 2024: changed individual decisions despite similar average quality.
• Zhao et al., QSPEC, EMNLP 2025, Appendix A.6: runtime comparisons of FP16 and weight quantization.

Part of the series AI Engineering Explained. Related: KV Cache, Hosting Your Own AI Models or Renting?, and Continuous Batching.