Quantization stores a model’s weights, and often its activations, in fewer bits, for example 8-bit integers (INT8) instead of 32-bit floats (FP32). The model then takes less memory and can run faster on hardware that supports low-precision math. INT8 weights take a quarter of the space of FP32 weights, usually at a small cost in accuracy.
Below: why models need slimming down, how the float-to-integer mapping actually works (with a worked example), the main methods (PTQ, dynamic quantization, QAT), and where quantization falls short.
Why Do AI Models Need to Be Slimmed Down?
Modern models, especially large language models, have billions of parameters. Their scale brings three practical problems:
- High computational costs. Large models need powerful accelerators and draw a lot of power.
- Slow inference. Real-time applications such as voice assistants or driver-assistance systems need fast responses, and big models struggle to deliver them.
- Deployment constraints. Phones, cameras and IoT devices have limited memory and compute.
The problem: how do we keep a large model’s capabilities while making it light and efficient?
The solution: quantization. The simple memory math below shows why it matters.
How much memory do the weights need?
Each parameter costs a fixed number of bytes, depending on its format. For a model with 7 billion parameters (weights only, not counting activations or runtime overhead):
| Format | Bits per parameter | Bytes per parameter | Weights for 7B parameters |
|---|---|---|---|
| FP32 | 32 | 4 | ~28 GB |
| FP16 / BF16 | 16 | 2 | ~14 GB |
| INT8 | 8 | 1 | ~7 GB |
| INT4 | 4 | 0.5 | ~3.5 GB |
Going from FP32 to INT8 cuts weight storage by 75%. Going to INT4 cuts it by 87.5%. That is often the difference between a model that fits on a device and one that doesn’t.
What Is Quantization?
Quantization reduces the numerical precision of a model’s parameters (weights) and intermediate results (activations). It moves them from high-precision formats like FP32 to lower-precision formats like FP16, INT8 or INT4.
Simplified Analogy
It is like compressing an image:
- Original image (high precision): high resolution, large file, slow to load.
- Compressed image (low precision): smaller file and slightly lower quality, but faster and more efficient.
You lose some detail, but for most purposes the picture is still clearly the same picture.
How Does It Work?
The key idea is to represent each value with fewer bits while losing as little information as possible. There are two steps.
1. Numerical Range Mapping
First, find the range of values to cover (for example, the smallest and largest weight in a layer). Then map that range onto the integer range of the target format. For unsigned 8-bit integers, that range is 0 to 255.
2. Float-to-Integer Conversion
Two numbers describe the mapping:
- Scale: how much one integer step is worth in float terms.
scale = (max − min) / 255 - Zero point: the integer that represents 0.0 exactly.
zero_point = round(−min / scale)
To quantize a value: q = clip(round(x / scale) + zero_point, 0, 255)
To get an approximate float back (dequantize): x̂ = (q − zero_point) × scale
Worked example: range [−2.0, 2.0] to [0, 255]
With this range, scale = 4.0 / 255 ≈ 0.01569 and zero_point = round(127.5) = 128.
| Original value (FP32) | Quantized (INT8, unsigned) | Dequantized back | Error |
|---|---|---|---|
| −2.0 | 0 | ≈ −2.0078 | ≈ 0.0078 |
| 0.0 | 128 | 0.0 | 0 |
| 1.0 | 192 | ≈ 1.0039 | ≈ 0.0039 |
| 2.0 | 255 (clipped) | ≈ 1.9922 | ≈ 0.0078 |
Two things to notice:
- 0.0 maps to an integer exactly. This matters because zeros (padding, ReLU outputs) are common in neural networks.
- Every other value picks up a small rounding error, at most about half a step. Quantization trades exactly that error for a 4x smaller representation.
Symmetric vs asymmetric, per-tensor vs per-channel
- Asymmetric quantization (as above) uses a zero point, so it fits ranges that aren’t centered on zero.
- Symmetric quantization fixes the zero point at 0 and uses a range like [−127, 127]. It is simpler and is common for weights.
- Per-tensor quantization uses one scale for a whole tensor.
- Per-channel quantization uses a separate scale for each output channel. It usually keeps more accuracy, because different channels can have very different ranges.
Result: the model runs at lower precision but keeps the information it needs to make accurate predictions.
Core Processes and Methods
1. Weight Quantization
- What it does: converts model parameters from FP32 (or FP16) to INT8 or lower.
- Effect: greatly reduces storage and memory use, but may introduce small errors.
- LLMs: weight-only quantization to 4 or 8 bits is a popular way to run them on limited hardware. Methods such as GPTQ and AWQ are designed for this.
2. Activation Quantization
- What it does: quantizes the intermediate results computed during inference.
- Effect: with both weights and activations in INT8, the matrix math itself can run in integer arithmetic. This further reduces compute demands, but the hardware and runtime must support it.
3. Post-Training Quantization (PTQ)
- What it does: quantizes an already-trained model without retraining it.
- Static PTQ: runs a small calibration dataset through the model to choose scales and zero points for activations ahead of time.
- Effect: fast and cheap to apply. Accuracy is usually close to the original at INT8, but can drop more at lower bit widths.
4. Dynamic Quantization
- What it does: quantizes the weights ahead of time. Activation ranges are computed on the fly for each input during inference.
- Effect: needs no calibration data and is easy to apply. It works well for models dominated by linear layers, such as many Transformer and LSTM models. It adds a little runtime overhead for computing activation ranges.
5. Quantization-Aware Training (QAT)
- What it does: simulates quantization during training (or fine-tuning), so the model learns to tolerate low-precision calculations.
- Effect: usually keeps higher accuracy than post-training quantization, especially at low bit widths. The cost is extra training time and the need for training data.
| Method | Needs retraining? | Needs calibration data? | Typical accuracy | Effort |
|---|---|---|---|---|
| Dynamic quantization | No | No | Good | Lowest |
| Static PTQ | No | Yes (small set) | Good at INT8 | Low |
| QAT | Yes | Training data | Highest | Highest |
Learn more from this article:
Real-World Applications
1. Voice Assistants on Mobile Devices
Voice assistants need fast responses, but large speech models drain the battery. A quantized speech recognition model is smaller and cheaper to run, so it can run locally on a phone. That cuts latency and avoids a round trip to a server.
2. Image Classification on Edge Devices
Security cameras and other edge devices process continuous video with little memory and compute. Converting a vision model such as ResNet from FP32 to INT8 shrinks its weights to a quarter of their size. It can also speed up inference on chips with INT8 support.
3. Running LLMs on Your Own Machine
4-bit and 8-bit quantized versions of open LLMs are what make it practical to run them on laptops and single workstations. Without quantization, many of these models would not fit in memory.
4. Real-Time Object Detection in Autonomous Vehicles
Autonomous driving needs object detection that is both accurate and low-latency. Quantization-aware training lets detection models keep their precision while running faster, so the system can respond sooner to sudden obstacles.
Limitations
Despite its benefits, quantization has real limits:
- Accuracy loss. Rounding errors add up. That matters in high-accuracy tasks like medical diagnostics, and it gets worse at very low bit widths (4 bits and below).
- Outliers. A few extreme values can stretch the range so that everything else gets squeezed into very few integer steps. LLM activations are known for this. Per-channel scales and outlier-aware methods help.
- Hardware and runtime dependency. Memory savings are automatic, but speed gains only appear when the hardware and inference runtime have efficient low-precision kernels. On unsupported hardware, a quantized model may simply be dequantized back to float, with little or no speedup.
- Not every layer quantizes well. Some layers are more sensitive than others, and so are some complex or multimodal architectures. They are often kept at higher precision.
Quantization vs Knowledge Distillation vs Pruning
These three techniques all make models cheaper to run, and they can be combined:
| Technique | What it changes | Model architecture |
|---|---|---|
| Quantization | The number of bits per value | Same |
| Knowledge distillation | Trains a smaller “student” model to mimic a larger “teacher” | New, smaller model |
| Pruning | Removes weights, neurons or channels that matter least | Same, but sparser or slimmer |
A common pattern is to distill a large model into a smaller one, then quantize the smaller one. See Knowledge Distillation for how distillation works and how it pairs with quantization.
The Future
- Mixed precision computing: combining low-precision (e.g., INT8 or INT4) and higher-precision (e.g., FP16/FP32) operations in the same model to balance speed and accuracy.
- Better quantization-aware training: training methods that automatically shape weight distributions so they survive quantization with less loss.
- Lower bit widths: formats at 4 bits and below, together with chips and kernels designed for them, to reduce memory and energy use further.
One-Line Summary
Quantization moves AI models from “high precision” to “high efficiency.” It trades a small, controlled rounding error for models that are several times smaller and often faster, which makes it an essential tool for deploying modern AI.
FAQ
Does quantization reduce accuracy? Usually a little. At INT8 the drop is often small. At 4 bits or below it can be larger, and methods like QAT, per-channel scales and outlier handling help close the gap.
How much smaller is an INT8 model than an FP32 model? The weights take a quarter of the space, because INT8 uses 1 byte per value and FP32 uses 4. Total memory use also depends on activations and runtime overhead.
What is the difference between PTQ and QAT? Post-training quantization (PTQ) quantizes a finished model without retraining, sometimes using a small calibration set. Quantization-aware training (QAT) simulates quantization during training, so the model adapts to it. QAT usually keeps more accuracy but costs more effort.
Does quantization always make inference faster? No. It always reduces memory, but speedups depend on whether your hardware and runtime have efficient low-precision operations.
Is quantization the same as knowledge distillation? No. Quantization keeps the same model and stores its numbers in fewer bits. Distillation trains a new, smaller model to imitate a larger one. They are often used together.
Keep Reading
- To put a quantized model into production, see how Ray Serve handles model serving with autoscaling and batching: Ray Serve
- How knowledge distillation pairs with quantization: Knowledge Distillation
- More background on foundation models vs task models: Foundation Models Explained
- More posts in the AI Insights series: Daily AI Insight
Thanks for reading!