Quantization stores a model's weights using fewer bits. It is the reason a model that needs a datacentre GPU at full precision can run on a laptop, and a major reason serving costs keep falling. The trade is always the same: less memory and more speed in exchange for some loss of quality. The interesting part is which quality you lose, and how much it matters for your task.
This guide explains what quantization does, the main methods (GPTQ, AWQ, GGUF k-quants and bitsandbytes), what typically degrades at 8, 4 and 3 bits, and a practical method for choosing a level.
What quantization actually does
Model weights are normally stored as 16-bit floating-point numbers (FP16 or BF16). Quantization maps them onto a smaller set of values: 256 possible values at 8 bits, 16 at 4 bits, 8 at 3 bits. Each group of weights shares a scale (and often an offset) so that the small integer values can be converted back to approximately the original numbers during computation.
Two consequences follow:
- Memory shrinks roughly in proportion to bits. An 8B model needs about 16 GB at 16 bits, about 8 GB at 8 bits and roughly 4.5 to 5 GB at 4 bits, once scaling data is included.
- Generation gets faster, because producing each token requires reading all the weights from memory, and smaller weights take less time to read.
The cost is rounding error. Each weight is slightly wrong, and those errors accumulate as signals pass through dozens of layers.
Why some weights are harder than others
Rounding error is not spread evenly. Research on large models found that a small number of outlier features, values much larger than the rest, carry a disproportionate share of the model's behaviour. Dettmers et al. (2022), "LLM.int8()," showed that naive 8-bit quantization broke down for large models because of these outliers, and that keeping them in higher precision solved the problem.
Modern methods all deal with this in some way: by protecting important weights, by choosing scales carefully per small group of weights, or by measuring which weights matter most using sample data.
The main methods
| Method | Typical use | Key idea |
|---|---|---|
| bitsandbytes (LLM.int8, NF4) | Quick loading in Python, QLoRA fine-tuning | 8-bit with outlier handling; 4-bit NormalFloat format |
| GPTQ (Frantar et al., 2022) | GPU inference at 4 bits | Quantizes layer by layer, correcting errors using calibration data |
| AWQ (Lin et al., 2023) | GPU inference at 4 bits | Protects the weights that matter most for activations |
| GGUF k-quants (llama.cpp) | CPU, Apple Silicon and mixed inference | Block-wise quantization with mixed precision per tensor |
GGUF files are named by their scheme, for example Q8_0, Q5_K_M, Q4_K_M and Q3_K_S. The number is the approximate bits per weight; K-variants use smarter block-wise scaling; the S/M/L suffix indicates how many sensitive tensors are kept at higher precision.
What you lose at each level
The figures here are general patterns reported widely across models and methods, not measurements from one benchmark. Larger models usually tolerate quantization better than small ones, because they have more redundancy.
8 bits
For most tasks, 8-bit quantization is very hard to distinguish from 16-bit. It is the safe choice when memory allows and you need maximum fidelity, for example for evaluation baselines or precise extraction.
5 and 6 bits
Very small quality loss for most tasks. Often the best choice when you have a little extra memory and care about exact recall or code.
4 bits
The most popular level for local and cost-sensitive inference. Fluent writing, summarisation and conversation usually survive well. Degradation shows up first in:
- exact recall: long numbers, rare names, precise quotations;
- multi-step arithmetic and code, where one wrong token breaks the result;
- long contexts, where small errors are amplified across many tokens;
- less common languages, which have less redundancy in the model.
3 bits and below
Quality loss becomes visible in everyday use: more repetition, more factual slips, weaker instruction following. Sometimes worthwhile to fit a much larger model into limited memory, because a large model at 3 bits can still beat a small model at 8 bits, but test carefully.
Measure on your task, not on perplexity
Perplexity, a measure of how well the model predicts text, is the standard quick check for quantized models. It is a fine sanity check and a poor decision tool. Two variants with nearly identical perplexity can differ sharply on structured extraction or code, because perplexity averages over all tokens while your task may depend on a few critical ones.
A better method:
- Pick 50 to 100 examples of your real task, with automatic checks where possible.
- Run them at 16-bit or 8-bit as the reference.
- Run the same examples at the candidate levels with identical settings.
- Compare pass rates, and read every example where the quantized version fails but the reference passes.
- Choose the smallest level whose failures you can accept.
Include several runs per example if you sample at non-zero temperature; otherwise random variation can hide or exaggerate differences.
Quantizing the KV cache and activations
Weights are not the only thing that can be stored with fewer bits. Two other targets matter in production:
- KV cache quantization. During generation, the model stores keys and values for every previous token. At long context lengths this cache can rival or exceed the size of the weights. Storing it at 8 bits, or even 4, lets a server handle longer contexts or more simultaneous users on the same hardware. The quality impact is usually small at 8 bits and more noticeable at lower precision on long-context tasks.
- Activation quantization. Some serving stacks also quantize the intermediate values (activations) so that matrix multiplications run in low-precision integer or floating-point formats such as INT8 or FP8, which modern GPUs accelerate. This mainly increases throughput on the server side.
If you use a hosted API, you will rarely control these settings directly, but they partly explain why the same open model can behave slightly differently from one provider to another. When results differ between providers, ask which precision they serve the model at.
Quantization, fine-tuning and distillation
Quantization combines well with other efficiency techniques. QLoRA (Dettmers et al., 2023) fine-tunes adapters on top of a 4-bit base model, making fine-tuning possible on a single GPU; see LoRA in plain English. Distillation produces smaller models that are then often quantized for deployment; see distillation explained for builders.
If you are running models locally, our step-by-step guide to running an 8B LLM on a laptop covers memory, tools and speed estimates. And before you squeeze a big model into a small machine, check whether a small model already handles the job.
Frequently asked questions
What is quantization in LLMs?
Quantization stores model weights with fewer bits, for example 4 bits instead of 16, which reduces memory use and speeds up inference at the cost of small rounding errors that can reduce quality.
Is 4-bit quantization good enough?
For many tasks, such as chat, summarisation and rewriting, 4-bit models are close to full precision. Tasks that depend on exact recall, arithmetic or code are more sensitive, so test 4-bit against a higher-precision reference on your own examples.
What is the difference between GPTQ, AWQ and GGUF?
GPTQ and AWQ are quantization methods mainly used for GPU inference at 4 bits; GPTQ corrects errors layer by layer, and AWQ protects activation-sensitive weights. GGUF is the file format used by llama.cpp and related tools, with block-wise "k-quant" schemes suited to CPUs and Apple Silicon.
Should I use a bigger quantized model or a smaller full-precision model?
Often the bigger quantized model wins at the same memory budget, down to around 4 bits. At 3 bits and below the trade-off becomes less predictable, so compare both on your own tasks.
