Model weights are numbers, and numbers can be stored at different precisions. Training generally happens in 32-bit or 16-bit floating point. Quantization asks whether inference needs that much, and the answer is usually no. Dropping to 8-bit integers halves memory against fp16; dropping to 4-bit quarters it. The weights get coarser, and for most tasks the model barely notices.
The memory arithmetic is what makes this compelling rather than academic. A 70-billion-parameter model at fp16 needs roughly 140 GB just for weights, which means two 80 GB accelerators before you have allocated a single byte of KV cache. The same model at 4-bit lands near 35 GB and fits on one card with room to serve. That is not a cost optimization, it is the difference between a deployment being possible on available hardware and not. The same logic is why capable models run on laptops at all.
The methods divide into post-training quantization, which converts an already-trained model and is what almost everyone uses, and quantization-aware training, which simulates low precision during training and preserves more quality at much higher cost. In practice you will meet the formats rather than the theory: GPTQ and AWQ for GPU serving, GGUF for llama.cpp and local inference, bitsandbytes for quick experiments, and native FP8 support on newer hardware. Weights are not the only target either. Quantizing the KV cache is an independent lever and often the more valuable one for long-context serving, since cache memory is what actually bounds concurrency.
What degrades is the part teams get wrong, because the average benchmark score is reassuring and misleading. Aggregate accuracy on standard benchmarks often drops only a point or two at 4-bit. The damage concentrates in the tail: multi-step reasoning chains where small errors compound, adherence to structured output formats, long-context retrieval accuracy, rare languages, and reliable tool-call argument formatting. These are exactly the behaviors production agents depend on and exactly the ones short benchmark prompts do not exercise. The recurring failure is a team validating a quantized model on a hundred short questions, seeing no regression, shipping it, and then watching JSON parse errors climb in an agent loop.
Why It Matters
Quantization determines whether a given model fits your hardware budget, and for self-hosted deployments it is usually the single largest cost lever available. It is also a decision that must be validated against your own evals rather than published benchmark deltas, because the capabilities that degrade first are the agentic ones that public benchmarks measure least. Getting it right can cut serving cost by three quarters; getting it wrong produces intermittent failures that are hard to attribute back to a precision change made weeks earlier.
Example
A team self-hosting a 70B model moves from fp16 to 4-bit AWQ to fit two concurrent workloads on one GPU. Their standard eval set of short question-answer pairs shows a 1.2 point drop, which they accept. Two weeks later the agent pipeline shows a rising rate of malformed tool calls, roughly one in twenty against one in four hundred before. The cause is the quantized model producing subtly invalid JSON on longer tool schemas. They keep the quantized model for the summarization path and revert to fp16 for the tool-calling path, capturing most of the savings without the failure mode.