VRAMfit guide · updated 2026-06-27
LLM quantization explained: Q4_K_M vs Q8_0 vs FP16
In short: quantization stores a model's weights in fewer bits. Going from FP16 (16-bit) to Q4_K_M (~4.8-bit) cuts VRAM roughly 3x with a small, usually acceptable quality loss. Q4_K_M is the default most people should start with; use Q8_0 or higher when you have VRAM to spare, and Q5/Q6 as middle ground.
What do the names mean?
GGUF quantization names encode bits-per-weight and method: Q4_K_M is a 4-bit "K-quant" (medium variant), Q8_0 is 8-bit, FP16 is the unquantized half-precision original. Lower bits = smaller and faster to load, with gradually increasing quality loss; modern K-quants keep 4-bit remarkably close to the original.
How much VRAM does each level need?
Total VRAM (weights + KV cache + overhead) at a 8,192-token context:
| Quantization | Bits/weight | Llama 3.1 8B | Llama 3.1 70B |
|---|---|---|---|
| Q4_K_M | 4.5 bits | 6.4 GB | 41.2 GB |
| Q5_K_M | 5.5 bits | 7.4 GB | 50.0 GB |
| Q6_K | 6.5 bits | 8.4 GB | 58.8 GB |
| Q8_0 | 8 bits | 9.9 GB | 71.9 GB |
| FP16 | 16 bits | 17.9 GB | 141.9 GB |
Which should you pick?
VRAMfit's rule, used by its recommender: take the highest-quality quantization that still fits comfortably on your card. The fit board computes this per model; every model profile shows a quant-by-quant fit ladder.
Frequently asked questions
Is Q4 quantization good enough?
For most chat and coding use, yes - modern K-quants at ~4 bits retain most quality. Tasks sensitive to precision (heavy math, long-form reasoning) benefit from Q6_K or Q8_0 when VRAM allows.
Does quantization make inference faster?
Generally yes: decoding is memory-bandwidth bound, and fewer bits means fewer bytes to stream per token.
What is the difference between GGUF and the model itself?
GGUF is the container format used by llama.cpp and Ollama; the quantization level (Q4_K_M, Q8_0...) describes how the weights inside are compressed.