GGUF quantization format names — Q4_K_M, Q5_K_S, Q8_0, IQ2_XS — are cryptic at first glance but follow a consistent pattern once you understand the system. Choosing the right quantization is one of the most impactful decisions in local LLM deployment: it determines memory usage, inference speed, and output quality. This guide decodes the naming system, explains what each format actually does, and gives practical guidance on which to use.
What Quantization Is
Language model weights are stored as floating-point numbers. A full-precision model uses 32-bit floats (F32) or 16-bit floats (F16/BF16) — each weight taking 4 or 2 bytes respectively. A 7B parameter model in F16 weighs about 14GB. Quantization reduces the number of bits used per weight, shrinking the model. A 4-bit quantization reduces each weight to 4 bits — one-quarter the size of F16 — resulting in a 7B model that fits in about 4GB. The trade-off is precision: using fewer bits to represent each weight means rounding to the nearest representable value, introducing a small error. The key engineering question is how to distribute those bits most efficiently to minimise quality loss.
The GGUF Naming System Decoded
GGUF quantization names have up to four components: the prefix (Q or IQ), the bit count, the quantization method (K = k-quants, 0 = simple), and the size variant (S = small, M = medium, L = large, XL = extra large). Breaking down common names:
Q4_K_M: Q (quantized) + 4 (4-bit) + K (k-quants method) + M (medium variant). The K-quant method uses block-wise quantization with different precision for different weight matrices — more bits for weights that have more impact on quality, fewer bits for less impactful weights. The S/M/L variants allocate bits differently: S is smallest and fastest, M is the balance point, L is largest with best quality.
Q5_K_M: Same as Q4_K_M but 5-bit — slightly larger and slower, with meaningfully better quality.
Q8_0: 8-bit quantization using the simpler “0” method (not K-quants). Very close to F16 quality at half the size. The “0” method is uniform — all weights get the same precision. Slower than Q4 variants due to larger memory footprint.
IQ2_XS, IQ3_S: “Importance quantization” — a newer method that allocates bits based on the importance of each weight to the model output. IQ methods achieve better quality at the same bit width than K-quants for aggressive quantizations (2-3 bit). The XS/S/M variants in IQ follow the same size pattern as K-quants.
Quality vs Size Trade-offs
The quality ordering from best to worst (for a given model family) roughly follows bit count: F16 > Q8_0 > Q6_K > Q5_K_M > Q5_K_S > Q4_K_L > Q4_K_M > Q4_K_S > IQ4_XS > Q3_K_L > Q3_K_M > Q3_K_S > IQ3_M > IQ3_S > IQ2_M > IQ2_XS > IQ2_XXS. The perceptual quality difference between F16 and Q4_K_M is small for most practical tasks — benchmarks typically show 1-3% performance degradation on standard evaluations, which translates to outputs that are almost indistinguishable in everyday use. The quality difference between Q4_K_M and Q2 variants is much more noticeable — outputs from aggressively quantized models are clearly worse on complex reasoning tasks, even if they remain usable for simple tasks.
Figure 1 — GGUF Quantization Quick Reference (7B model)
The Practical Recommendation
For most users and most models: Q4_K_M is the default choice. It offers very good quality (indistinguishable from F16 for most everyday tasks), the best generation speed among common quantizations (because the smaller model fits more efficiently in GPU memory and the 4-bit arithmetic is hardware-efficient), and a reasonable file size. If you have memory headroom and care about quality on nuanced tasks, step up to Q5_K_M. If you are memory-constrained and need to fit a model that otherwise would not fit, step down to Q4_K_S or IQ4_XS. Q8_0 is worth considering only if you have consistently noticed quality problems with Q4 variants on your specific use case — for most interactive applications, the quality difference is not worth the doubled memory cost and slower generation speed. Anything below Q4 (Q3, Q2, IQ2) should be reserved for situations where the model genuinely cannot fit in your available memory at Q4 size and you accept noticeable quality degradation as the trade-off.
How Ollama Chooses Quantization
When you run ollama pull llama3.1, Ollama automatically selects the quantization based on your available memory — it tries to find the best quality quantization that fits comfortably. This usually lands on Q4_K_M for most configurations. You can explicitly request a specific quantization using the tag system: ollama pull llama3.1:8b-instruct-q5_k_m pulls the Q5_K_M variant, ollama pull llama3.1:8b-instruct-q8_0 pulls Q8_0. Check ollama.com for the available tags for each model — not all quantizations are available for all models, and the naming conventions vary slightly between model families.
Differences Between Model Families
Not all models are equally sensitive to quantization. Models that rely heavily on precise weight values for their capability (smaller models, models with unusual architectures) show more quality degradation from aggressive quantization than larger models. A 70B model at Q4_K_M often performs close to its F16 baseline because the sheer number of parameters provides redundancy that absorbs the quantization error. A 3B model at Q4_K_M shows more relative degradation because there are fewer parameters to absorb the error. If you are using a small model (3B or below) and notice quality issues, stepping up to Q5_K_M or Q6_K can meaningfully improve outputs in a way that may not be as noticeable on larger models. For large models (70B+), Q4_K_M is almost always the right choice — the quality difference from higher quantization is minimal and the memory savings are significant.
Context Windows and Quantization Interaction
Quantization affects model weights but not the KV cache, which stores the attention state for the current context. The KV cache is stored in F16 by default in most inference engines. At long context lengths (16K+ tokens), the KV cache can consume as much memory as the model weights themselves. Flash Attention reduces KV cache memory by 30-50% at long context by computing attention without materialising the full cache. The practical implication: at short contexts, quantization dominates memory consumption and choosing Q4_K_M over Q8_0 halves your memory use. At very long contexts (32K+), the KV cache becomes significant too, and enabling Flash Attention is as important as quantization choice for fitting models with long context into available memory.
When Quantization Choice Actually Matters
For the tasks most people use local LLMs for — interactive chat, writing assistance, code help, summarisation — the quality difference between Q4_K_M and Q5_K_M is very difficult to notice in outputs. The difference between Q4_K_M and Q8_0 is similarly small for these common tasks. The cases where quantization choice noticeably matters: complex multi-step mathematical reasoning (aggressive quantization introduces more errors in long arithmetic chains), precise JSON or code generation from structured prompts (more quantization errors can push outputs toward malformed formats), and tasks where the model is already at the edge of its capability (a 7B model doing difficult reasoning tasks benefits more from higher quantization than the same model doing simple summarisation). If you are using a 13B+ model for everyday tasks, quantization choice rarely matters perceptibly — use Q4_K_M and focus your attention elsewhere. If you are using a 7B model for demanding analytical tasks, experimenting with Q5_K_M or Q6_K may produce noticeably better results at the cost of slightly slower generation and higher memory use.
Finding and Downloading Specific Quantizations
On Hugging Face, the most prolific GGUF re-quantizers are Bartowski, LM Studio (the tool, which uploads model quants), and the original model authors. Search Hugging Face for your model name plus “GGUF” to find the available quantizations. Bartowski in particular maintains a comprehensive library of GGUF quantizations for most major models, updating within days of new releases. The file naming convention is consistent: MODEL-NAME-SIZE-QUANTIZATION.gguf. For models in the Ollama library, run ollama tags MODEL or check the model page on ollama.com to see all available quantization tags. Downloading directly from Hugging Face with huggingface-cli gives you more control over which specific quantization you get compared to the automatic selection Ollama does, which is useful when you specifically need Q5_K_M or Q8_0 rather than the default Q4_K_M that Ollama typically selects.
A Note on Future Quantization Methods
Quantization methods continue to evolve rapidly. The IQ (importance quantization) series introduced in 2024-2025 achieved better quality at 2-3 bits than previous methods, and further improvements are likely as the research community continues to find more efficient ways to represent model weights. The GGUF format is designed to accommodate new quantization types — new formats are added as llama.cpp updates and automatically become available in all tools built on it, including Ollama and llama-cpp-python. The practical implication: if you download a model with a quantization type you have not seen before, it is likely a newer format that your version of Ollama or llama.cpp may or may not support yet. Updating to the latest Ollama version or rebuilding llama.cpp from source adds support for new formats quickly. The naming conventions will continue to follow the same general pattern described in this guide, so the framework for understanding any new format name remains valid even as the specific formats evolve.