Quantized LLMs Explained: Q4 vs Q8 vs FP16

Quantization has emerged as the breakthrough technique that makes running powerful language models on consumer hardware practical. Without quantization, a 7-billion parameter model would require 28GB of RAM at full precision—placing it beyond the reach of most users. With 4-bit quantization, that same model runs comfortably in 6GB, transforming accessibility completely. Yet despite its importance, quantization remains poorly understood, with conflicting information about quality tradeoffs and confusion about which quantization level to choose.

This comprehensive guide demystifies quantization by explaining what actually happens when you quantize a model, examining the real quality differences between 4-bit, 8-bit, and 16-bit precision, and providing practical guidance for selecting the right quantization level for your specific needs. Whether you’re trying to maximize quality on limited hardware or optimize performance, understanding quantization enables informed decisions that dramatically impact your local LLM experience.

What Quantization Actually Does to Model Weights

Before comparing specific quantization levels, understanding the fundamental process clarifies why quantization works and what tradeoffs it involves.

The Basics of Neural Network Precision

Neural networks store knowledge in parameters—numerical weights and biases that determine how the network transforms inputs into outputs. During training, these parameters are typically stored as 32-bit floating-point numbers (FP32), providing extremely high precision with about 7 decimal digits of accuracy.

For inference (actually using the trained model), this extreme precision proves unnecessary. Research has shown that models maintain performance with significantly reduced precision. The parameters don’t need perfect accuracy—they need to be “close enough” to their original values to produce similar outputs.

This insight enables quantization: converting high-precision floating-point numbers into lower-precision representations that consume less memory and compute faster, while maintaining acceptable model performance.

How Quantization Reduces Precision

Imagine you’re measuring distances with a ruler. A ruler marked in millimeters provides high precision, but for many purposes, a ruler marked only in centimeters suffices. You lose some precision, but measurements remain useful for most tasks.

Quantization applies this same concept to model weights. A 32-bit floating-point number can represent an enormous range of values with high precision. An 8-bit integer can represent only 256 distinct values (-128 to 127 typically). A 4-bit integer represents just 16 values.

The quantization process maps the continuous range of floating-point values to this smaller set of discrete values. A sophisticated mapping function ensures the most important numerical ranges get more resolution while less critical ranges accept more compression.

Modern quantization techniques don’t simply round numbers to the nearest integer. They employ complex schemes that:

Calibrate value ranges by analyzing which numerical ranges the model’s weights actually occupy, focusing precision where it matters most.

Use asymmetric quantization where different parts of the model get different quantization schemes based on their sensitivity to precision loss.

Apply group-wise quantization that quantizes small groups of weights together, optimizing the mapping for each group independently.

Implement mixed-precision strategies where critical layers maintain higher precision while less sensitive layers accept aggressive compression.

These sophisticated techniques explain why modern 4-bit quantization maintains impressive quality despite representing weights with just 16 possible values.

FP16: The High-Quality Baseline

Sixteen-bit floating-point (FP16 or half-precision) serves as the de facto standard for model distribution and the baseline against which other quantization levels are measured.

Technical Characteristics

FP16 uses 16 bits (2 bytes) per parameter, representing numbers with approximately 3-4 decimal digits of precision. While this represents half the precision of FP32, it proves more than sufficient for model inference in virtually all scenarios.

The format allocates 1 bit for sign, 5 bits for exponent, and 10 bits for fraction, enabling it to represent a wide range of values from very small to very large while maintaining reasonable precision across that range.

Memory and Performance Profile

A 7-billion parameter model at FP16 precision requires approximately 14GB of memory (7 billion parameters × 2 bytes). This calculation includes only the model weights themselves—actual memory usage increases with overhead from the inference framework, operating system, and activation memory during computation.

Performance-wise, FP16 offers the fastest inference among precision formats on modern GPUs that include dedicated FP16 hardware acceleration. Many NVIDIA GPUs since the Pascal generation provide 2-4x higher throughput for FP16 operations compared to FP32, making FP16 attractive for both memory and speed reasons.

Quality Characteristics

FP16 maintains essentially perfect quality compared to the original FP32 models. The precision loss from FP32 to FP16 is imperceptible in practice—you would be hard-pressed to distinguish FP16 outputs from FP32 outputs in blind tests.

This makes FP16 the quality reference point: when we discuss quality loss from quantization, we measure against FP16 rather than FP32, since FP16 represents the practical quality ceiling for inference.

When to Choose FP16

Select FP16 when you have sufficient memory (typically 32GB+ system RAM or 16GB+ VRAM for 7B models) and want absolute maximum quality with best GPU performance. The format makes sense for:

• Production applications where quality cannot be compromised
• Research scenarios requiring reproducible results closest to original models
• Systems with abundant memory resources
• Situations where GPU memory bandwidth is the performance bottleneck

The primary downside is memory consumption—FP16 models require roughly 2x the memory of 8-bit and 4x the memory of 4-bit quantization.

Q8: The Sweet Spot for Many Use Cases

Eight-bit quantization (Q8) represents a compelling middle ground—halving memory requirements compared to FP16 while maintaining quality so close to the original that differences are difficult to detect.

Technical Implementation

Q8 represents each parameter using 8 bits (1 byte), typically as a signed integer ranging from -128 to 127. The quantization process involves determining the optimal mapping from the continuous floating-point range to these 256 discrete integer values.

Advanced Q8 implementations use per-channel or per-group quantization, where different portions of the model use different scaling factors. This granular approach captures more nuance than naive quantization that applies the same scale across the entire model.

Memory and Performance

A 7B model at Q8 requires approximately 7GB of memory—exactly half of FP16. This reduction proves transformative for accessibility. A system with 16GB RAM can comfortably run Q8 models that would strain or fail at FP16.

Performance characteristics depend on hardware. On CPUs, Q8 inference is slightly faster than FP16 due to reduced memory bandwidth requirements and faster integer operations. On GPUs, the picture is more nuanced—some GPUs have optimized FP16 hardware that outperforms integer operations, while others show Q8 advantages.

The key performance benefit comes from memory bandwidth reduction. Moving half the data means less time waiting for memory transfers, which often bottlenecks LLM inference more than compute capability.

Quality Preservation

This is where Q8 shines. The quality difference between Q8 and FP16 is minimal to nonexistent for most practical purposes. In careful testing across various benchmarks, Q8 models typically score 98-99% of FP16 performance.

For everyday usage—conversations, coding assistance, content generation, analysis—you won’t notice quality differences between Q8 and FP16. The additional precision of FP16 matters in specific edge cases or extremely sensitive applications, but not in typical use.

Some users report Q8 models feeling slightly less “creative” or producing marginally more conservative outputs, but these differences are subtle and inconsistent. Blind tests often show users cannot reliably distinguish Q8 from FP16 outputs.

When Q8 Makes Sense

Choose Q8 when you want to maximize quality while gaining meaningful memory savings. Ideal scenarios include:

• Systems with 16-32GB RAM that couldn’t fit FP16 models comfortably
• Applications where quality matters but FP16 memory requirements are prohibitive
• Production systems requiring reliability and quality with reasonable hardware
• Situations where you want larger models (13B at Q8 instead of 7B at FP16)

Q8 represents the best quality-to-memory ratio for users who can afford slightly larger models but want to ensure quality remains excellent.

Q4: Maximum Compression with Surprising Quality

Four-bit quantization represents the aggressive end of practical quantization, compressing models to 25% of FP16 size while maintaining usable—often impressive—quality.

The Challenge of 4-bit Representation

Representing neural network weights with only 4 bits (16 possible values) seems impossibly limiting. Early naive 4-bit quantization produced severely degraded models. The breakthrough came with sophisticated quantization methods that maximize information preservation within severe constraints.

Modern Q4 techniques like GPTQ (Gradient-based Post-Training Quantization), AWQ (Activation-aware Weight Quantization), and GGUF’s various Q4 variants employ clever strategies:

Importance-weighted quantization applies more precision to weights that have greater impact on outputs, accepting more degradation in less critical weights.

Group-wise quantization divides weights into small groups (typically 32-128 weights) and optimizes quantization parameters for each group independently, capturing local patterns that global quantization would miss.

Outlier handling identifies and treats weights with extreme values specially, preventing them from skewing the quantization scale for all other weights.

Mixed-precision within layers maintains higher precision for activation functions and certain critical operations while aggressively compressing less sensitive components.

Memory and Performance Benefits

A 7B model at Q4 requires approximately 3.5-4GB of memory—one-quarter of FP16. This dramatic reduction enables running capable models on modest hardware. An 8GB RAM system can run 7B Q4 models comfortably, and 16GB systems can run 13B or even 30B models at Q4.

Performance is excellent. The smaller memory footprint means faster loading, less memory bandwidth consumption, and more available resources for other applications. Q4 models often generate tokens faster than FP16 simply because less data must move through the memory subsystem.

Quality Considerations

Here’s where nuance matters. Q4 quality depends heavily on several factors:

The quantization method used makes enormous differences. GPTQ Q4 typically outperforms simple linear Q4. AWQ provides another quality boost for certain model architectures. The difference between a well-quantized Q4 model and poorly-quantized one is dramatic.

The model architecture affects how well it quantizes. Some models are naturally more robust to quantization than others. Llama-2 models quantize well; other architectures may be more sensitive.

The specific task influences perceived quality loss. Simple conversations and Q&A might show minimal degradation, while complex reasoning, mathematics, or very technical domains might reveal Q4 limitations more clearly.

The Q4 variant matters significantly. Q4_K_M (medium), Q4_K_S (small), and Q4_0 represent different quality-size tradeoffs within the 4-bit space.

In practical terms, modern Q4 models maintain 90-95% of FP16 quality for most tasks. You’ll notice some quality loss if you compare directly—responses might be slightly less nuanced, creativity might diminish marginally, complex reasoning might occasionally falter. But for many use cases, Q4 models remain highly capable and perfectly usable.

When Q4 Is the Right Choice

Select Q4 when memory is the primary constraint and you’re willing to accept small quality tradeoffs for dramatic memory savings:

• Systems with 8-16GB RAM that need to run capable models
• Mobile or embedded deployments where resources are strictly limited
• Wanting to run larger parameter models (30B at Q4 vs. 7B at FP16)
• Batch processing where slight quality loss is acceptable for efficiency
• Experimentation and development where rapid iteration matters more than perfection

Q4 has democratized local LLMs. Models that once required expensive workstations now run on laptops and even high-end smartphones, thanks to aggressive 4-bit quantization.

Practical Decision Framework: Choosing Your Quantization Level

Selecting the right quantization requires balancing memory constraints, quality requirements, and specific use cases.

Memory-First Decision Path

When RAM is your primary limitation, work backward from available memory:

8GB Total RAM: You have approximately 5-6GB available for models after OS overhead. This firmly places you in Q4 territory for 7B models. Q3 variants of smaller 3B models also work. Accept that quality will be the tradeoff for accessibility.

16GB Total RAM: With 12-13GB available for models, you gain flexibility. Q4 works comfortably for 7B models with room for multitasking. Q8 becomes viable for 7B models if you close background applications. Q4 enables running 13B models, opening access to more capable options.

32GB Total RAM: Memory constraints largely disappear for consumer-grade models. FP16 for 7B models, Q8 or FP16 for 13B models, and Q4 for 30B models all become practical. Choose based on quality preferences rather than hard constraints.

64GB+ Total RAM: Run essentially any publicly available model. FP16 for models up to 30B parameters, Q4 or Q8 for 70B models. Memory is no longer the bottleneck—CPU/GPU performance becomes the limiting factor.

Quality-First Decision Path

When quality matters most and you want minimal degradation:

Start with FP16 if your memory allows it. This provides baseline quality against which everything else is measured.

Try Q8 as your first compromise. In blind testing, most users cannot distinguish Q8 from FP16 for standard tasks. The 50% memory savings come with essentially no perceptible quality loss.

Use Q4 only when necessary. While modern Q4 quantization is impressive, quality loss is noticeable in direct comparison. Reserve Q4 for memory-constrained scenarios or when running larger models (30B at Q4) that wouldn’t fit at higher precision.

Avoid Q3 and lower unless experimenting or working with extremely constrained environments. Quality degradation at 3-bit becomes significant enough to impact practical utility for many tasks.

Task-Specific Recommendations

Different use cases tolerate different levels of quantization:

General Conversation and Q&A: Q4 performs excellently. The quality loss is minimal for straightforward dialogue, and the memory savings enable running larger, more capable models.

Code Generation: Q8 or FP16 recommended. Code requires precision—off-by-one errors or incorrect syntax appear more frequently with aggressive quantization. The 8-bit precision maintains code quality well.

Creative Writing: Q4 or Q5 work well. Creative tasks benefit from the randomness and variability that quantization introduces. Some writers prefer slightly quantized models for this reason.

Mathematical Reasoning: Q8 or FP16 strongly recommended. Mathematics is where quantization impact shows most clearly. Precision matters for calculations, and Q4 models struggle noticeably with complex math.

Document Summarization: Q4 performs adequately. Summarization doesn’t require the precision of math or code, making it a good candidate for aggressive quantization.

Technical Documentation and Analysis: Q8 preferred. Technical accuracy matters, and the additional precision of Q8 over Q4 prevents subtle errors in technical explanations.

Understanding Q4 Variants: Not All 4-bit is Equal

Within the Q4 quantization space, multiple variants offer different tradeoffs. Understanding these helps optimize for your specific needs.

Q4_K_M: The Balanced Choice

Q4_K_M (K-quant, medium) represents the most common Q4 variant. It uses mixed quantization where different layers and components get different bit allocations within the overall 4-bit budget.

Critical layers like attention mechanisms receive slightly higher precision, while less sensitive layers accept more aggressive compression. This strategic allocation maintains quality where it matters most.

File sizes typically run 3.5-4GB for 7B models. Quality sits in the middle of the Q4 range—better than Q4_0 or Q4_K_S but not quite reaching Q4_1 or Q5 levels.

Q4_K_S: Maximum Compression

Q4_K_S (small) pushes compression further, using slightly more aggressive quantization to reduce file size by another 10-15% compared to Q4_K_M.

A 7B model might compress to 3.2GB instead of 3.5GB. The quality difference from Q4_K_M is subtle but noticeable in extended use—slightly more repetition, marginally less coherent long-form responses.

Choose Q4_K_S when every megabyte counts—mobile deployments, embedded systems, or situations where you’re pushing absolute memory limits.

Q4_0 and Q4_1: Legacy Variants

These older quantization formats appear in some model repositories. Q4_0 represents basic 4-bit quantization without the sophisticated optimizations of K-quant variants. Q4_1 adds some improvements but still lags behind Q4_K_M.

Generally, prefer Q4_K_M over these legacy formats unless you have specific compatibility requirements. The quality improvement from better quantization methods is meaningful.

The Impact of Quantization on Inference Speed

Beyond memory savings, quantization affects how fast models generate tokens.

Memory Bandwidth as the Bottleneck

LLM inference is typically memory-bandwidth-bound rather than compute-bound. The GPU or CPU spends more time waiting for data to arrive from memory than performing calculations.

Quantization reduces the amount of data that must move through memory. Q4 moves one-quarter the data of FP16, Q8 moves half. This directly translates to speed improvements on systems where memory bandwidth is the limiting factor.

Hardware-Specific Considerations

The speed impact varies by hardware:

On CPUs: Quantization usually improves speed. Integer operations are fast on modern CPUs, and the reduced memory bandwidth requirement provides clear benefits. Q4 might generate tokens 1.5-2x faster than FP16 on CPU.

On consumer GPUs: Results vary. High-end GPUs with massive memory bandwidth might see minimal speed differences between FP16 and Q8, while Q4 shows clear advantages. Budget GPUs with limited bandwidth benefit more from quantization.

On Apple Silicon: The unified memory architecture and high bandwidth mean quantization benefits are less dramatic than on PCs. However, Q4 and Q8 still typically outperform FP16 due to reduced memory movement.

Real-World Speed Comparisons

On a typical setup (16GB RAM, mid-range GPU), you might observe:

• FP16: 12-18 tokens per second
• Q8: 15-22 tokens per second
• Q4: 20-30 tokens per second

The exact numbers depend on your specific hardware, but the pattern holds: more aggressive quantization generally means faster inference, with Q4 providing the best speed.

Common Quantization Mistakes to Avoid

Understanding pitfalls helps you get the most from quantized models.

Assuming All Q4 Models Perform Equally

Not all 4-bit quantization is created equal. A Q4_0 model from 2023 will perform noticeably worse than a Q4_K_M model using modern techniques. Always check which quantization method was used.

When downloading models, look for recent quantizations using GPTQ, AWQ, or GGUF K-quant formats. These employ sophisticated techniques that maintain quality far better than naive 4-bit compression.

Over-Optimizing for Memory

Some users select the most aggressive quantization possible to minimize memory usage. A Q3 model in 2GB might fit more comfortably than Q4 in 3.5GB, but the quality loss often makes the model unusable for practical purposes.

Better to run a slightly larger, higher-quality model that produces good results than save 1GB with a model that frustrates you with poor outputs.

Ignoring Task-Specific Needs

Choosing quantization based solely on memory or general recommendations misses task-specific requirements. If you primarily use models for mathematics, Q4 will disappoint regardless of memory constraints. Either allocate more resources for Q8, or accept the limitations and verify outputs carefully.

Not Testing Before Committing

Quantization tolerance is personal and task-specific. Download multiple variants (Q4, Q8, maybe Q5) and test them with your actual use cases. What bothers one person might be imperceptible to another.

Spend an hour testing different quantizations with real workflows before deciding. This investment prevents frustration from choosing poorly.

Conclusion

Understanding quantization transforms how you approach local LLMs, shifting the focus from “can I run this model?” to “which version should I run?” FP16 provides maximum quality for users with abundant memory, Q8 delivers near-identical quality with 50% memory savings making it the sweet spot for most users, and Q4 democratizes access by enabling capable models on modest hardware despite noticeable but acceptable quality tradeoffs. The choice isn’t about finding a universally “best” option but matching quantization level to your specific constraints, tasks, and quality requirements.

Modern quantization techniques have advanced remarkably, with today’s Q4 models maintaining quality that would have required Q8 just a year ago. Start by testing multiple quantization levels with your actual use cases rather than relying solely on theoretical specifications—the difference between reading that Q4 maintains “90-95% quality” and experiencing whether that’s acceptable for your needs is substantial. With this knowledge, you can confidently select quantization levels that balance accessibility, performance, and quality for your specific local LLM applications.