Quantization loss measured: AWQ versus GGUF and EXL2

Compressing open-weight language models from 16-bit floating-point representations to lower precisions such as 4-bit, 3-bit or even 2-bit integers is the foundation of self-hosted inference. Without precision reduction, a modern open model with tens of billions of parameters simply will not fit in the video memory of affordable graphics cards or compact servers. Post-training quantization is never free, however. Reducing the number of bits per weight introduces rounding noise, distorts attention vectors and can, in the worst case, lead to subtle reasoning errors, syntax errors in code output or persistent hallucinations in automated pipelines.

In this signal report we compare the three dominant quantization architectures developers deploy on their own hardware: Activation-aware Weight Quantization (AWQ), GPT-Generated Unified Format (GGUF) within the llama.cpp ecosystem, and ExLlamaV2 (EXL2). For those who want to explore the theoretical fundamentals of numerical precision and rounding errors first, the explainer page quantization explained offers a clear starting point on converting continuous tensors into discrete integers. Here we focus specifically on the mathematical mechanisms behind quality degradation, the differences in memory architecture and the practical trade-offs when selecting the right runtime format.

The measurement methodology: perplexity versus task-specific downstream evaluation

To determine how much model quality is lost during quantization, evaluation tools in practice use two complementary angles: probabilistic perplexity measurements on standardized corpora and deterministic evaluations on specific tasks. Perplexity computes the cross-entropy loss over an unseen text sequence, such as WikiText-2 or C4. The number indicates how well the compressed model is still able to accurately predict the natural probability distribution of successive tokens. A rise in perplexity score relative to the uncompressed FP16 or BF16 base model serves as a reliable first indication of general loss of linguistic feel.

Perplexity alone is an incomplete gauge, however. A model can retain a nearly identical perplexity score on flowing prose while structurally failing whenever a task requires strict deterministic dependencies. Quantized artifacts are therefore tested in parallel on downstream benchmarks such as multi-step mathematical derivations, program code generation and factual questions. On such logical tasks it turns out that compression noise does not manifest evenly across all network layers; the deeper MLP layers (multi-layer perceptrons) and attention mechanisms in particular show a sudden drop in abstract deductive ability when reduction is too aggressive.

Structuring evaluations also requires a constant comparison between operational costs and delivered accuracy; anyone wanting to analyze that balance more deeply can consult the methodology in quality versus cost to see how evaluation tests are constructed. A clean comparison always requires the compressed model to run at temperature zero, with fixed prompt templates and an identical context buffer, so that non-deterministic sampling does not muddy the comparison data.

The mathematical degradation mechanism in precision reduction

When the weight matrix of a neural network is reduced from 16 bits to, say, 4 bits, the continuous value range is divided over only 16 discrete levels. With naive uniform rounding this inevitably produces a substantial quantization error. In transformer architectures this error is not randomly distributed. Research shows that the activation vectors during the forward pass contain extreme outliers: a fraction of the hidden dimensions exhibits activation values that are a multiple of the average magnitude.

When these extreme activations are multiplied by weights that are even slightly distorted by rounding, the absolute deviation at the output of the linear layer explodes. This snowball effect disrupts so-called LayerNorm stability and propagates through successive transformer blocks. Quality loss at low bit depths is therefore caused primarily by the inability to represent these activation outliers without error. The three formats each solve this mathematical problem in a fundamentally different way.

AWQ: protecting salient weights through activation scaling

Activation-aware Weight Quantization takes an elegant mathematical detour to minimize matrix errors. Instead of treating all weights equally or keeping a small percentage in the expensive FP16 format (which severely hampers parallel compute speed on GPUs), AWQ analyzes activation statistics using a small calibration set. This reveals that weights corresponding to channels with large activations — the so-called salient weights — are crucial for preserving model functionality.

AWQ computes an optimal per-channel scaling factor by which the salient weights are multiplied, while the corresponding inputs are scaled inversely. Because the multiplication remains mathematically neutral for the final outcome, the relative rounding error on the most important channels is drastically reduced when the matrix is then uniformly rounded to INT4. All weights thus remain neat 4-bit integers, which keeps hardware alignment with NVIDIA Tensor Cores optimally intact.

The practical result is that AWQ delivers exceptionally high stability at 4-bit inference without any speed loss from mixed instruction sets. The drawback of this approach is the lack of granularity. AWQ offers no flexible intermediate steps between 4-bit and 8-bit, which ties administrators to fixed VRAM jumps. If you want to convert models yourself and test different formats, the practical guide in quantization in practice: compressing a model yourself shows how to apply calibration sets correctly.

GGUF: block-wise quantization and modular layer offloading

GGUF was designed as the universal container format for llama.cpp and emphasizes deployability across heterogeneous hardware. Instead of relying on uniform matrix compression, GGUF introduced the concept of k-quants (such as Q4_K_M, Q5_K_S or Q3_K_L). Within a k-quant, weights are divided into small super-blocks of, say, 256 weights, which are in turn subdivided into sub-blocks with their own scaling factors and offsets.

The great advantage of this block-wise design is the possibility of mixed-precision quantization within the same model file. In a configuration such as Q4_K_M, the attention layers and gate tensors — which are extremely sensitive to quantization noise — are stored at 5-bit or 6-bit precision, while less critical feed-forward layers are reduced to 4-bit. This keeps the total memory footprint limited while protecting the vulnerable nodes in the network against degradation.

GGUF also excels at memory allocation through layer offloading. When a model does not fit entirely in the GPU's video memory, layers can be distributed seamlessly between VRAM and regular system RAM. Earlier overviews of local hardware showed how this flexibility became the standard for local developers; see the analysis in local LLMs and Ollama: signals from July 2026 on efficiently combining shared memory on compact workstations. The price of this flexibility is throughput: as soon as layers are moved across the PCI Express bus to the CPU, processing speed drops to the bandwidth of system memory.

EXL2: continuous bitrates and second-order error correction

ExLlamaV2 follows a fundamentally different philosophy aimed specifically at pure GPU acceleration. Where GGUF works with discrete steps (3-bit, 4-bit, 5-bit), EXL2 introduces a continuous bitrate architecture. An administrator can quantize a weight matrix to exactly 4.25, 3.85 or 5.12 bits per weight (bpw). This lets a model be tuned precisely to the physical VRAM capacity of a specific graphics card (such as 16 GB, 24 GB or 48 GB) without leaving expensive memory unused.

Mathematically, EXL2 leans on an advanced variant of second-order error optimization. During compression the algorithm computes the inverse Hessian matrix of the reconstruction error over a calibration corpus. When a weight is rounded to a discrete bit value, the remaining, not yet quantized weights in the same layer are immediately adjusted numerically to compensate for the total deviation at the matrix output. EXL2 also automatically assigns different bit depths to individual matrices based on a measured sensitivity analysis: projections with a high impact on coherence get higher precision (e.g. 6-bit), while wide intermediate layers are compressed more heavily (e.g. 3-bit).

For homelab environments where model servers run under virtualized hypervisors, EXL2 offers unmatched throughput per stream. Anyone building such a host environment with GPU assignment will find in the step-by-step guide GPU passthrough on Proxmox for local model servers the configurations for setting up hardware-level PCIe access for ExLlamaV2 and vLLM containers.

Model size as the decisive factor: the scale difference between 8B and 70B

A crucial regularity in quantization architecture is that sensitivity to precision reduction is inversely proportional to the size of the model. Large language models with 70 billion parameters or more have high internal parameter redundancy. Knowledge and logical patterns are spread across thousands of wide attention and projection layers, so the loss of individual numerical precision is relatively easily absorbed by the surrounding network.

In compact models of 7 billion to 9 billion parameters, by contrast, that redundancy is minimal. Every weight matrix contains a compact representation of skills, so rounding errors have directly measurable consequences. Where a 70B model retains virtually its full reasoning capacity on MMLU or GSM8K under aggressive 4-bit quantization, an 8B model at the same bit level already shows clear vulnerability on complex logical prompts or multilingual translation.

Below is a qualitative comparison of expected behavior and degradation risks across different scale levels and compression depths:

VRAM allocation, KV cache quantization and inference speed

When sizing a local inference stack, static weight memory is only one part of the equation. Total VRAM usage is determined by the sum of three factors: the loaded model weights, the contextual KV cache (key-value cache) and the dynamic runtime overhead of the CUDA kernels. With long context windows in particular, the KV cache can exceed the weight memory in size.

# Starten van een vLLM server met AWQ en FP8 KV-cache optimalisatie
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --quantization awq \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 16384 \
  --tensor-parallel-size 2

In production scenarios with high concurrency, AWQ scales superbly under engines such as vLLM thanks to PagedAttention. Memory for context tokens is managed in fragmented virtual pages, which eliminates memory waste from up-front reservations. Combined with FP8 quantization of the KV cache, servers can serve dozens of concurrent users without running out of memory. This ties into broader strategies for efficient token management; see the methods in token savings: proven tricks from the community which focus on context compression and cache optimization.

ExLlamaV2 targets single-stream latency primarily and offers native 4-bit (INT4) and 8-bit (FP8) KV cache compression directly within the kernel. This lets a local builder load a context of 32,000 tokens on a single graphics card, where an uncompressed FP16 cache would immediately cause out-of-memory errors. Through llama.cpp, GGUF likewise offers options for -ctk q8_0 and -ctv q4_0 for compressing the context cache, which yields considerable scaling benefits on Apple Silicon Unified Memory in particular.

Decision tree for model builders and infrastructure choices

To determine which quantization architecture best fits a specific homelab or server setup, we apply three clear infrastructural criteria:

1. Choose AWQ for centralized inference servers: Does the infrastructure run on vLLM, SGLang or TensorRT-LLM with the aim of exposing an API for multiple concurrent users or agents? AWQ offers standardized INT4 weights that work seamlessly with enterprise schedulers and advanced paged-caching architectures.

2. Choose EXL2 for interactive single-user pipelines on NVIDIA hardware: Does the model run locally on one or more NVIDIA RTX graphics cards for interactive use, fast coding assistance or local agent loops where minimal wait time per token is decisive? EXL2 achieves the highest generation speed and lets administrators fit the model into available VRAM down to the last megabyte through fractional bitrates.

3. Choose GGUF for platform independence and heterogeneous memory: Does the model have to work on macOS (Apple Silicon M chips), CPU-based servers, compact mini computers or systems where layer offloading to system RAM is unavoidable? GGUF via llama.cpp offers the broadest support and the most fine-grained control over hybrid memory allocation.

This infrastructural trade-off is an integral part of private hosting strategies; see also the broader overview in homelab and self-hosted AI: signals from July 2026 on hardware choices and energy efficiency in local model servers.

Conclusion and implementation guidelines

Post-training quantization has evolved from a simple rounding operation into an advanced discipline of error compensation and activation scaling. The technical mechanisms behind AWQ, GGUF and EXL2 show that compressing foundation models to 4 bits per weight has become a mature standard, provided the model is large enough to absorb the rounding noise.

For compact models, restraint remains warranted: the loss of structural precision under sub-4-bit compression translates directly into higher error rates on logical tasks and code generation. By matching the quantization format to the specific runtime environment and hardware capacity — AWQ for multi-user throughput, EXL2 for dedicated GPU speed, and GGUF for flexible cross-platform deployment — developers build a local AI foundation that performs optimally in both compute and output quality.