LLM quantization explained: FP8, NVFP4, AWQ, GPTQ and GGUF

Which model in which format fits your hardware? WZ-IT sets up local language models on your own infrastructure, chooses model and quantization to match GPU, context length and number of users, and checks quality with your own tasks. Explore managed GPU servers · Explore the AI Cube · Book a meeting

Whether a language model fits on a GPU and how many requests it can serve at once depends to a large extent on numerical precision. A model with 32 billion parameters takes around 66 GB in BF16 and less than a third of that as a 4-bit variant. In between lie formats with different assumptions about hardware and software: FP8, NVFP4, MXFP4, AWQ, GPTQ and GGUF. This article classifies the formats, shows which GPU accelerates which format natively and explains how the KV cache enters the calculation. As of October 2026.

Table of contents

• What quantization means for LLMs
• The formats at a glance
• FP8: the default from Ada and Hopper onwards
• NVFP4 and MXFP4: 4-bit floating point
• AWQ and GPTQ: 4-bit weights with calibration
• GGUF: the format of llama.cpp and Ollama
• Memory footprint by example
• Quantizing the KV cache
• Which format for which hardware
• Measuring quality loss instead of guessing

What quantization means for LLMs

Language models are now mostly published in BF16, that is with 16 bits per parameter. Quantization maps these values onto a format with fewer bits, typically 8 or 4. So that the small numbers cover the original value range, all methods additionally store scale factors, per tensor, per channel or per block of a few values.

Three things can be quantized, and the shorthand shows which is meant:

The benefit has two sides. First, the memory needed for the weights falls, so the model fits on smaller or fewer GPUs. Second, the GPU reads all active weights from memory for every generated token. Smaller data types consume less memory bandwidth, which according to the vLLM documentation on LLM Compressor often means higher throughput, especially for memory-bound workloads. Whether the computation itself also runs at low precision (W8A8, W4A4) depends on whether the GPU has Tensor Cores for it.

How weights, KV cache and overhead add up to the total VRAM requirement is worked through in Sizing GPU and VRAM for LLMs.

The formats at a glance

Sources: vLLM, FP8 W8A8, vLLM, Quantization, NVIDIA, Introducing NVFP4, llama.cpp, quantize. The GGUF values apply to Llama 3.1 8B as measured by llama.cpp; other models differ slightly.

Note: the overview table in the vLLM documentation ends with the Hopper column (as of October 2026). Blackwell is listed on the pages of the individual methods, for example FP8 W8A8 and INT4 W4A16.

FP8: the default from Ada and Hopper onwards

FP8 halves the memory requirement compared with BF16. For inference, E4M3 is the most common variant: 1 sign bit, 4 exponent bits, 3 mantissa bits, values up to ±448. The E5M2 variant has more range (up to ±57,344) but less precision (vLLM, FP8 W8A8).

FP8 is the obvious starting point for vLLM on current NVIDIA GPUs, because many vendors publish official FP8 checkpoints. One example is Qwen3-8B-FP8 with block-wise FP8 quantization in blocks of 128 × 128. vLLM recognises such checkpoints from their quantization_config and loads them without further parameters. With --quantization fp8_per_tensor or fp8_per_block, vLLM also converts a BF16 model to FP8 at load time, without a pre-quantized checkpoint (vLLM, Online Quantization). For production systems a tested checkpoint, for example from llm-compressor, is the more traceable option.

NVFP4 and MXFP4: 4-bit floating point

NVFP4 is NVIDIA's 4-bit format for Blackwell. Each value is a 4-bit floating-point number (E2M1, range roughly -6 to +6). Every 16 values share a scale factor in FP8 (E4M3), and there is an additional FP32 factor per tensor. Including the scales, this comes to about 4.5 bits per value (NVIDIA, Introducing NVFP4).

The smaller blocks and the finer FP8 scale are intended to reduce rounding error compared with MXFP4. Native FP4 compute units come with the fifth-generation Tensor Cores, that is Blackwell GPUs such as the RTX PRO 4000 and RTX PRO 6000 Blackwell and the GB10 chip.

vLLM loads NVFP4 checkpoints from NVIDIA Model Optimizer and from llm-compressor (vLLM, NVIDIA Model Optimizer). At startup vLLM selects a suitable GEMM kernel. If the GPU lacks a native FP4 kernel, vLLM falls back to W4A16 via Marlin and logs a warning; the memory benefit remains, the compute benefit is lost. Which kernel is actually active is shown in the startup log and should be checked at every deployment.

MXFP4 became known mainly through gpt-oss: according to the model card, the MoE weights were quantized to MXFP4 during post-training, and all published benchmarks refer to this variant. gpt-oss-120b therefore runs on a single 80 GB GPU, gpt-oss-20b within 16 GB of memory.

On quality there are vendor figures, not an independent guarantee:

The trend is clear: large models and MoE models tolerate 4 bits better than small dense models.

AWQ and GPTQ: 4-bit weights with calibration

AWQ and GPTQ are not data types but methods that convert weights after training (post-training quantization), usually into 4-bit integers. Both use a small set of calibration texts to keep rounding errors low. Activations stay in 16 bits (W4A16).

The AWQ authors show that protecting only about 1 % of the most important weights already reduces the quantization error considerably. According to the paper, GPTQ quantized a model with 175 billion parameters to 3 to 4 bits in about four GPU hours.

In vLLM, AWQ and GPTQ are supported on Turing, Ampere, Ada and Hopper according to the hardware table, GPTQ also on Volta. The INT4 page lists Ampere, Ada Lovelace, Hopper and Blackwell as compute platforms (vLLM, INT4 W4A16). For GPUs without FP4 compute units, AWQ and GPTQ are thus the common 4-bit formats. On Blackwell, NVFP4 is an alternative where a checkpoint is available.

GGUF: the format of llama.cpp and Ollama

GGUF is the file format of llama.cpp. It holds weights, tokenizer and metadata in one file and comes with its own quantization types. Common choices are the K-quants (such as Q4_K_M, Q5_K_M, Q6_K) and the I-quants for very low bit widths. According to llama.cpp, an importance matrix (imatrix) built from calibration texts reduces quality loss, which is measured there via perplexity and KL divergence.

Measurements from the llama.cpp documentation for Llama 3.1 8B:

Ollama uses GGUF as its model format. According to the Ollama documentation, Ollama does not quantize GGUF models during import; quantization happens beforehand with llama-quantize. GGUF runs on CPUs, NVIDIA and AMD GPUs and Apple Silicon and can split layers between GPU and system memory. That makes it the first choice for single workstations and small installations. vLLM supports GGUF only through a separate plugin and describes the support as "highly experimental and under-optimized" (vLLM, GGUF). For vLLM, FP8, NVFP4, AWQ or GPTQ are the appropriate formats. The trade-off between the two engines is covered in vLLM vs. Ollama.

Memory footprint by example

Qwen3-32B has around 32.8 billion parameters (Apache 2.0). The memory required for the weights is the parameter count times bits per weight divided by 8:

These are calculated values. Real files differ because individual layers such as embeddings or the output layer often remain at higher precision.

The table shows the gradation: in BF16 the model needs a 96 GB card, in FP8 it fits there with plenty of room for the KV cache. A 24 GB card only holds it in 4 bits, and then only a few gigabytes remain for the KV cache and runtime.

Quantizing the KV cache

Besides the weights, the KV cache takes memory: for every token in the context, each layer stores keys and values. The requirement per token is:

KV per token = 2 × layers × KV heads × head dimension × bytes per value

For Qwen3-32B (64 layers, 8 KV heads, head dimension 128 according to its config.json) this gives:

The KV cache grows with context length and concurrency, not with the quantization of the weights. A 4-bit model with long context and many users can therefore need more memory for the cache than for the weights.

vLLM: --kv-cache-dtype fp8 (or fp8_e4m3, fp8_e5m2) stores the cache in FP8. Without calibration vLLM sets all scale factors to 1.0; calibrated scales via llm-compressor are recommended. With the Flash Attention 3 backend, queries are also computed in FP8 (vLLM, Quantized KV Cache).

Ollama: OLLAMA_KV_CACHE_TYPE allows f16 (default), q8_0 (about half the memory, according to the documentation usually without noticeable quality impact) and q4_0 (about a quarter, small to medium precision loss, more noticeable with long context). Flash Attention is required, and the setting applies globally to all models (Ollama FAQ).

Which format for which hardware

The choice of format follows from the GPU's compute capability, memory size and memory bandwidth. The hardware data comes from NVIDIA (CUDA GPUs, RTX PRO 4000 Blackwell, RTX PRO 6000 Blackwell Max-Q, DGX Spark hardware), as of October 2026:

This leads to typical starting points:

The boundaries are fluid and depend on the model. Which models run sensibly on 128 GB of unified memory is shown in LLM models on 128 GB unified memory; running gpt-oss-120b on the AI Cube is described in gpt-oss-120b on the AI Cube. A practical example of FP8 on two 96 GB cards is Qwen 122B with vLLM on two RTX PRO 6000. When a model is split across several GPUs is covered in Multi-GPU inference without NVLink.

Measuring quality loss instead of guessing

Vendor figures refer to public benchmarks such as MMLU, GPQA or HumanEval. They show the direction but not how a quantized model behaves with German contract clauses, structured output or tool calls. Especially with small models and at 4 bits, a comparison of your own is worthwhile.

A sound comparison has four steps:

1. Set a reference. Run the model in BF16 or FP8 on the same tasks.
2. Choose everyday tasks. 50 to 200 real requests with expected answers, plus cases with JSON output or tool calls if the system uses them.
3. Configure identically. Same prompt, same temperature, same context length; only the format changes.
4. Evaluate deviations. Do not only count hits, also check format errors, truncated answers and outliers.

For standardised benchmarks the vLLM documentation uses lm-evaluation-harness. llama.cpp ships llama-perplexity, a tool for perplexity and KL divergence against the original. For RAG systems, answer quality with sources belongs in the measurement, as described in Measuring RAG quality.

The licence also matters: a quantized variant is derived from the original model and is generally still subject to its licence terms. More on this in Open LLM licences for commercial use.

What this means for your project

Quantization is not a final polish but part of the hardware decision. Knowing which model should run in which format allows GPU size, context length and the number of concurrent users to be planned reliably. Buying the hardware first fixes the possible formats.

We set up models with vLLM or Ollama, choose the format to match the GPU, check in the startup log which kernel actually runs and compare the quantized variant against the reference using your own tasks. The basis is WZ-IT's managed GPU servers with NVIDIA RTX PRO 4000 Blackwell (24 GB GDDR7 ECC) or RTX PRO 6000 Blackwell Max-Q (96 GB GDDR7 ECC) and the AI Cube with 128 GB unified memory. Support, consulting and implementation by WZ-IT.

Model choice itself is covered in Which LLM to self-host?, the total memory requirement in Sizing GPU and VRAM for LLMs. The difference between inference and training is explained in Inference vs. training, and the inference engines are compared in vLLM, Ollama and llama.cpp compared.