How much VRAM does your model need, and which GPU fits?
The Managed GPU Server comes in two fixed sizes: 24 GB VRAM for €699 excl. VAT per month and 96 GB VRAM for €1,799 excl. VAT per month, each plus a one-time setup fee and including model, vLLM, Open WebUI and operations. For larger models, servers with two, three or four RTX PRO 6000 (192 to 384 GB) are available on request. For teams inside their own network, the AI Cube offers 128 GB unified memory.
"Does this model fit on my GPU?" is the practical core question when building local AI - and it can be calculated. The VRAM demand of a language model consists of three components, and anyone who looks only at model size regularly miscalculates. This article shows the rules of thumb, the KV cache formula with worked examples for context length, and the effect of an FP8 KV cache. Model data as of October 2026.
Table of contents
- Three memory components
- Model weights: the rule of thumb
- The KV cache: often the bottleneck
- Calculating KV cache and context length
- FP8 KV cache: twice the context
- Overhead and the total
- A worked example
Three memory components
The VRAM demand of a language model in operation has three parts (Kwon et al., PagedAttention):
- Model weights - the memory for the model's parameters.
- KV cache - intermediate attention values during generation.
- Overhead - activations, framework and CUDA context.
Only all three together give the real demand. The most common mistake is to start from the weights alone and underestimate the KV cache.
Model weights: the rule of thumb
The memory for the weights follows from parameter count and numeric precision. The practical rule of thumb:
- FP16 (full precision): roughly 2 GB of VRAM per billion parameters.
- 4-bit quantization (AWQ, GPTQ): roughly 0.5 GB per billion parameters.
A 70-billion model thus needs about 140 GB at FP16 - more than a single 80 GB card (H100, A100) holds. Quantized to 4-bit, the real demand drops to around 40 to 45 GB and fits on an 80 GB GPU. Quantization is therefore the most important lever to get large models onto available hardware - usually with only a small quality loss. Which formats (FP8, NVFP4, AWQ, GPTQ, GGUF) suit which hardware is covered in LLM quantization.
The KV cache: often the bottleneck
The KV cache is the most frequently overlooked item. It stores the intermediate attention values during generation, and its demand grows linearly with sequence length and the number of concurrent requests - not with model size.
This has an important consequence: with short context the weights dominate memory demand. With long context and many parallel users the KV cache becomes the actual bottleneck and can occupy more memory than the quantized weights. The efficient memory management of vLLM (PagedAttention) prevents reserved but unused cache from being wasted (Kwon et al.). It does not reduce the demand itself; shorter contexts, less concurrency or a quantized KV cache do.
Calculating KV cache and context length
The KV cache can be calculated exactly from a model's config.json. For every token, each attention layer stores one key and one value vector per KV head:
KV cache per token = 2 × attention layers × KV heads × head dimension × bytes per value
Total KV cache = KV cache per token × context length × concurrent sessions
Bytes per value are 2 for FP16/BF16 and 1 for FP8. What counts is the number of KV heads, not attention heads: modern models use grouped-query attention, where several query heads share one key-value pair.
| Model (config.json) | Layers with KV cache | KV heads | Head dim. | per token FP16 | 32,768 tokens FP16 | 32,768 tokens FP8 |
|---|---|---|---|---|---|---|
| Qwen3-32B | 64 | 8 | 128 | 256 KiB | 8 GiB | 4 GiB |
| Mistral Small 3.2 24B | 40 | 8 | 128 | 160 KiB | 5 GiB | 2.5 GiB |
| Qwen3.8-27B | 16 of 64 | 4 | 256 | 64 KiB | 2 GiB | 1 GiB |
The values apply per session. Ten users working simultaneously with 32,768 tokens of context therefore need 80 GiB of cache in FP16 with Qwen3-32B. That is more than the model's weights in FP8 (around 33 GB at 32.8 billion parameters, model card).
Hybrid models shift the calculation. Qwen3.8-27B alternates, in 16 blocks, three layers of linear attention (Gated DeltaNet) with one layer of classic attention. Only the 16 classic layers build a growing KV cache; the linear layers keep a fixed state per session that does not grow with context length. gpt-oss-120b works similarly, alternating full attention with a sliding window of 128 tokens. For such models only the full-attention layers count in the formula.
An example on 96 GB. By default vLLM uses 92 percent of GPU memory (engine arguments, --gpu-memory-utilization), around 88 GB on a 96 GB card. After roughly 33 GB for Qwen3-32B in FP8 and a few GB for activations and CUDA graphs, roughly 50 GB remain for the cache. At 256 KiB per token that is around 200,000 tokens, about six concurrent sessions with a full 32k context. An FP8 KV cache doubles this. vLLM reports the actual figure at startup in its log as available KV cache memory and maximum concurrency.
An example on 24 GB. Qwen3.8-27B with 4-bit weights occupies roughly 14 to 16 GB. Of around 22 GB usable memory, about 5 GB remain for the cache, around 80,000 tokens at 64 KiB per token. A classic model with 256 KiB per token would reach only around 20,000 tokens on the same 5 GB.
FP8 KV cache: twice the context
The KV cache can be quantized independently of the weights. vLLM supports FP8 in the variants fp8_e4m3 and fp8_e5m2; it is enabled with --kv-cache-dtype fp8 (vLLM: Quantized KV Cache). A value then takes 1 instead of 2 bytes, and the cache per token halves.
| Setting | Effect | Note |
|---|---|---|
--kv-cache-dtype auto |
cache in the model's data type (usually BF16) | default, highest memory demand |
--kv-cache-dtype fp8 |
half the cache per token, twice the token capacity | without calibration, scaling factors of 1.0 apply |
| FP8 with calibrated scales | as above, scales from a calibration dataset (llm-compressor) | recommended by vLLM for the highest accuracy |
Two points belong in the plan:
- Accuracy has to be tested, not assumed. The vLLM documentation gives no blanket quality figure. Before production use, compare against a fixed set of test questions, as described in Measuring RAG quality.
- An FP8 KV cache is not weight quantization. A model in BF16 stays in BF16; only the intermediate attention values are stored more compactly. Weight formats such as FP8, NVFP4 or AWQ are covered in LLM quantization.
If even the quantized cache is not enough, the remaining options are shorter context limits (--max-model-len), fewer concurrent sessions (--max-num-seqs) or distribution across several GPUs, as described in Multi-GPU inference without NVLink.
Overhead and the total
On top of the sum of weights and KV cache comes an overhead of around 15 to 20 percent for activations, framework and CUDA context. The total calculation is therefore:
VRAM ≈ (weights + KV cache) × 1.15 to 1.2
Anyone who omits this buffer risks out-of-memory breakdowns under load - exactly when many users query simultaneously.
A worked example
A medium-sized model with 8 billion parameters, operated in FP16 for a few concurrent users with moderate context:
- Weights: 8 × 2 GB = 16 GB.
- KV cache: depending on context and concurrency roughly 2 to 6 GB.
- Overhead: around 15 to 20 percent on top.
Result: about 22 to 26 GB - fits on a 24 GB card if context and concurrency stay moderate, otherwise better a larger GPU. The same 8-billion model quantized (4-bit) would get by with around 4 GB of weights and leave considerably more room for cache and users.
The choice of model, quantization and GPU is thus directly linked - Which LLM to self-host? places the model side, What is local AI? the overall picture. Which Hetzner GPU servers offer which VRAM size and what one GB of VRAM costs there is shown in the Hetzner GPU server comparison. WZ-IT Managed GPU servers are available with 24 GB (NVIDIA RTX PRO 4000 Blackwell) and 96 GB of GPU memory (RTX PRO 6000 Blackwell Max-Q); we check the model fit for context and concurrency before provisioning.
