Sizing GPU and VRAM for LLMs correctly

Timo Wevelsiep•Updated: 23.07.2026

Editorial note: Versions, commands and prices may change. Please verify critical steps independently before production use. This guide does not replace individual consulting.

Have the right GPU sized for your AI project? WZ-IT plans and operates GPU servers for local AI - sized to model, context and user count, GDPR-compliant from one team. See GPU servers

"Does this model fit on my GPU?" is the practical core question when building local AI - and it can be calculated. The VRAM demand of a language model consists of three components, and anyone who looks only at model size regularly miscalculates. This article shows the rules of thumb and a worked example. As of July 2026.

Three memory components
Model weights: the rule of thumb
The KV cache: often the bottleneck
Overhead and the total
A worked example

Three memory components

The VRAM demand of a language model in operation has three parts (GPU sizing guide):

Model weights - the memory for the model's parameters.
KV cache - intermediate attention values during generation.
Overhead - activations, framework and CUDA context.

Only all three together give the real demand. The most common mistake is to start from the weights alone and underestimate the KV cache.

Model weights: the rule of thumb

The memory for the weights follows from parameter count and numeric precision. The practical rule of thumb:

FP16 (full precision): roughly 2 GB of VRAM per billion parameters.
4-bit quantization (AWQ, GPTQ): roughly 0.5 GB per billion parameters.

A 70-billion model thus needs about 140 GB at FP16 - more than a single 80 GB card (H100, A100) holds. Quantized to 4-bit, the real demand drops to around 40 to 45 GB and fits on an 80 GB GPU. Quantization is therefore the most important lever to get large models onto available hardware - usually with only a small quality loss.

The KV cache: often the bottleneck

The KV cache is the most frequently overlooked item. It stores the intermediate attention values during generation, and its demand grows linearly with sequence length and the number of concurrent requests - not with model size.

This has an important consequence: with short context the weights dominate memory demand. With long context and many parallel users the KV cache becomes the actual bottleneck. A 70-billion model with 32k context and ten concurrent users can need around 112 GB (FP16) for the cache alone - considerably more than the quantized weights. Techniques like the efficient memory management of vLLM (PagedAttention) and quantization of the KV cache mitigate this.

Overhead and the total

On top of the sum of weights and KV cache comes an overhead of around 15 to 20 percent for activations, framework and CUDA context. The total calculation is therefore:

VRAM ≈ (weights + KV cache) × 1.15 to 1.2

Anyone who omits this buffer risks out-of-memory breakdowns under load - exactly when many users query simultaneously.

A worked example

A medium-sized model with 8 billion parameters, operated in FP16 for a few concurrent users with moderate context:

Weights: 8 × 2 GB = 16 GB.
KV cache: depending on context and concurrency roughly 2 to 6 GB.
Overhead: around 15 to 20 percent on top.

Result: about 22 to 26 GB - fits on a 24 GB card if context and concurrency stay moderate, otherwise better a larger GPU. The same 8-billion model quantized (4-bit) would get by with around 4 GB of weights and leave considerably more room for cache and users.

The choice of model, quantization and GPU is thus directly linked - Which LLM to self-host? places the model side, What is local AI? the overall picture.

Rather have it operated?

You'd rather not run Local & Sovereign AI yourself? WZ-IT handles setup, operations and maintenance - GDPR-compliant from Germany.

Ai Server →

Frequently Asked Questions

Answers to the most important questions

The demand consists of three parts: the model weights, the KV cache and an overhead. As a rule of thumb, the weights need roughly 2 GB of VRAM per billion parameters at FP16, about 0.5 GB at 4-bit quantization. Add the KV cache (depending on context length and concurrency) and around 15 to 20 percent overhead.

Considerably. A 70-billion model needs about 140 GB just for the weights at FP16 and thus fits on no single 80 GB card. At 4-bit quantization (AWQ, GPTQ) the real demand drops to around 40 to 45 GB and fits on an 80 GB GPU - with limited room for the KV cache. Quantization is thus the most important lever to get large models onto available hardware.

The KV cache stores the intermediate attention values during generation. Its demand grows linearly with sequence length and the number of concurrent requests - not with model size. With long context and many parallel users, the KV cache often becomes the actual bottleneck, not the model weights.

Because in production the KV cache weighs heavily. A model may fit on the card with its weights, but as soon as several users with long context query simultaneously, the KV cache can demand more memory than the weights themselves. The sizing must account for both.

Around 15 to 20 percent on top of the sum of weights and KV cache - for activations, framework overhead and the CUDA context. Anyone who omits this buffer risks the operation breaking under load with out-of-memory errors.

There are several levers: choose a smaller or more heavily quantized model, limit the maximum context length, cap the number of concurrent requests or quantize the KV cache. If that is not enough, you distribute the model across multiple GPUs - an inference engine like vLLM supports distributed serving.