What is vLLM? The inference engine for LLMs

Have models run in production on your own GPU? WZ-IT plans, builds and operates local AI infrastructure - from GPU hardware through vLLM to the complete LLM stack, GDPR-compliant from one team. See GPU servers

Anyone running a language model themselves needs software that loads it and answers requests - an inference engine. vLLM is the most widespread choice for production, high-throughput operation. This article explains what vLLM does, why it is so fast and when it is the right choice over simpler tools. As of July 2026.

Table of contents

• What vLLM is for
• PagedAttention and continuous batching
• OpenAI-compatible API server
• Hardware and models
• vLLM or Ollama

What vLLM is for

vLLM is a fast, open-source library for LLM inference and serving. It originally emerged at UC Berkeley's Sky Computing Lab and is today developed further by a large community. Its job is the bottom layer of an AI system: load the model, keep it in GPU memory and answer requests efficiently.

vLLM's ambition is not "getting a model to run" - many tools can do that - but serving it with high throughput, so that many users and requests are served simultaneously on the same hardware. That is exactly what makes it the engine for production operation.

PagedAttention and continuous batching

vLLM's high throughput rests on two core ideas:

PagedAttention manages the memory for the attention intermediate values (the key-value cache) efficiently - similar to how an operating system manages memory in pages. This leaves less GPU memory unused, and more requests fit simultaneously on the same card. It is vLLM's namesake optimization.

Continuous batching bundles incoming requests continuously instead of waiting for a full batch, complemented by techniques like chunked prefill and prefix caching. The result is state-of-the-art throughput under concurrent load - the practical difference between a model for one user and a service for many.

OpenAI-compatible API server

vLLM provides an OpenAI-compatible API server (alongside, among others, the Anthropic Messages API and gRPC). That is decisive in operation: applications and a gateway like LiteLLM address vLLM in the familiar OpenAI format, without proprietary interfaces. This way vLLM fits seamlessly into existing tools and the rest of the stack.

Hardware and models

vLLM runs primarily on GPUs (NVIDIA and AMD) and additionally supports CPUs as well as other accelerators such as TPUs. For production operation of larger models, a GPU with sufficient VRAM is decisive; the right sizing depends on model size, context length and desired throughput.

On models, vLLM is broadly positioned: over 200 architectures from Hugging Face are supported, from decoder-only models through mixture-of-expert and multimodal models to embedding models. Widespread open families such as Llama, Qwen, Mistral and DeepSeek can be self-hosted with it.

vLLM or Ollama

The most common question in practice: vLLM or Ollama? In short - Ollama is the simplest entry point (one command, one model) and ideal for tests, single users and small loads. vLLM is built for throughput and concurrent requests and thus the choice when a model should reliably serve many users.

Both are interchangeable inference layers in the open-source LLM stack - behind them lies the same idea, to run the model on your own infrastructure. For production, sovereign AI in the enterprise, the path usually leads to vLLM as soon as more than a few concurrent requests come together.