vLLM production deployment: an operations guide for Docker, parameters and security

Have vLLM operated instead of running it yourself?

On the Managed GPU Server, the model, vLLM and Open WebUI run on a dedicated GPU in Germany, with monitoring and updates, from EUR 699 net per month plus a one-time setup fee. The ongoing review of the application on top (traces, evaluations, costs) is covered by Managed LLMOps.

View Managed GPU Server · View LLMOps

A vllm serve runs within minutes. Production needs more: a memory calculation, bounded parameters, security, metrics, an update procedure and a load test. This guide is an operations manual for vLLM with Docker, checked against the vLLM documentation and the source code. Status 28 October 2026, current version vLLM 0.32.0. What is vLLM? explains what vLLM is, vLLM vs. Ollama compares it with Ollama.

Table of contents

• Plan memory: weights and KV cache
• Start vLLM with Docker Compose
• The server parameters that matter
• Kubernetes: notes
• Health checks and readiness
• Metrics, LiteLLM and Langfuse
• Securing vLLM
• Updates and version pinning
• Typical errors
• Load testing without invented numbers
• How to evaluate vLLM consulting
• Status and methodology
• Our approach at WZ-IT
• Further guides

Plan memory: weights and KV cache

Production starts with context and concurrency that fit the GPU memory. GPU memory splits into weights, KV cache and a reserve:

1. Weights = file size of the checkpoint (parameters × bytes per parameter).
2. KV cache per token = 2 × layers × KV heads × head dimension × bytes (classic attention, values from config.json). With multi-head latent attention, vLLM stores only kv_lora_rank + qk_rope_head_dim values per layer (source code).
3. Available for the cache = GPUs × GPU memory × --gpu-memory-utilization (default 0.92, CLI reference) - weights - reserve. The sum over all GPUs holds as long as tensor parallelism does not exceed the number of KV heads. With MLA models that have one latent KV head, the cache for each token sits on every GPU of the tensor-parallel group, so the budget applies per GPU (remedy: --decode-context-parallel-size, vLLM docs).
4. Sessions = available ÷ (KV cache per token × context length).

Example Ministral 3 14B in FP8 (model card): 15.73 GB of weights (15,726,706,472 bytes), 40 layers, 8 KV heads, head dimension 128, so 160 KiB per token in BF16.

Assumptions: 4 GB reserve per GPU for activations and CUDA graphs (planning value, not a measurement), KV cache in BF16, 1 GB = 10^9 bytes. The table uses the default 0.92, the Compose file below sets 0.90 and reduces the budget by 1.9 GB (96 GB) or 0.5 GB (24 GB). On the 24 GB card the cache by calculation cannot hold a single context of 32,768 tokens, which calls for a smaller --max-model-len (or auto, which picks the largest value that fits). At startup vLLM logs the actual values (Available KV cache memory, GPU KV cache size: … tokens, Maximum concurrency for … tokens per request). The derivation is in Sizing GPU and VRAM for LLMs, quantization is covered by LLM quantization.

Start vLLM with Docker Compose

The model is downloaded once beforehand (HF_HOME=/srv/models/huggingface hf download mistralai/Ministral-3-14B-Instruct-2512), which the vLLM troubleshooting guide recommends. After that this compose.yaml is enough (sized for the 96 GB row of the table, on 24 GB --max-model-len has to be smaller):

services:
  vllm:
    image: vllm/vllm-openai:v0.32.0
    restart: unless-stopped
    shm_size: "16gb"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    environment:
      VLLM_API_KEY: ${VLLM_API_KEY}
      VLLM_NO_USAGE_STATS: "1"
      HF_HUB_DISABLE_TELEMETRY: "1"
      HF_HUB_OFFLINE: "1"
    volumes:
      - /srv/models/huggingface:/root/.cache/huggingface
      - vllm-cache:/root/.cache/vllm
    ports:
      - "127.0.0.1:8000:8000"
    command:
      - --model=mistralai/Ministral-3-14B-Instruct-2512
      - --served-model-name=assistent
      - --tokenizer-mode=mistral
      - --config-format=mistral
      - --load-format=mistral
      - --max-model-len=32768
      - --gpu-memory-utilization=0.90
      - --max-num-seqs=16
      - --enable-auto-tool-choice
      - --tool-call-parser=mistral
volumes:
  vllm-cache:

What differs from the tutorial default:

• Fixed image version instead of latest. The image starts vllm serve, parameters follow the image name. vLLM needs --ipc=host or --shm-size (Docker docs).
• Compile cache on a volume (/root/.cache/vllm). Otherwise every new container compiles the model artifacts again.
• GPU reservation as in the Compose documentation, with the NVIDIA Container Toolkit installed.
• Port on loopback only. Published Docker ports are otherwise reachable from outside as well (Docker docs).
• Key in .env (file mode 600), not in the Compose file.
• Tokenizer mode and tool-calling flags come from the model card. Other models need other parsers.
• Optionally the container runs without root: --user 2000:0 with mounts under /home/vllm (Docker docs).

The server parameters that matter

Sources: CLI reference, optimization guide and parallelism.

Parameters can also be kept in a YAML file (vllm serve --config), values on the command line take precedence (docs).

Kubernetes: notes

Kubernetes pays off with several GPU nodes, replicas or an existing cluster with an operating process, for a few GPUs Compose is the smaller operational surface. The vLLM guide for Kubernetes and the Kubernetes documentation on GPUs name the key points:

• GPU under limits: nvidia.com/gpu belongs in limits. Requests may be omitted (Kubernetes then takes the limit) or must be equal, requests without limits are not allowed.
• Shared memory: an emptyDir with medium: Memory at /dev/shm.
• Model cache on a PersistentVolumeClaim, a Hugging Face token as a Secret only for gated models.
• Startup probe with a sufficient failureThreshold. If it is too low, Kubernetes kills the container while it loads (log: KeyboardInterrupt: terminated). The value comes from measuring the startup time without a probe.
• Fixed image version. The examples in the docs use latest.

Several replicas with routing and autoscaling are covered by the production-stack with a Helm chart, maintained by the vLLM project. General cluster requirements are covered by the Kubernetes production checklist.

Health checks and readiness

/health reports the state of the engine, without an API key. The docs use the endpoint for liveness and readiness. Three additions belong in operations:

1. Startup probe for the loading time of large models (Kubernetes probes).
2. Synthetic request: monitoring regularly sends a short chat request with max_tokens: 1 and the API key to /v1/chat/completions and measures the time until the answer starts. /health shows that the process is alive, not that the model answers.
3. Probes bypass the proxy: /health does not pass the reverse proxy, probes go directly to the container or pod.

Metrics, LiteLLM and Langfuse

vLLM exposes Prometheus metrics at /metrics (metrics docs), without authentication even when --api-key is set (security docs). Counters carry the suffix _total in Prometheus, for example vllm:prompt_tokens_total in the Grafana example of the docs. Check the names with curl http://localhost:8000/metrics before writing alert rules, the table lists the counters with this suffix.

The service level of the application sets alert thresholds, not a table. Persistently high cache usage with rising preemptions means: less context, less concurrency or more GPU memory. Deprecated metrics are hidden one minor version later and removed one more version after that (metrics docs), so dashboards belong in the update test.

A gateway complements the engine metrics. LiteLLM addresses vLLM via hosted_vllm/<model> with api_base (docs) and writes traces to Langfuse with success_callback: ["langfuse"] (docs). That puts latency and costs per application next to the engine values. The gateway is an attack target itself, see LiteLLM vulnerability and gateway hardening.

Securing vLLM

The vLLM security documentation explicitly warns against relying on --api-key alone.

• --api-key protects only /v1, /v2, /inference and /cohere. Reachable without a key remain, among others, /invocations (the same inference as /v1), /pooling, /classify, /score, /rerank, /metrics, /health, /tokenize, /load and control paths such as /pause, /abort_requests and /update_weights.
• Reverse proxy with an allowlist. Only the required paths pass, everything else ends with 403. Rate limiting and logging sit at the proxy. Example for nginx (test before use; without buffering the response goes to the client synchronously, nginx docs):

location ~ ^/v1/(chat/completions|completions|models)$ {
    proxy_pass http://127.0.0.1:8000;
    proxy_http_version 1.1;
    proxy_buffering off;      # streaming without intermediate buffer
    proxy_read_timeout 600s;  # default 60s, long answers need more
}
location / { return 403; }

• Isolate the network. With several nodes, PyTorch distributed traffic and KV transfer are unencrypted and belong in an isolated network, VLLM_HOST_IP gets a fixed address. From outside this network only the API port may be reachable.
• Dangerous switches off: VLLM_SERVER_DEV_MODE=1 (opens /collective_rpc, among others) never in production, --enable-tokenizer-info-endpoint only when needed.
• Multimodal inputs: --allowed-media-domains limits fetches against SSRF. Publicly reachable services set VLLM_MAX_N_SEQUENCES low (the docs give 64 or 128 as an example).
• A key is not a permission concept. Keys per team, limits and attribution belong in a gateway.
• Close data paths. By default vLLM collects anonymous usage data, which VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1 switches off (docs). Hugging Face telemetry stops with HF_HUB_DISABLE_TELEMETRY=1, calls to the Hub with HF_HUB_OFFLINE=1 (docs). --enable-log-requests writes parameters to the log, and at DEBUG level prompts as well. With personal data it stays off.

Application risks such as prompt injection are covered by Protection against prompt injection, access from outside by Provide secure remote access to local AI.

Updates and version pinning

According to PyPI, 0.28.0 (26 August 2026), 0.29.0, 0.30.0, 0.31.0 and 0.32.0 (18 October 2026) were released in just under eight weeks. The Docker tags carry the version (v0.32.0), alongside latest and nightly.

1. Pin the image to a version number or digest, never to latest.
2. Test first: update an environment with the same GPU and the same model, then compare startup log (cache size, concurrency), load test and dashboards.
3. Keep a fallback: the previous Compose file and previous image stay available.
4. Check model requirements: new models often need new vLLM versions, the Mistral Medium 3.5 model card, for example, names a nightly build.
5. Plan for caches: changes to the model, configuration, torch or GPU invalidate the compile cache (docs), the first start after an update takes longer.
6. Follow security notices: the project coordinates vulnerabilities in a dedicated process (vulnerability management) and publishes security advisories.

Typical errors

Load testing without invented numbers

Throughput figures from other setups say nothing about your prompts, your model and your GPU. The vLLM docs position vllm bench serve for feature and regression tests and recommend GuideLLM for production servers (benchmarking). Procedure:

1. Your own prompts as .jsonl with the field prompt, in a realistic length distribution of the inputs (short chat versus RAG with long context). The answer length is set by --custom-output-len (default 256), with --custom-output-len -1 the field output_tokens of each line applies. Do not copy real data with personal references into test tools.
2. Steps: raise concurrency gradually (--max-concurrency, optionally --request-rate).
3. Measure: time to first token, time between tokens and end-to-end latency as percentiles, error rate, together with kv_cache_usage_perc, num_requests_waiting and num_preemptions_total from the metrics.
4. Mind the prefix cache: repeated runs with the same prompts inflate throughput. Vary the seed or restart the server (docs).
5. Define acceptance beforehand, with --goodput as a target per request. The business side sets the target values.
6. Repeat after every update and model change.

vllm bench serve --backend openai-chat --endpoint /v1/chat/completions \
  --model mistralai/Ministral-3-14B-Instruct-2512 --served-model-name assistent \
  --host 127.0.0.1 --port 8000 \
  --header "Authorization=Bearer ${VLLM_API_KEY}" \
  --dataset-name custom --dataset-path prompts.jsonl --num-prompts 200 \
  --skip-chat-template --max-concurrency 8 \
  --percentile-metrics ttft,itl,e2el --metric-percentiles 50,95,99

The options come from the docs for vllm bench serve. The call runs per step with a different --max-concurrency. --skip-chat-template stops the client from applying the chat template in addition to the server (the /v1/chat/completions endpoint applies it itself). The client loads the model's tokenizer, without Hub access --tokenizer points to a local copy. Read percentiles, not averages.

How to evaluate vLLM consulting

Internal or external: these questions must be answerable with files and measurements.

1. Which image version is running, how is it pinned, what does the fallback look like?
2. Is the memory calculation available with assumptions (weights, KV cache, reserve)?
3. Which endpoints are reachable from outside, and where is the allowlist?
4. Which metrics and alerts exist, and who receives them?
5. Is there a load test with your own prompts and written acceptance values?
6. How does an update run, and which tests belong to it?
7. Who owns configuration, keys and access, and how does the handover back work?
8. Which data paths leave the environment (telemetry, logs, model downloads)?

Status and methodology

• Status: 28 October 2026. The basis is the vLLM documentation (version latest) and the source code on the main branch, retrieved on 28 October 2026.
• Calculation: weights from the file sizes of the Hugging Face repositories, cache values from config.json. The reserve of 4 GB per GPU is an assumption.
• Not measured: the guide contains no benchmarks and no throughput figures. They result from your own load test.
• Example model: Ministral 3 14B (Apache 2.0), flags for other models are in their model cards.

Our approach at WZ-IT

We run vLLM on the Managed GPU Server: a dedicated NVIDIA GPU in a German data centre with a matched model, vLLM and Open WebUI. Before provisioning we check model, quantization, context and concurrency by calculation against 24 or 96 GB of GPU memory. Operations include monitoring of host, GPU and vLLM, maintenance of operating system, driver and vLLM with CVE assessment, and incident handling. Monitoring runs 24/7, the response to P1 incidents follows the service level. External APIs, telemetry and backup targets are agreed as separate data paths. Those who want to review the application on top of the inference continuously (traces, evaluations, costs, latency) use Managed LLMOps with Langfuse. On your own hardware in your own network, vLLM and Open WebUI run preinstalled on the AI Cube. The vLLM expertise page describes the field, LLM hosting compares the operating options.

Further guides

• What is vLLM? - engine, PagedAttention and continuous batching
• vLLM vs. Ollama - when which engine fits
• Sizing GPU and VRAM for LLMs - derivation of the memory calculation
• Multi-GPU inference without NVLink - tensor and pipeline parallelism
• LLM quantization - FP8, NVFP4 and AWQ compared
• What is LiteLLM? - the gateway in front of vLLM
• What is Langfuse? - traces and evaluations
• vLLM, Ollama or llama.cpp - inference servers compared
• vLLM with Qwen3.5-122B on two RTX PRO 6000 - worked example with Compose and a hybrid model

Putting vLLM into operation? WZ-IT runs vLLM on a dedicated GPU server in Germany or reviews your application continuously. Managed GPU Server · Managed LLMOps · LLM hosting