vLLM, Ollama or llama.cpp: which inference server for business use

Local AI for several people at once?

The AI Cube ships with vLLM and Open WebUI preinstalled and generates up to ten answers at the same time, for teams of five to twenty people: from €6,490 excl. VAT one-time plus AI Cube Care from €349.90 excl. VAT per month. For more load we size GPU servers from €699 excl. VAT per month against your actual usage or provide managed LLM hosting.

Explore AI Cube · Clarify your sizing with us

The question of the right inference server is almost always framed as a speed question and almost always answered wrongly, because one figure is missing: how many people work at the same time.

For a single user, the three common tools sit close together. At ten concurrent requests they separate by factors, not percentages. Anyone choosing on single-user measurements and then putting a team on it has bought something that does not hold up in daily use.

This article assesses the three along that figure. All figures from the repositories, as of 24 August 2026. Since October 2026 a section on SGLang, the fourth widely used serving system, has been added (as of October 2026).

Table of contents

• Three tools, three levels
• The three at a glance
• The decisive difference: concurrent users
• llama.cpp
• Ollama
• vLLM
• SGLang as an alternative to vLLM
• What we ship in the AI Cube and why
• Which tool for which situation
• What a throughput comparison does not answer
• How we approach this at WZ-IT
• Further guides

Three tools, three levels

The most common mistake in comparisons is placing the three side by side as if they were interchangeable. They sit at different levels:

llama.cpp is the engine. It runs the model, works on CPU and GPU, and is the foundation much else builds on.

Ollama is the usability layer. It uses llama.cpp and adds convenient management on top: pull models, start them, address them. Ollama has also had an engine of its own since 2025, and since v0.40.0 (25 September 2026) supported model architectures run on MLX by default on Apple silicon. The principle stays the same: comparing Ollama with llama.cpp essentially compares a wrapper with its contents.

vLLM is a standalone serving system. It does not build on llama.cpp but takes a different approach designed for GPU operation with many concurrent requests.

It follows that the real decision lies between the llama.cpp branch and vLLM. Whether you address llama.cpp directly or through Ollama is a question of convenience.

The three at a glance

Figures from the public repositories, as of 24 August 2026.

All three are unrestricted open source and actively developed. The star counts reflect visibility, not suitability: Ollama is the best known because it is the easiest entry point, not because it is the best choice for production.

The decisive difference: concurrent users

This is where the tools separate, and not gradually.

Published comparisons cite factors of ten and more for vLLM under concurrency against the llama.cpp branch. The reason is not a better implementation of the same thing but a different technique: vLLM batches requests arriving at the same time and processes them together rather than sequentially.

With one user that advantage largely disappears. There, llama.cpp used directly is even slightly faster than Ollama, because Ollama as a wrapper adds overhead, on the order of a low double-digit percentage.

Practical consequence: the jump from one to ten users is not a scaling problem solved with more hardware. It is an architectural question. A setup that works well for one person does not become good for a team through a larger graphics card but through a different serving approach.

The question before any hardware purchase is therefore: how many people actually work at the same time? Not how many should get access. Those two numbers usually differ by an order of magnitude.

Several people at once, but no GPU of your own?

The Managed GPU Server runs vLLM in a German data centre, with an agreed model, Open WebUI and operation by WZ-IT, from €699 excl. VAT per month and cancellable monthly.

Explore Managed GPU Server · Explore LLM hosting

llama.cpp

The engine beneath much else, under MIT, with 125,380 stars. Runs on CPU, on GPU and on Apple silicon, and is the right choice wherever no GPU is available.

Strength: runs practically everywhere, the smallest resource footprint, the greatest control over settings. Anyone working with embedded systems or CPU-only machines ends up here.

Limit: no management layer, no model management, no multi-user operation in a meaningful sense.

Ollama

The most widely adopted of the three at 179,309 stars, under MIT. Installation and a first model take minutes, and the handling borrows from container tooling.

Strength: by far the easiest entry point. For development, demos and the question "is a local model any good for our case at all", Ollama is the right tool.

Limit: it remains a layer over its engines and by default processes only one request per model at a time (OLLAMA_NUM_PARALLEL, default 1, Ollama FAQ), as of October 2026 with v0.40.3. For a team working in parallel it is the wrong foundation, and that only becomes apparent in operation.

vLLM

Under Apache-2.0, 89,860 stars, built for GPU serving. Offers an OpenAI-compatible API, so applications usually connect without modification.

Strength: the only one of the three designed for many concurrent requests. That is where the factors sit, not the percentages.

Limit: practically requires a GPU, is more involved to set up than Ollama, and is oversized for a single user. Anyone just experimenting does not start here.

SGLang as an alternative to vLLM

SGLang sits on the same level as vLLM: a standalone serving system for GPU operation with many concurrent requests. It is hosted by LMSYS, a non-profit open-source organisation, and offers an OpenAI-compatible interface (SGLang documentation).

What RadixAttention does. SGLang stores the KV cache of already processed tokens in a prefix tree. When several requests start with the same text, such as the same system prompt, the same tool descriptions of an agent or the same conversation history so far, that part is not recomputed. The SGLang paper from 2023 reports up to 6.4 times the throughput of the systems of that time, measured on such workloads. The figure is not a current comparison with vLLM: vLLM now also has automatic prefix caching.

Where SGLang is strong. Agent workloads with long, recurring prefixes, multi-turn conversations and large deployments across several GPUs. New models often arrive in both systems promptly; the v0.5.20 release notes list, for example, GLM-5.3-Flash and Qwen3.8-Flash-Next, plus optimisations for DeepSeek-V4 on RTX PRO 6000. v0.5.21 moves the prefix cache to a Rust core by default and adds a Decisions API (/v1/decisions) that turns models into low-latency classifiers, plus a Score API that scores all candidates in one request (release notes v0.5.21).

What to watch in operation.

• Version changes. The two-week cadence also brings breaking changes. v0.5.20, among other things, removed prefill context parallelism v1, and the Responses API now only stores results with --enable-response-store (release notes).
• Driver level. v0.5.19 was the last version with wheels and images for CUDA 12. Updating to v0.5.20 or later requires a matching driver and CUDA level.
• Blanket speed claims do not hold. Whether SGLang or vLLM is faster depends on model, quantization, hardware, version and above all on the share of shared prefixes. Only a test with the target model and your own load is reliable.

For the question of this article SGLang changes little: the line still runs between the llama.cpp branch for single seats and a serving system for many concurrent users. SGLang is a second option on the serving side, not a substitute for asking about actual concurrency.

What we ship in the AI Cube and why

In the AI Cube we ship vLLM together with Open WebUI preinstalled. That decision follows directly from the device's purpose: it sits inside a company and is used by several people.

With Ollama, setup would have been simpler and the device worse in daily use. The experience behind that is unremarkable: a setup that convinces in a test by one person disappoints at five concurrent requests, and then the purchase is questioned rather than the software choice.

Open WebUI sits on top as the interface, because users expect a chat interface with permissions, knowledge areas and history, not a programming interface. How that connects to knowledge sources belongs to setup rather than to the inference server.

Which tool for which situation

What a throughput comparison does not answer

Does the model fit in memory at all? The choice of serving system is secondary to whether model and context length fit into available GPU memory. An oversized model is not made fast by any serving approach.

How long are the contexts? Long inputs, for instance in document search, consume memory per concurrent request. The number of users that can be served in parallel drops considerably, and that is regularly overlooked in sizing.

Who updates models and the serving layer? Both move quickly. A setup left untouched for a year keeps running but loses access to newer models.

What happens if the GPU fails? A local AI system without a fallback is entirely down on a hardware fault. Whether that is acceptable belongs settled before the purchase.

Still unsure which tool fits? The AI opportunity check tests which approach holds up on a concrete process and ends with a pilot recommendation. From €3,900 excl. VAT.

How we approach this at WZ-IT

We ask about the number of concurrent users before choosing a tool, and we insist, because the first answer is almost always the number of people with access. From that figure follows the serving approach, then the hardware, and only then the model.

For trial setups and the question of basic suitability we use Ollama ourselves, because it is the fastest tool for that. What goes to production runs on vLLM.

What we do not do is scale up an Ollama setup as user numbers grow. The switch belongs made before waiting times become noticeable, not after.

Further guides

• AI Cube - local AI with vLLM and Open WebUI preinstalled
• GPU servers - sizing against actual concurrency
• LLM hosting - managed models without your own hardware
• vLLM, Ollama and Open WebUI - the components individually
• Size a local AI server by users - the calculation behind it
• Sizing GPU and VRAM for LLMs - KV cache, context length and FP8 KV cache

Unclear how many people your planned hardware supports? We work it through against model, context length and concurrency before anything is purchased. Book a call

Sources

• vLLM on GitHub
• Ollama on GitHub
• llama.cpp on GitHub
• vLLM: documentation
• Open WebUI: documentation
• SGLang on GitHub
• SGLang: release v0.5.20
• SGLang: release v0.5.21
• Ollama: release v0.40.0 (MLX on Apple silicon)
• SGLang: documentation
• Zheng et al.: SGLang, Efficient Execution of Structured Language Model Programs
• vLLM: Automatic Prefix Caching
• LMSYS: About