vLLM, Ollama or llama.cpp: which inference server for business use

Local AI for several people at once? WZ-IT ships the AI Cube with vLLM and Open WebUI preinstalled, sizes GPU servers against actual concurrency or provides managed LLM hosting. Clarify your sizing with us

The question of the right inference server is almost always framed as a speed question and almost always answered wrongly, because one figure is missing: how many people work at the same time.

For a single user, the three common tools sit close together. At ten concurrent requests they separate by factors, not percentages. Anyone choosing on single-user measurements and then putting a team on it has bought something that does not hold up in daily use.

This article assesses the three along that figure. All figures from the repositories, as of 24 August 2026.

Table of contents

• Three tools, three levels
• The three at a glance
• The decisive difference: concurrent users
• llama.cpp
• Ollama
• vLLM
• What we ship in the AI Cube and why
• Which tool for which situation
• What a throughput comparison does not answer
• How we approach this at WZ-IT
• Further guides

Three tools, three levels

The most common mistake in comparisons is placing the three side by side as if they were interchangeable. They sit at different levels:

llama.cpp is the engine. It runs the model, works on CPU and GPU, and is the foundation much else builds on.

Ollama is the usability layer. It uses llama.cpp and adds convenient management on top: pull models, start them, address them. Comparing Ollama with llama.cpp therefore compares a wrapper with its contents.

vLLM is a standalone serving system. It does not build on llama.cpp but takes a different approach designed for GPU operation with many concurrent requests.

It follows that the real decision lies between the llama.cpp branch and vLLM. Whether you address llama.cpp directly or through Ollama is a question of convenience.

The three at a glance

Figures from the public repositories, as of 24 August 2026.

All three are unrestricted open source and actively developed. The star counts reflect visibility, not suitability: Ollama is the best known because it is the easiest entry point, not because it is the best choice for production.

The decisive difference: concurrent users

This is where the tools separate, and not gradually.

Published comparisons cite factors of ten and more for vLLM under concurrency against the llama.cpp branch. The reason is not a better implementation of the same thing but a different technique: vLLM batches requests arriving at the same time and processes them together rather than sequentially.

With one user that advantage largely disappears. There, llama.cpp used directly is even slightly faster than Ollama, because Ollama as a wrapper adds overhead, on the order of a low double-digit percentage.

Practical consequence: the jump from one to ten users is not a scaling problem solved with more hardware. It is an architectural question. A setup that works well for one person does not become good for a team through a larger graphics card but through a different serving approach.

The question before any hardware purchase is therefore: how many people actually work at the same time? Not how many should get access. Those two numbers usually differ by an order of magnitude.

llama.cpp

The engine beneath much else, under MIT, with 125,380 stars. Runs on CPU, on GPU and on Apple silicon, and is the right choice wherever no GPU is available.

Strength: runs practically everywhere, the smallest resource footprint, the greatest control over settings. Anyone working with embedded systems or CPU-only machines ends up here.

Limit: no management layer, no model management, no multi-user operation in a meaningful sense.

Ollama

The most widely adopted of the three at 179,309 stars, under MIT. Installation and a first model take minutes, and the handling borrows from container tooling.

Strength: by far the easiest entry point. For development, demos and the question "is a local model any good for our case at all", Ollama is the right tool.

Limit: it remains a layer over llama.cpp and inherits its behaviour under concurrent requests. For a team working in parallel it is the wrong foundation, and that only becomes apparent in operation.

vLLM

Under Apache-2.0, 89,860 stars, built for GPU serving. Offers an OpenAI-compatible API, so applications usually connect without modification.

Strength: the only one of the three designed for many concurrent requests. That is where the factors sit, not the percentages.

Limit: practically requires a GPU, is more involved to set up than Ollama, and is oversized for a single user. Anyone just experimenting does not start here.

What we ship in the AI Cube and why

In the AI Cube we ship vLLM together with Open WebUI preinstalled. That decision follows directly from the device's purpose: it sits inside a company and is used by several people.

With Ollama, setup would have been simpler and the device worse in daily use. The experience behind that is unremarkable: a setup that convinces in a test by one person disappoints at five concurrent requests, and then the purchase is questioned rather than the software choice.

Open WebUI sits on top as the interface, because users expect a chat interface with permissions, knowledge areas and history, not a programming interface. How that connects to knowledge sources belongs to setup rather than to the inference server.

Which tool for which situation

What a throughput comparison does not answer

Does the model fit in memory at all? The choice of serving system is secondary to whether model and context length fit into available GPU memory. An oversized model is not made fast by any serving approach.

How long are the contexts? Long inputs, for instance in document search, consume memory per concurrent request. The number of users that can be served in parallel drops considerably, and that is regularly overlooked in sizing.

Who updates models and the serving layer? Both move quickly. A setup left untouched for a year keeps running but loses access to newer models.

What happens if the GPU fails? A local AI system without a fallback is entirely down on a hardware fault. Whether that is acceptable belongs settled before the purchase.

How we approach this at WZ-IT

We ask about the number of concurrent users before choosing a tool, and we insist, because the first answer is almost always the number of people with access. From that figure follows the serving approach, then the hardware, and only then the model.

For trial setups and the question of basic suitability we use Ollama ourselves, because it is the fastest tool for that. What goes to production runs on vLLM.

What we do not do is scale up an Ollama setup as user numbers grow. The switch belongs made before waiting times become noticeable, not after.

Further guides

• AI Cube - local AI with vLLM and Open WebUI preinstalled
• GPU servers - sizing against actual concurrency
• LLM hosting - managed models without your own hardware
• vLLM, Ollama and Open WebUI - the components individually
• Size a local AI server by users - the calculation behind it

Unclear how many people your planned hardware supports? We work it through against model, context length and concurrency before anything is purchased. Book a call

Sources

• vLLM on GitHub
• Ollama on GitHub
• llama.cpp on GitHub
• vLLM: documentation
• Open WebUI: documentation