The open-source LLM stack: running models yourself

Have local AI operated in production? WZ-IT builds and operates the complete open-source LLM stack on your own infrastructure - GPU hardware, inference, gateway, observability and RAG, GDPR-compliant from one team. See LLM hosting

Self-hosting an LLM is rarely "install one tool". Production needs a stack of several open-source components, each covering one layer - from raw inference to auditable multi-user use. This article places the layers and the key tools: vLLM, Ollama, LiteLLM, Langfuse, Open WebUI and the vector database. As of July 2026.

Table of contents

• Why a stack instead of one tool
• The five layers at a glance
• Inference: serving the model
• Gateway and observability
• Frontend and knowledge
• Operated sovereignly, not cobbled together

Why a stack instead of one tool

Getting a model running is easy today - one command, and Ollama serves an LLM. The leap to a production system is the hard part: several users and applications access it, access must be controlled and costs capped, every request should be traceable, and knowledge from your own documents should flow in without violating permissions.

No single component solves all of that. That is why a modular stack has become established, in which each layer takes on a clearly defined task and talks to the others over open, mostly OpenAI-compatible interfaces. The big advantage: each layer is replaceable, and everything runs on your own infrastructure.

The five layers at a glance

Not every installation needs all five. A single use case gets by with inference and gateway; a knowledge assistant adds frontend and vector database. The stack grows with the requirements.

Inference: serving the model

The bottom layer serves the language model and answers requests. Two tools dominate:

• vLLM - a fast inference and serving engine with high throughput (PagedAttention, continuous batching) and an OpenAI-compatible API server. The right choice when many requests should run efficiently across GPUs.
• Ollama - the simplest entry point: one command, and a model runs locally. Ideal for quick tests and small loads, with less throughput than vLLM under concurrent requests.

Which engine fits depends on load and ambition - details in What is vLLM?.

Gateway and observability

Above inference sit two layers that turn a running model into an operable service:

LiteLLM is the gateway: a single, OpenAI-compatible interface across more than 100 models. Applications talk only to the gateway, whether vLLM, Ollama or an external model answers behind it. This brings central authentication, budgets, rate limits and fallback into one place - more in What is LiteLLM?.

Langfuse is the observability: an open-source platform that records every request as a traceable trace, plus evaluations, metrics and prompt management. That is valuable not only for debugging but also for the evidence and logging requirements the EU AI Act places on AI systems.

Frontend and knowledge

For users a surface is missing, for knowledge systems a memory:

Open WebUI is the widespread frontend: a self-hosted web interface with user, role and group management, a connection to the gateway, a RAG function and audit logging, connected to your own directory via OIDC or LDAP. It turns the stack into a tool that non-technical staff can use too.

The vector database (such as Qdrant or the Postgres extension pgvector) stores the knowledge from your own documents as vectors so the model can access it - the basis for retrieval-augmented generation (RAG). Decisive in the enterprise: permissions must be enforced technically here, not left to the model.

Operated sovereignly, not cobbled together

The real value of the open-source stack is sovereignty: each layer runs on your own infrastructure - GPU server, Proxmox or bare metal - so that neither prompts nor documents leave the building. That is the difference from cloud AI services and the basis for GDPR compliance.

The price is integration work: five components have to interlock cleanly, be secured, monitored and kept up to date. This is exactly where the experiment separates from the production system. WZ-IT plans, builds and operates this stack from one team - from the hardware to the RAG knowledge system. How the gateway and inference work in detail is deepened in What is LiteLLM? and What is vLLM?.