Setup & Configuration (optional)
On request: installation of the models you want (Llama, Qwen, Gemma, Mistral, DeepSeek, GPT-OSS), Ollama or vLLM as the model server, OpenWebUI as the web interface and optional API endpoints for your applications.
LLM hosting in Germany: run open-source LLMs on a dedicated GPU
You want to host a language model yourself, without sending data to cloud APIs and without a team of your own for GPU, vLLM and updates. We run open-source LLMs such as Qwen, gpt-oss or Mistral on dedicated GPU servers in Germany, with an OpenAI-compatible API.
- Qwen · gpt-oss · Mistral
- OpenAI-compatible API
- Ollama & vLLM
Why local
Why host an LLM yourself? Reasons for operating in Germany
Full control over your AI infrastructure
Data Protection & GDPR
German locations can be selected for models, storage and processing. External integrations, remote access and subprocessors are documented so that the specific privacy assessment remains traceable.
Data Sovereignty
You determine the models, interfaces and data flows. Unneeded external model APIs can be omitted; enabled integrations and outbound connections are documented transparently.
Compliance & Audits
Meet strict compliance requirements in regulated industries such as healthcare, finance or public administration.
Cost Efficiency
No hidden API costs, no surprises when it comes to billing. Predictable monthly fixed costs instead of pay-per-token with cloud providers.
Performance & Latency
Dedicated GPU resources without sharing. Optimal latency for your applications without dependence on global cloud services.
Adaptability
Fine-tuning and customization of your models to your specific requirements. No restrictions from API limits or vendor lock-ins.
Comparison
LLM Hosting vs. Cloud APIs
Why self-hosting is often the better choice
| Feature | Hosted by WZ-IT | Cloud APIs (OpenAI, etc.) |
|---|---|---|
| Data Privacy | Hosted in Germany, privacy-focused | Data goes to US providers |
| Costs | Fixed monthly costs | Variable token prices, often more expensive |
| Control | Full control over models | Dependency on provider |
| Customization | Fine-tuning possible anytime | Limited or expensive |
| Latency | Optimal (Germany) | Variable, depends on region |
| Availability | Dedicated resources, no rate limits | Rate limits, outages possible |
Service scope
LLM hosting with operations: what is included
We take care of everything - you simply use your LLMs. From initial setup to daily operations: Our Managed LLM Hosting handles all technical aspects.
Operations & Maintenance
We monitor your LLM infrastructure around the clock, perform system updates and optimize GPU performance. How quickly we respond to an alert depends on the agreed service level.
Support & Consulting
Our team supports you with model selection, integration into your applications and optimization for your specific use cases. Priority support via email and optionally phone/video.
Transparent Pricing
Fixed monthly costs without hidden fees, no pay-per-token billing. Single-GPU servers are cancellable monthly; the term of multi-GPU servers is stated in the proposal. We agree the concrete price based on the configuration in the initial call.
Models
Open-source LLMs for hosting: supported models
A selection of current open-weight models, as of October 2026. Sizes refer to the weights; context and concurrent requests need additional memory.
gpt-oss-120b and gpt-oss-20b
OpenAI's open models under Apache 2.0: 120b is a mixture-of-experts with 117B parameters, 5.1B active, and 128K context. 120b fits on one 96 GB card, 20b on a 24 GB card.- Weights (MXFP4): about 65 GB (120b), about 14 GB (20b)
Gemma 4
Google's current generation under Apache 2.0, as a 31B dense model or a 26B mixture-of-experts with 3.8B active parameters, each with 256K context.- Weights: about 62 GB (31B, BF16), about 23 GB (31B, W4A16)
DeepSeek-V4-Flash
Large mixture-of-experts model under the MIT licence with 284B parameters, 13B active, and up to 1M tokens of context. Realistic on the server with four RTX PRO 6000 (384 GB).- Weights: about 167 GB (mixed FP4/FP8)
Mistral Small 4
European model by Mistral AI under Apache 2.0: 119B parameters as a mixture-of-experts, 6.5B active, with 256K context. In NVFP4 it fits on one 96 GB card.- Weights: about 71 GB (NVFP4), about 121 GB (FP8)
EuroLLM 22B
European open-weight model under Apache 2.0, designed for European languages: 22B dense parameters, 32K context. Fits on one 96 GB card.- Weights: about 45 GB (BF16)
Qwen3.8-27B
Alibaba's current dense model under Apache 2.0 with 27.8B parameters and 262K context, a strong all-rounder for text and code. For larger tasks there is Qwen3.5-122B on two GPUs.- Weights: about 31 GB (FP8), about 22 GB (NVFP4)
Model pages with licence, sizes and recommended server tier:
Qwen hosting in Germanygpt-oss hosting in GermanyGemma hosting in GermanyMistral hosting in GermanyDevstral hosting in GermanyDeepSeek hosting in GermanyLlama hosting in GermanyRuntime
We Support Ollama & vLLM
Ollama
- Simple installation and operation
- OpenAI-compatible API
- Automatic model management
The easiest way to run LLMs locally. Perfect for development, prototyping, and small to medium production workloads.
vLLM
- Considerably higher throughput under high load
- PagedAttention for efficient memory management
- Continuous batching for maximum throughput
High-performance inference engine for production workloads. Optimized for maximum throughput and minimal latency under high load.
Which framework is right for you? The managed GPU server package is built on vLLM because it handles many concurrent requests efficiently. Ollama suits simple use cases, development and moderate load. We clarify which framework fits your use case in the initial consultation.
Infrastructure
Infrastructure Requirements for LLM Hosting
What is needed for professional LLM hosting? Hosting Large Language Models places special demands on the infrastructure. We ensure that everything is optimally configured.
Hardware & GPU
LLMs need GPUs with enough VRAM. Rule of thumb for the weights: quantized (Q4) roughly half a gigabyte per billion parameters, in FP16 about four times that. Context and KV cache add on top depending on context length and concurrency - so we plan around 48 GB for Llama 3.1 70B and around 20 GB for Gemma 3 27B.
Memory & Storage
In addition to GPU memory, you need sufficient RAM (at least 64 GB) and fast NVMe storage for model files and caching. The model files themselves take roughly 15 GB (7B) to 150 GB (70B) on disk in FP16; quantized operation needs correspondingly less.
Network & Bandwidth
For production applications with multiple users, a stable, fast network connection is essential. Our servers offer gigabit connectivity with low latency within Germany.
Software & Updates
The software stack including CUDA drivers, vLLM as the model server, Open WebUI and the container runtime is installed, configured and kept up to date by us. On request we set up Ollama for development and testing.
Monitoring & Security
Professional monitoring of GPU utilization, temperature monitoring and security updates are part of the service. Automated backups are available as an option.
Scaling & Load Balancing
As requirements grow, we scale your infrastructure horizontally (multiple servers) or vertically (more powerful GPUs). Load balancing between multiple instances is possible.
Minimum recommended server tier by model size
The starting point is the size of the model weights in the chosen format. Memory for context, concurrent requests and the runtime comes on top.
| Weights up to about | Minimum recommended | Examples (as of October 2026) |
|---|---|---|
| 19 GB | Managed GPU Server 24 | gpt-oss-20b (MXFP4, about 14 GB) |
| 75 GB | Managed GPU Server 96 | Qwen3.8-27B (FP8, about 31 GB), gpt-oss-120b (MXFP4, about 65 GB), Mistral Small 4 (NVFP4, about 71 GB) |
| 150 GB | Managed GPU Server 192 | Qwen3.5-122B (FP8, about 127 GB), Mistral Small 4 (FP8, about 121 GB) |
| 300 GB | Managed GPU Server 384 | DeepSeek-V4-Flash (about 167 GB) |
Which tier fits your use depends on context length and concurrent requests. We check this before the proposal. The three-GPU server (288 GB) is meant for several models side by side.
View server tiers and pricesPricing
LLM hosting on a managed GPU server
Fixed monthly costs without hidden fees, no pay-per-token billing. Single-GPU servers are cancellable monthly; the term of multi-GPU servers is stated in the proposal. We agree the concrete price based on the configuration in the initial call.
Managed GPU Server 24€699 excl. VAT / month
What's included?
- Dedicated GPU server in German data center
- Data center, power & network
- 24/7 monitoring
- Security updates & system maintenance
- Privacy-focused infrastructure; the data center operator is certified to ISO 27001 & BSI C5
- Ollama/vLLM installation (optional)
- Free initial consultation on model selection
- Provisioning as agreed in the initial call
Payment
- Bank transfer
All prices are net and exclude statutory VAT. The offers are addressed to businesses.
Related Services
TurnkeyAI platform
Open WebUI with gpt-oss, Qwen and Ministral, operated in Germany by German providers, SSO and Section 203 supplementary agreement, without your own GPU.
View →AI Server with GPUGPU Server for AI
Our hardware base for LLM hosting. Powerful GPUs for inference & fine-tuning
View →On-premisesAI Cube
Local AI without vendor lock-in - AI Cube from €7,990 excl. VAT one-time plus AI Cube Care from €349.90 excl. VAT / month.
View →Who it fits
Who Benefits from LLM Hosting?
Typical use cases and target groups
Agencies & Service Providers
Offer your clients AI-powered services like content creation, SEO analysis or chatbots - with your own LLM infrastructure instead of expensive OpenAI API costs.
Research & Education
Universities, research institutions and educational organizations use their own LLMs for scientific work, studies and teaching without data privacy concerns.
Mid-sized Companies
SMEs in Germany use LLMs for internal knowledge bases, customer service automation, code analysis or document classification.
Internal Chatbot with RAG
Connect your LLM with your own documents, wikis or databases. Employees ask questions in natural language and receive precise answers from your knowledge base.
Code Assistance
Development teams use LLMs locally for code completion, review and documentation - without sending source code to external APIs.
Content Generation
Create product descriptions, marketing texts or social media content with your own LLM in your corporate language.
Document Analysis
Analyze and classify large volumes of documents, contracts or emails automatically and privacy-focused.
RAG (Retrieval-Augmented Generation)
With RAG, you connect your LLM with external knowledge sources. The model searches your documents and generates answers based on actual facts from your database. Ideal for company wikis, support databases or research archives.
Internal AI assistant with RAG
Overview
What are Large Language Models (LLMs)?
Large Language Models (LLM) are AI models that can understand and generate natural language. For companies, they offer enormous opportunities: from automating customer communication and intelligent document analysis to coding assistants and knowledge management.
With our LLM Hosting Germany, you can operate these powerful models in your own privacy-focused infrastructure - without having to pass on your sensitive data to global cloud providers.
Whether gpt-oss, Qwen, Mistral or other open models - we handle installation, operations and optimisation of your LLM infrastructure.
Frequently Asked Questions (FAQ)
Everything you need to know about LLM hosting
- Yes, if vLLM supports the model and its licence allows us to operate it. Common families such as Qwen, Gemma, Mistral, DeepSeek and gpt-oss qualify. We check licence, size and the right GPU before the proposal and support you in converting or quantising your own models.
- Absolutely. You can upgrade to a more powerful GPU at any time (e.g. from RTX PRO 4000 to RTX PRO 6000) or add additional servers for load balancing. We offer both vertical and horizontal scaling.
- Models that vLLM supports and whose licence allows us to operate them, for example gpt-oss, Qwen3.8 and Qwen3.5, Gemma 4, Mistral Small 4, DeepSeek-V4-Flash, EuroLLM and Llama 3.3. Meta's use policy excludes companies based in the EU from the multimodal Llama 4 models, so we do not operate them. We check which model fits which GPU before the proposal. As of October 2026.
- The managed GPU server package uses vLLM on an RTX PRO 4000 Blackwell (24 GB) or RTX PRO 6000 Blackwell (96 GB). With vLLM, significantly higher throughput is possible under high load; in its own benchmarks the vLLM project measures up to 24x more throughput compared to HuggingFace Transformers. Exact performance depends on the model, context length and the number of simultaneous requests.
- Yes, both Ollama and vLLM provide OpenAI-compatible APIs. You can easily switch your applications by changing the API URL - the code remains almost identical. Additionally, there are native client libraries for Python, JavaScript, Go and more languages.
- Our monitoring usually detects problems before they have an impact. In case of disruptions, we respond immediately. As a managed service customer, you have priority support via email, with optional phone/video support available.
- Yes, depending on the available VRAM you can load multiple models in parallel or switch between them. Ollama manages this automatically. On the Pro configuration with 96 GB VRAM you can, for example, keep a 27B and a 7B model loaded at the same time.
- We keep the system (OS, CUDA, Ollama/vLLM) up-to-date and install security updates promptly. Model updates are done on request - you decide when you want to use a new model release.
- A managed GPU server with one GPU costs €699 excl. VAT / month (24 GB) or €1,799 excl. VAT / month (96 GB), each plus a one-time setup fee and cancellable monthly. Servers with two to four GPUs are available on request. There is no billing per token; model, vLLM, Open WebUI and operations are included.
- You need a GPU with enough memory for the model weights and context, a model server such as vLLM, an interface or API for users, and someone who takes care of drivers, updates and monitoring. Our guide to the open-source LLM stack describes the setup; you can hand operations over to us.
- Operation in a German data centre under a data processing agreement removes the hard parts: no third-country transfer and no data sent to the model vendor. Legal basis, transparency and, where needed, a data protection impact assessment remain your responsibility; we provide the technical and contractual basis.
Knowledge base
In-depth guides on this topic
Guides
Related tutorials & guides
- Open WebUI, LibreChat, AnythingLLM & Co. Compared 2026Six open-source platforms are widely used for AI chat in the enterprise: Open WebUI, LibreChat, AnythingLLM, LobeHub, Onyx and PrivateGPT. They solve different tasks but...
- Hetzner GPU Servers Compared: GEX45, GEX63 and GEX131 (October 2026)Hetzner rebuilt its GPU range in autumn 2026: the GEX45 arrived on 1 September, the GEX63 on 6 October, and the GEX44 can no longer...
- IT and Digitalisation Funding in NRW 2026: What Is Open and What Has EndedAnyone planning an IT or digitalisation project with public funding in 2026 will find many guides that still mention programmes that no longer exist. go-digital...
- LiteLLM Vulnerability GHSA-7hp6-4w63-5g45: Check Versions and Harden the GatewayOn 30 September 2026, BerriAI published a security advisory for the LiteLLM proxy: GHSA-7hp6-4w63-5g45. An authenticated user with the internaluser role can escalate to proxyadmin...
- Shadow AI in companies: why a policy alone is not enoughIn many companies, AI is in use before anyone has decided on it. Employees draft emails with a private chat account, have contracts summarised or...
- Local, GDPR-Compliant Speech Recognition: Whisper, Voxtral and Parakeet ComparedSpeech recognition is one of the AI applications with the clearest benefit: dictation becomes text, meetings become minutes, phone calls become searchable notes. At the...
- RAG Knowledge Base for Public Authorities: A Guide for the Public SectorPublic authorities manage large collections that have grown over years: decrees and circulars, procedural instructions, technical reports, expert opinions, statements, minutes. People looking for a...
- RAG and the EU AI Act: when a RAG system is high-risk AIAnyone planning a RAG knowledge base soon runs into the question of whether the system falls under the high-risk rules of the EU AI Act....
- Extract and Validate Documents with AI: Self-Hosted Document ProcessingSince 1 January 2025 every German company has to be able to receive e-invoices. Part of the incoming documents now arrive as structured XML and...
- Self-Hosted Langfuse: LLM Observability and AI Logging for the EU AI ActAnyone bringing AI into production business processes quickly faces an uncomfortable question: what is actually happening in there? Which prompt went to which model, why...
- Bleeding Llama (CVE-2026-7482): Why Self-Hosted AI Isn't Automatically Secure AIThree unauthenticated API calls. No login, no exploit framework, no privilege escalation. Three POST requests to a default port, and the machine's memory is on...
- EU AI Act: High-Risk Obligations Deferred to December 2027 - What Still Starts in August 2026The EU AI Act's high-risk obligations do not start on 2 August 2026. Stand-alone high-risk systems under Annex III now have until 2 December 2027,...
- GPT-OSS 120B on the AI Cube: Run OpenAI's Open-Weight Model LocallyOpenAI released GPT-OSS 120B as an open-weight reasoning model on 5 August 2025. Its native MXFP4 quantisation allows OpenAI to position the model for a...
Reviews & projects
What clients say about working with us
From a knowledge base with an AI assistant and local AI integration to data sovereignty and ongoing operations.
AI Platform
We were looking for a secure yet easy-to-use alternative to the large American providers and came across WZ-IT. WZ-IT advised us on choosing the right models and implemented the integration for us. Beforehand, we were able to try everything out calmly on a test instance. Today WZ-IT runs Open WebUI for us on our own domain, with a convenient Microsoft login and open-source models hosted by a German provider. Our team can now work productively with AI without resorting to external tools or shadow AI. Working together is straightforward, and questions are answered quickly and clearly. With WZ-IT we also have a partner at our side for further AI projects. (translated)
Carl-Frederic KornHead of E-Commerce & Online Marketing, Krüger Dirndl GmbH
Knowledge Management & AI
The knowledge from many years in the solar industry lived in the head of our former managing director, and it still lives in the heads of our employees today. We wanted to secure it, not as a folder full of notes, but in a way that lets us ask questions in daily work and get reliable answers. WZ-IT advised us from the start and weighed different approaches and knowledge management systems against each other with us before we decided. What was built is a custom knowledge base with an AI assistant that brings together conversations, notes, documentation and manuals: from the platform through onboarding our team to support in operation. Conversations are transcribed and facts are extracted; we review and correct everything ourselves before it is stored, because we do not want to trust an AI blindly. The system runs in our own environment, and the data stays with us. What surprised us was how efficient it was: WZ-IT used existing open-source software and extended it specifically for our requirements, instead of selling an expensive custom build from scratch, as is often usual with IT service providers. Timo and Robin understood our actual problem, preserving knowledge during the generational handover, and implemented it step by step: from the proof of concept through setup and validation with our own data to productive operation, explaining everything clearly along the way. Next, we want to extend this to the knowledge of our employees, so that this does not depend on individual people either.
Joshua von AmelnManaging Director, solarwerkstatt berlin GmbH
Read client reviews
International
Selected projects
- LiveRemote access platform for distributed water treatment plantsABCO Water Systems · Australia · Secure Access & OT
- 81%AWS migration with substantially lower operating costsEVA Real Estate · UAE · Cloud exit & infrastructure
- 98%Provision new sites in minutes instead of hoursnextGYM GmbH · Germany · Automation & IoT
WZ-IT designs, develops and operates infrastructure and software for clients in Germany and internationally. We deliver projects remotely and continue supporting them in ongoing operations after go-live.
Enquiry
Assess your LLM hosting requirements
Describe models, access, and data sources. We derive the infrastructure and operational scope.
- Straight with Timo and Robin - no sales team, no pitch
- A professional assessment of your situation
- Concrete next steps for infrastructure, software or AI
The conversation is free and non-binding.
WZ-IT's advice on our Azure migration was technically sound and completely non-binding right from the intro call - we took away a great deal.
Jakob ÖschlbergerInno7 GmbH
WZ-IT's advice on our Azure migration was technically sound and completely non-binding right from the intro call - we took away a great deal.