Which LLM to self-host? Models and licenses

Select the model, runtime and GPU using real tasks? WZ-IT benchmarks exact model releases against your evaluation set, reviews licences and sizes inference, concurrency and operations together. See GPU servers

“Which LLM should I self-host?” cannot be answered by a leaderboard. A production model must solve real tasks in the required languages, run reliably on the chosen runtime, be legally usable and remain economical under concurrency. This article provides a reproducible selection process. As of August 2026.

Table of contents

• Not "the best" model but the right one
• Model size by use case
• The license matters too
• Open model families at a glance
• How to choose the model
• Capabilities that should actually be tested
• VRAM, context and concurrency
• From model test to safe operations
• How WZ-IT implements selection and operations
• Sources

Not "the best" model but the right one

Leaderboards and benchmarks tempt you to always want the largest, newest model. For production that is rarely the right decision. A model that reliably solves a well-defined task and runs on the available GPU is more valuable than a top model that blows the hardware or whose license restricts use.

The decision spans task quality, language, licence, context, tool and output formats, hardware, concurrency, latency and operational maturity. A model can be convincing in chat while being unreliable for valid JSON, German domain terminology or high concurrent load.

Model size by use case

The parameter count is the most important lever for hardware demand and capability. The rule of thumb: as large as necessary, not as large as possible.

• Smaller models (single-digit billion range) suffice for well-defined tasks - classification, simple summaries, structured extraction - and run on modest hardware.
• Medium models cover a broad everyday range: knowledge retrieval via RAG, drafts, assistance functions.
• Large models bring the broadest capabilities but demand considerably more GPU memory and compute more slowly.

With mixture-of-experts models, only part of the parameters may be active per token. Depending on runtime and distribution, all weights still need storage or placement across GPUs. Active parameter count is therefore not a complete VRAM figure. See Sizing GPU and VRAM.

The license matters too

An often overlooked point: not every “open” model has the same terms. Approval applies to an exact repository and release, not the brand.

• Permissive licences such as Apache 2.0 and MIT normally permit broad commercial use but still impose conditions such as preserving notices. Specific Qwen, Mistral or DeepSeek releases may use them.
• Community and model-specific licences add conditions. Meta's Llama licence includes attribution, acceptable-use terms and special conditions above a very large user threshold.
• Terms can differ inside one family. Mistral, for example, publishes some models under Apache 2.0 and other releases under a separate licence. Gemma uses its own terms.

Weights, inference code, tokenizer, fine-tune and dataset may have separate terms. A permissive model licence is often simpler, but it is not unrestricted and says nothing about data protection or quality.

Open model families at a glance

Families help with market orientation, while approval remains release-specific:

• Qwen - a broad range of sizes and capabilities; individual releases may use Apache 2.0.
• Mistral - a European provider with compact and larger models; terms vary by release.
• DeepSeek - general, reasoning and code-oriented releases; check the exact licence.
• Llama - broad ecosystem and runtime support under its own community licence.
• Gemma - Google's model family under separate terms of use.

This is not a ranking. A family name says little about domain-specific German, structured output or quality on your own document corpus.

How to choose the model

A pragmatic path in four steps:

1. Define tasks and failure limits, including expected output, prohibited errors and required languages.
2. Shortlist candidates by licence, context, runtime support, model card and hardware boundary.
3. Create an internal evaluation set of 30 to 100 representative and difficult cases.
4. Measure end to end, including prompt, RAG, tools and output validation.
5. Test operations, including concurrency, peaks, failure, updates and rollback.
6. Document approval, model ID, version, licence, quantisation, runtime and results.

Capabilities that should actually be tested

VRAM, context and concurrency

Memory includes weights, runtime overhead and KV cache. KV cache grows with context and simultaneous requests. Ollama documents that memory requirements scale with parallel requests multiplied by context length. A model that fits for one user may overload under four concurrent users.

Quantisation reduces memory but can affect quality and support. Benchmark the exact combination of model, quantisation, runtime, context and concurrency, including headroom for maintenance and rolling updates.

From model test to safe operations

Approved models remain versioned. New releases pass regression tests, a limited rollout and only then production switching. Gateway aliases avoid hard-coding applications to model IDs. Monitoring observes quality, latency and errors without retaining sensitive prompts by default.

A portfolio can be useful: a small extraction model, a stronger drafting model and a separate embedding model. LiteLLM can centralise routes, limits and fallbacks. What is local AI? explains the complete platform.

How WZ-IT implements selection and operations

WZ-IT combines model evaluation and infrastructure. We build a test set from intended tasks, review exact licences, measure quality and load, and size GPU servers or LLM hosting. Managed AI adds versioning, gateway, monitoring, updates, backup and rollback.

The target is not the model with the highest public score, but the smallest robust combination that meets quality and operational goals and remains replaceable.

Sources

• Qwen: example of an exact Apache 2.0 model licence
• Mistral 7B v0.3: Apache 2.0 declaration
• Mistral: example of a different model-specific licence
• Meta: Llama Community License
• Google: Gemma Terms of Use
• Ollama documentation: context, concurrency and memory