Sizing AI infrastructure properly? WZ-IT plans and operates inference environments on your hardware, in the AI Cube or in an EU cloud - including monitoring and cost control. See LLM hosting · See the operations layer
Anyone considering their own AI hears two terms that are constantly confused - and the confusion gets expensive: those who believe they need training plan clusters; those who only need inference often manage with a single GPU. As of October 2026.
Table of contents
- Inference and training compared
- Why most companies need no training
- When fine-tuning does pay off
- Sizing hardware correctly
- The cost calculation
Inference and training compared
Training creates or changes a model's weights: massive data, GPU clusters, days to weeks of compute. That is what model makers like Mistral or Meta do. Inference uses the finished model: a request goes in, an answer comes out - permanently, latency-critical, and it drives your running costs.
| Training | Inference | |
|---|---|---|
| Purpose | create/change the model | use the model |
| Hardware | GPU clusters (80 GB+ per card, many cards) | from one GPU, depending on the model |
| Duration | days to weeks, one-off | milliseconds, permanent |
| Who needs it | model makers, special labs | practically every company with its own AI |
Why most companies need no training
The most common fallacy: "The model should know our data, so we must train it." In most cases RAG is the right answer - the model stays unchanged and receives relevant documents at query time. That is cheaper, instantly updatable, and lets you enforce permissions technically - which trained knowledge never can: whatever sits in the model can be queried by anyone allowed to use it.
When fine-tuning does pay off
There are legitimate cases: a specific writing style, a domain language with its own terminology, structured output formats. Even then almost always as a LoRA adapter instead of full training - a small overlay is trained, the base model stays untouched, and the hardware requirement stays manageable. The order still holds: exhaust RAG first, then assess fine-tuning.
Sizing hardware correctly
For inference, VRAM matters most - the (quantized) model must fit into memory, plus room for context:
- up to 8B parameters (internal assistants, classification): one consumer GPU or L4, 16-24 GB VRAM
- up to 70B quantized (demanding assistance): 48-80 GB, one large card or two mid-size
- details: GPU and VRAM sizing for LLMs
Serving software like vLLM extracts a multiple of the throughput from the same hardware compared with naive setups - the software choice is a cost lever, not a detail.
The cost calculation
Own inference has largely fixed costs: GPU server, power, operations - regardless of whether 10 or 200 employees ask. APIs scale per token: cheap for sporadic use, expensive under sustained load. Break-even depends on volume and privacy requirements; a gateway like LiteLLM makes both paths usable in parallel and costs visible per team. Assessing which path pays off belongs in every intro call - both paths compared transparently, not as a matter of faith.
