A knowledge system on your own documents - sovereign and with real permissions? WZ-IT builds and operates RAG systems on your own infrastructure - retrieval, vector database and permissions enforced technically, GDPR-compliant from one team. See the internal assistant
A language model knows the world up to its training cutoff - but not your contracts, manuals or tickets. RAG closes this gap by connecting the model with your own documents. This article explains what retrieval-augmented generation is, how it works, why it reduces hallucinations and where the real crux lies in the enterprise. As of July 2026.
Table of contents
- The problem: LLMs don't know your data
- How RAG works
- Embeddings and the vector database
- RAG against hallucinations
- The crux: permissions
The problem: LLMs don't know your data
A language model answers from what it learned during training. Two limits follow: its knowledge ends at a cutoff date, and it knows no internal, non-public information - exactly the documents that matter in the enterprise. Ask a pure LLM about the content of a specific contract, and at best it guesses plausibly.
You could try to train all the knowledge into the model - that is expensive, slow and impractical with changing data. RAG takes the other route: the model stays as it is and is given the relevant information at runtime.
How RAG works
RAG runs in two phases. During indexing, the documents are prepared:
- Each document is split into manageable chunks.
- Each chunk is translated into a vector (embedding) that captures its meaning.
- The vectors land in a vector database.
On a query, the flow reverses: the question is also translated into a vector, the vector database delivers the most similar passages by content, and these are passed to the language model together with the question as context. The model formulates the answer from this supplied context - ideally with a reference to the source.
Embeddings and the vector database
The core of RAG is the similarity search over embeddings. An embedding is a numeric representation of text that maps its meaning into many dimensions: two passages with similar content produce vectors that lie close together - regardless of whether they use the same words.
The vector database stores these vectors and quickly finds the nearest ones to a query vector. Widespread options are Qdrant and the Postgres extension pgvector - both open source and self-hostable. Beyond pure similarity search they can apply additional conditions as filters, which becomes decisive for permissions (see below).
RAG against hallucinations
RAG's biggest practical advantage is verifiability. Because the model answers from concretely retrieved passages instead of from training memory alone, answers become checkable and can be provided with source references. That reduces hallucinations considerably.
They are not eliminated, though. If the retrieval does not find the right passage, or it is poorly phrased, the model can still be wrong. RAG thus shifts the quality question from "What does the model know?" to "How good is the retrieval?" - and this is exactly where the quality of a RAG system is decided: at chunking, embeddings and the selection of hits.
The crux: permissions
In the enterprise the decisive requirement is added: a user may only see the content via RAG that they are authorized for. The obvious but wrong solution is to tell the model via a prompt not to show certain documents. A language model is not an access check - a single unlucky hit is enough, and a confidential passage lands in the answer.
The right way is to enforce access control technically before or during the vector search: the model receives only passages that are already authorized for the requesting user. What they may not see does not reach the model at all. That is exactly the difference between a nice prototype and a production knowledge system - and the reason RAG in the enterprise needs its own retrieval service with clean ACL logic. What that looks like in detail is shown by the path to connecting Open WebUI to Nextcloud with real permissions. Where RAG sits in the bigger picture is placed by The open-source LLM stack.
