A knowledge assistant on your own documents, tested rather than demonstrated? The RAG Proof of Value tests one defined set of documents along the approach in this guide: reference questions with source and not-in-corpus tests, a baseline measurement, two optimisation cycles and a quality report, from €9,900 net plus separately itemised third-party costs. View the RAG Proof of Value · Internal AI assistant
A RAG system is quickly built as a demo. Whether it holds up in daily use shows in questions nobody picked for the presentation. This guide arranges the introduction in seven phases, each with a result and an acceptance criterion, and points to our detailed guides. It adds architecture options, cost types, failure patterns and the build-or-commission question. As of October 2026; third-party information was retrieved on 28 October 2026.
Table of contents
- The seven phases at a glance
- Phase 1: Use case and reference questions
- Phase 2: Data sources and permissions
- Phase 3: Ingestion and chunking
- Phase 4: Retrieval and reranking
- Phase 5: Generation
- Phase 6: Evaluation and acceptance
- Phase 7: Operations and updates
- Architecture options
- Cost types and drivers
- Typical failure patterns
- Build it yourself or commission it
- Data protection and permissions
- Status and methodology
- How WZ-IT does it
- Further guides
The seven phases at a glance
The test set exists before the first line of code, and every technical decision is measured against it. Microsoft also divides RAG development into phases (preparation, chunking, chunk enrichment, embedding, information retrieval, end-to-end evaluation of the language model) and advises evaluating each step independently for optimisation, keeping the end result in mind, and determining your own acceptance criteria for each step (Azure Architecture Center).
| Phase | Result | Acceptance criterion |
|---|---|---|
| 1 Use case and reference questions | Test set with question types, expected sources and criticality | The business domain has confirmed every question and source |
| 2 Data sources and permissions | Source inventory with owner, format, currency and permission model | Every source has an owner and a permission model; duplicates and old versions are decided |
| 3 Ingestion and chunking | Pipeline with extraction, chunking and metadata | A sample with tables, scans and headings passes |
| 4 Retrieval and reranking | Baseline search, then hybrid search and reranker | The effect of every measure is measured against the baseline |
| 5 Generation | Prompt, model and answer format with source citation | Not-in-corpus cases pass; every answer names a verifiable source |
| 6 Evaluation and acceptance | Measurement report per question type | Blocking cases pass; thresholds agreed in advance are met |
| 7 Operations and updates | Sync with the source, regression test, responsibility | Changes to model, index or prompt trigger a test run; deletions take effect |
Phase 1: Use case and reference questions
A use case is bounded when a business domain can name real questions and knows which document holds the answer. The questions come from tickets, emails and intranet searches, not from the documents. Add question types that demos leave out: exact identifiers, conflicting versions, questions without an answer in the corpus, and permission tests. Structure and size are covered in Measuring RAG quality.
First clarify whether RAG is the right answer. Structured data in databases is better served by text-to-SQL; training the model does not replace RAG for changing corpora (What is RAG?).
Phase 2: Data sources and permissions
For each source, record owner, format, currency, permission model and personal data. If three versions of a manual exist, the system will eventually cite the wrong one; old versions and duplicates are therefore decided before ingestion.
Permissions belong in the data model before the first file is indexed: the index stores who may see a passage, and the search filters before selecting hits. A note in the prompt does not replace that (RAG with permissions). For connecting Nextcloud, SharePoint and DMS see data sources.
Phase 3: Ingestion and chunking
Ingestion decides what the search can find. A sample shows whether tables, headings and scanned pages arrive as text; no better model compensates for errors here. Documents are split by structure such as chapters and paragraphs, and every passage carries source, version, date and permissions as metadata (chunking, contextual retrieval). The embedding model is recorded, because changing it requires re-indexing (embedding models for German).
Phase 4: Retrieval and reranking
Start with the simplest search; its measurement is the baseline. Measures are added one at a time and judged against it: hybrid search combining keyword and vector search for identifiers and technical terms, and a reranker for ordering (hybrid search and reranking). The vector database follows the requirements for filters and operations (Qdrant vs. pgvector). In Open WebUI, hybrid search is off by default (ENABLE_RAG_HYBRID_SEARCH) and RAG_TOP_K is set to 3 (environment variables); the test set decides both.
Phase 5: Generation
The model should answer only from the supplied context, name the source and state a knowledge gap instead of guessing. A clear system prompt and an output format with source citation suffice; the not-in-corpus cases show whether it holds. Documents are third-party input: instructions inside them must not steer the model (prompt injection). For model choice see Which LLM to self-host?.
Phase 6: Evaluation and acceptance
Measurement happens on two levels, because search and answer fail independently. The metrics are definitions, not standard values: context recall, context precision and faithfulness come from Ragas (Apache 2.0), Recall@k from classic information retrieval.
| Level | Metric | Definition |
|---|---|---|
| Retrieval | Recall@k | Share of the expected sources among the k hits passed to the model |
| Retrieval | Context recall | Share of the claims in the reference answer that the retrieved context backs (Ragas) |
| Retrieval | Context precision | Whether relevant passages rank before irrelevant ones, computed as the mean of precision@k (Ragas) |
| Answer | Faithfulness | Share of the claims in the answer that the retrieved context supports (Ragas) |
Faithfulness measures fidelity to the context, not correctness: an answer drawn from an outdated version can be fully supported and wrong. It is therefore read together with context recall. Frameworks define metrics differently; values are comparable only within the same framework and version.
Before measuring, three things are fixed in writing:
- Blocking cases that must pass without exception: permission tests, not-in-corpus cases and questions where an error does harm.
- Thresholds for the remaining metrics, derived from the risk of the use case and set relative to the baseline. Target figures from other projects mean little without reference to your own test set.
- The configuration state of the measurement: model versions, chunking, prompt and index state.
If a language model scores automatically, experts check a sample. Execution: Measuring RAG quality.
Phase 7: Operations and updates
The index must follow changes, deletions and revoked permissions in the source; for Open WebUI, oikb syncs files incrementally, that is changes and deletions, without permissions per file. Every change to model, embedding, chunking, reranker or prompt triggers the regression run on the test set. Real questions that failed join the test set. Langfuse handles tracing and scoring in operation. This also needs a named owner for the content: without one, corpora age and users lose trust.
Architecture options
The choice follows the granularity of permissions and the number of sources, not the tool.
| Option | Setup | Fits when | Limit |
|---|---|---|---|
| 1 Open WebUI with knowledge areas | Files or sync via oikb into knowledge bases; vector database selectable via VECTOR_DB (e.g. Chroma, pgvector, Qdrant, Milvus) |
a few knowledge areas, access per group is enough | Permissions apply per knowledge base, not per file; Chroma, the default (pgvector in the slim image), is according to the docs not suited to multiple workers or replicas in its default local mode |
| 2 Source-native indexing | e.g. Nextcloud Context Chat; the Context Agent provides an MCP endpoint for Open WebUI | the team works in one system whose permissions should apply | tied to the capabilities of the source system |
| 3 Own retrieval service | Index with an access list in the payload, identity passed through, connected as an MCP or OpenAPI tool | permissions per file, confidential corpora | the service is operated, tested and updated |
| 4 Own application | Pipeline, API and interface from building blocks | RAG is part of a product or process | highest own effort for evaluation and operations |
Source for VECTOR_DB and Chroma: Open WebUI documentation. Options 1 to 3 are described in Connect Open WebUI to Nextcloud. Since version 0.6.6, Open WebUI has been under its own Open WebUI License with a branding clause (assessment).
Cost types and drivers
A figure without corpus, permission and operations details is not comparable. Cost types:
| Cost type | Kind | Drivers |
|---|---|---|
| Inventory and preparation | one-time, later ongoing | number of sources, condition of the storage, resolving versions and permissions |
| Test set and acceptance | one-time, regression runs ongoing | question types, risk, time of the business domain |
| Connecting the sources | one-time and ongoing | interfaces, sync including deletions, carrying over permissions |
| Embedding, reranking and index | ongoing, one-time on re-indexing | number of passages, filters, availability, backup; local hardware, via API tokens |
| Language model | ongoing | users, requests and tokens per request; local hardware, via API token price |
| Interface and identity | one-time | SSO, roles, standard interface or own application |
| Operations and care | ongoing | updates, monitoring, content editing, service hours |
Tokens per request can be estimated: input = system prompt + question + k × passage size. With assumed 300 tokens of system prompt, 50 tokens of question and k = 5 passages of 500 tokens, that is 2,850 input tokens; at k = 10 it is 5,350. The number of passages is thus a cost lever that the test set decides. Multiply by the model's token price for the amount.
Published market figures vary (as of 28 October 2026). skill-sprinters states 8,000 to 25,000 euros one-time and 200 to 2,000 euros per month (field report). pexon states 15,000 to 20,000 euros for a proof of concept, from 25,000 euros in the closing block of the same page, plus 60,000 to 120,000 euros for the implementation and 500 to 2,000 euros per month for infrastructure (page). ANCUD states no amounts but cost drivers such as data sources, document volume and integrations (service page). Scope and assumptions get compared, not final prices.
Typical failure patterns
| Symptom | Common cause | Countermeasure |
|---|---|---|
| Answer cites an outdated version | Old versions and duplicates in the index, no sync with the source | Decide versions, date as metadata, sync including deletions |
| The existing passage is not found | Chunking breaks the context; pure vector search on identifiers | Chunk by structure, hybrid search, measure Recall@k |
| Answer looks supported and is wrong | Context from the wrong version: faithfulness high, context recall low | Read both metrics together, sample check by the business domain |
| System invents answers when knowledge is missing | No not-in-corpus tests | Treat not-in-corpus cases as blocking |
| Users see content from other areas | Permissions only in the prompt or only after the search | Filter before the search, permission tests including revoked permissions |
Build it yourself or commission it
| Route | Fits when | Prerequisite |
|---|---|---|
| Build it yourself | RAG is part of the product; a team has search, data and operations skills | Responsibility for operations, evaluation and updates |
| Configure a platform | Open WebUI is enough, a few knowledge areas | Running the platform, updates, care of the knowledge areas |
| Commission | Skills or capacity are missing; acceptance should be fixed by contract | Code, index and test set belong to you |
| Cloud service | Documents may leave your own environment | Data processing agreement, location and deletion are checked |
Every provider should answer these questions in writing before you commission RAG consulting:
- Who owns code, index and test set, also after the contract ends?
- Which reference cases and blocking criteria apply to acceptance?
- Where is access control enforced: before the search or only in the prompt?
- How do changes, deletions and revoked permissions in the source take effect in the index?
- Where do embedding model, index, language model and logs run?
- Who operates the system after handover, and which service hours apply?
Data protection and permissions
The index is a copy of the corpus in another form. OWASP lists weaknesses in vectors and embeddings as a risk of its own ("Vector and Embedding Weaknesses", LLM08:2025); this includes inadequate access control on embeddings. It recommends permission-aware vector stores, strict logical and access partitioning of datasets, and detailed immutable logs of retrieval activities (OWASP).
- Personal data in the corpus: Legal basis and record of processing activities also apply to the index; deletions must take effect there.
- Location: Check embedding model, index, language model and logs individually; for external APIs, clarify data processing agreements and third-country transfers.
- Professional secrecy and AI regulation: Local AI for professional secrecy holders, The EU AI Act for companies and RAG and the AI Act.
This is a technical assessment, not legal advice.
Status and methodology
- Status: 28 October 2026. Third-party documentation and price pages were retrieved that day and may change.
- Methodology: The phases follow our project practice (for comparison: Azure Architecture Center); the metrics follow the Ragas documentation. Market prices are publicly stated amounts, not a market study. The token calculation uses freely chosen example values.
- Not covered: Agentic RAG, GraphRAG, multimodal corpora.
How WZ-IT does it
We start with the test set and a defined set of documents. The RAG Proof of Value is that entry point: one knowledge area with an agreed data volume and formats, 40 to 60 reference questions including source and not-in-corpus tests, a baseline measurement, two optimisation cycles and a quality report with a recommendation for production. It costs from €9,900 net plus separately itemised third-party costs and is credited against the production knowledge assistant. Live connectors, organisation-wide SSO, complex permission inheritance and production operations are agreed separately.
For the next step you can choose the internal AI assistant, the AI platform with Open WebUI under its original branding and models in Germany at German providers, the AI Cube in your own network, or custom RAG and data integration. LLMOps covers tracing and evals in operation. Operations run on your infrastructure or hosted in Germany; a Section 203 German Criminal Code (StGB) confidentiality agreement is available on request (location Germany).
Further guides
- What is RAG? - fundamentals
- Measuring RAG quality - test set and acceptance
- RAG with permissions - access control
- Chunking for RAG
- Hybrid search and reranking
- RAG with Nextcloud, SharePoint, and DMS
- Connect Open WebUI to Nextcloud - permissions per file
Check knowledge search on your documents before you build? The RAG Proof of Value delivers reference questions, measurement and a reasoned recommendation for production. View the RAG Proof of Value · Internal AI assistant · Book a call
