RAG evaluation: test set, metrics and acceptance

A knowledge system with proven answer quality rather than a demo impression? The RAG Proof of Value assesses a scoped corpus against agreed reference questions with a baseline measurement, two optimization cycles and a quality report. For applications in production, LLMOps takes over the regular evaluation runs. See the RAG Proof of Value · Internal AI assistants

A RAG system almost always looks good in a demo: the prepared questions hit, the answers are fluent and cite sources. Whether it holds up in daily use only shows when it is measured with questions nobody picked for the demo. This article describes how a reliable test set is built, which metrics measure retrieval and which measure the answer, where automated judging by a language model helps and where it does not, and how that turns into acceptance criteria and regression tests. It is independent of any particular framework; Ragas and Langfuse serve as examples because both are open source and can be self-hosted. As of September 2026.

Contents

• Two levels: measure retrieval and answer separately
• The test set: reference questions from the business domain
• Retrieval metrics: Recall@k, Precision@k, MRR and nDCG
• Answer metrics in Ragas
• LLM-as-a-judge with a local model
• Defining acceptance criteria
• Regression tests after changes to model, index and prompt
• Evaluation in production
• Typical measurement mistakes

Two levels: measure retrieval and answer separately

A RAG system consists of two steps that can fail independently: retrieval fetches text passages from the index, and the language model writes an answer from them. How the two work together is explained in What is RAG?. For measurement this means a single overall score hides where the error lies.

The order of troubleshooting follows from this. If the right source is missing from the context, no model can produce a correct, supported answer, and any work on the prompt is wasted. Only once retrieval demonstrably delivers is it worth optimizing the answer side. The levers on the retrieval side are covered in the articles on chunking and on hybrid search and reranking.

The test set: reference questions from the business domain

Every metric is only as good as the questions it is measured on. The test set (often called a gold standard or golden dataset) is therefore the most important part of evaluation, and also the most neglected.

Where the questions come from. The best sources are questions people already ask today: tickets, emails to the specialist department, posts in the internal forum, search queries in the intranet. They use the users' terms rather than the documents' terms, and retrieval systems fail exactly at that gap. Questions a developer writes while reading a document adopt its wording and are therefore too easy. For German-language corpora the questions should be in German, phrased the way users actually write them.

What each entry contains.

Which question types belong in it. A test set made only of easy questions only proves that the system can answer easy questions.

Permission tests are not a quality question in the narrow sense, but they belong in the same set because they must run with every change. Why the filter has to sit before retrieval is described in RAG with permissions.

Size. There is no universal minimum. For a scoped knowledge base, 40 to 60 carefully selected questions are a workable start if every question type is represented. More important than the total is that each category has several cases, so a single outlier does not dominate the picture.

Synthetic questions. Ragas can generate test sets from the documents, including in languages other than English (Non-English Testset Generation). This is useful for breadth but has a built-in weakness: the questions are derived from the text the system is supposed to find and therefore match its wording. Synthetic questions supplement a core of real questions, they do not replace it, and every question adopted should have been read by a person.

Retrieval metrics: Recall@k, Precision@k, MRR and nDCG

Retrieval can be measured without a language model if the expected sources are known for each question. These metrics come from classic information retrieval (Manning, Raghavan, Schütze: Evaluation of ranked retrieval results) and are deterministic: the same index returns the same value for the same question.

For RAG, Recall@k is usually the most important number, because the model can only see what ends up in the context. k is the number of passages actually passed to the model, not the number of candidates before reranking. With a reranker in place, it is best to measure both stages: recall of the candidate list (does retrieval find the evidence at all?) and recall after reranking (does it land among the passages handed over?).

MRR suits questions with exactly one correct source, such as exact identifiers. nDCG pays off when domain experts grade sources, for example "answers the question" versus "provides background".

One prerequisite is often overlooked: expected sources need stable IDs. If a chunking change splits documents differently, chunk IDs change and the reference no longer matches. It is more robust to record sources at document and section level and to count a hit as correct when it comes from the expected section.

Answer metrics in Ragas

Ragas is an open-source library under the Apache 2.0 license, based on the paper Ragas: Automated Evaluation of Retrieval Augmented Generation. The current version on PyPI is 0.4.3 (as of September 2026). The documentation lists the RAG metrics under Available Metrics. The most common ones:

All scores range from 0 to 1. Three points are decisive for interpretation:

Faithfulness measures support, not correctness. A score of 1.0 means every claim can be inferred from the context. If the context comes from an outdated version, the answer is supported and still wrong. Faithfulness should therefore always be read together with context recall. As an alternative to an LLM verdict, Ragas offers the FaithfulnesswithHHEM variant, which uses Vectara's classifier model HHEM-2.1-Open.

Answer relevancy does not measure correctness. The documentation says so explicitly: the metric assesses how well the answer matches the intent of the question without evaluating factual accuracy. It penalizes evasive and padded answers but also rewards a fluent false statement. The current API lists it as AnswerRelevancy under ragas.metrics.collections, the older API, marked as deprecated, as ResponseRelevancy.

Context precision and context recall exist with and without an LLM. The ID-based variants compare retrieved_context_ids with reference_context_ids and are essentially precision and recall from the previous section. If the test set maintains expected sources, these variants can be computed deterministically and without model cost, leaving the LLM variants for cases where no IDs are available.

LLM-as-a-judge with a local model

Faithfulness, answer relevancy and the LLM variants of the context metrics let a language model make the call. That scales but has known weaknesses. The study Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Zheng et al. names position, verbosity and self-enhancement bias as well as limited reasoning ability, and found over 80 percent agreement between GPT-4 as a judge and human preferences, roughly the level of agreement between two humans. Over 80 percent also means that a relevant share of verdicts differ.

For operation on your own infrastructure this leads to five rules:

1. The judging model is interchangeable. The examples in the Ragas documentation use OpenAI models, but the judge can be a locally hosted model behind an OpenAI-compatible endpoint, for example with vLLM or Ollama. Test questions and document excerpts then stay in your own environment during evaluation as well.
2. Separate judge and generator. When a model grades its own answers, self-enhancement bias applies. A different, ideally stronger model as judge reduces it.
3. Adapt prompts to the target language. Ragas adapts metrics to a target language via adapt(). By default only the few-shot examples are translated and the instructions stay in English; with adapt_instruction=True they are translated too (Adapting Metrics to Target Language). The adapted prompts should be saved and versioned so measurements remain comparable.
4. Check structured output. For LLM-as-a-judge through a custom gateway, Langfuse requires the endpoint to support tool calling in the OpenAI format (LLM Connections). Small local models fail here more often than at the actual assessment.
5. Calibrate the judge. A sample of verdicts is cross-checked by domain experts before automated scores serve as an acceptance criterion. If the judge deviates systematically in a category, the human verdict counts there.

Automated judging is a tool to make many runs comparable. It does not replace acceptance by people who know the corpus.

Defining acceptance criteria

An acceptance needs criteria agreed before measurement. A proven approach separates blocking cases, which must pass without exception, from averages that must reach a threshold.

Specific thresholds depend on the risk of the use case and the variety of questions. An assistant for internal policies tolerates different thresholds than a system whose answers go to customers or into administrative decisions. Universal numbers such as "faithfulness above 0.9" say little without reference to your own test set.

A baseline works well in practice: the first measurement with a simple configuration sets the starting value, and every optimization is measured against it. This shows whether hybrid search, a reranker or different chunking actually help. Results should be broken down by question type; a good overall score can hide a category in which the system reliably fails.

For systems that may fall under the EU AI Act, documented tests and logs are required anyway. For context, see RAG and the AI Act.

Regression tests after changes to model, index and prompt

A RAG system changes continuously, and almost every change can shift quality. After acceptance, the test set is therefore not archived but becomes the regression test.

The most common trigger of unnoticed quality loss is a switch of the embedding model, because it affects the entire index. Which models suit German-language text is compared in the article on embedding models for German.

Implementation with Langfuse. Langfuse represents a test set as a dataset: each item has input, optionally expected_output and metadata. Every change to the items creates a new dataset version, so an experiment can later be repeated against exactly that state. With run_experiment() from the SDK, your own application runs against the dataset; item-level evaluators write scores to the traces, run-level evaluators compute aggregate values (Experiments via SDK). The runs appear as dataset runs and can be compared in the UI.

For the pipeline, Langfuse describes two release policies (Experiments in CI/CD): either every required case must reach an absolute quality threshold, or no case that passed in the approved baseline may regress. If the rule is violated, the build fails. Ragas metrics can be plugged in as evaluators or written to traces as scores (Cookbook: Evaluation of RAG with Ragas). How to run Langfuse yourself is covered in What is Langfuse? and in the article on self-hosted Langfuse.

Evaluation in production

The test set checks known questions. In production, unknown ones are added, and measurement has to capture them.

• Online evaluation. Langfuse can apply LLM-as-a-judge evaluators to individual observations in live operation, such as the retrieval step or the answer, as well as to experiments (LLM-as-a-Judge). This provides trends, not individual verdicts of acceptance quality.
• Human review. Annotation queues distribute selected traces to domain experts for assessment. Useful inputs are random samples plus all cases with negative user feedback or a low judge score.
• Feeding back into the test set. Every real question that went wrong is added to the dataset as a new item once it has been clarified. The test set thus grows along the actual weak spots.
• Reviewing unanswered topics. Questions the system correctly answers with "not in the corpus" point to missing documents. That list is an editorial task, not a system error.

Datasets, experiments, LLM-as-a-judge and annotation queues are included in the self-hosted open-source version of Langfuse (Self-Hosting Pricing); LLM-as-a-judge, annotation queues, prompt experiments and the playground were released under the MIT license on 4 June 2025 (Langfuse blog).

Typical measurement mistakes

• Assessing only the answer. Without retrieval metrics it remains unclear whether an error lies in retrieval or in the model.
• Test set from document text rather than user questions. Scores rise, real-world fitness does not.
• No unanswerable questions. Then it stays invisible whether the system states knowledge gaps or invents answers.
• Judge scores without calibration. An automated score no person has ever cross-checked is an estimate.
• Averages instead of categories. An average of 0.85 can mean every question about exact identifiers fails.
• Test set without a version. If questions and system change at the same time, two measurements are not comparable.
• Reference IDs not checked after re-indexing. After a chunking change the expected sources point nowhere, and recall appears to drop to zero.

What this means for your project

Evaluation starts before the first line of code: with real questions from the business domain, expected sources and an agreement on which cases are blocking. Building the test set early means every technical decision can be measured against a baseline rather than optimized by impression, and after acceptance there is a regression test for every later change.

The basics are in What is RAG?. How to make retrieval more robust is shown in hybrid search and reranking, contextual retrieval and chunking strategies. Why permission tests belong in every test set is explained in RAG with permissions, and where Langfuse sits in the stack is shown in The open-source LLM stack.

Sources

• Ragas, Available Metrics
• Ragas, Faithfulness
• Ragas, Answer Relevancy
• Ragas, Context Precision
• Ragas, Context Recall
• Ragas, Noise Sensitivity
• Ragas, Adapting Metrics to Target Language
• Ragas, Non-English Testset Generation
• Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217)
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685)
• Manning, Raghavan, Schütze: Evaluation of ranked retrieval results
• Langfuse, Datasets
• Langfuse, Experiments via SDK
• Langfuse, Experiments in CI/CD
• Langfuse, LLM-as-a-Judge
• Langfuse, LLM Connections
• Langfuse, Annotation Queues
• Langfuse, Cookbook: Evaluation of RAG with Ragas
• Langfuse, Self-Hosting Pricing
• Langfuse, Open-Sourcing Langfuse Product Features (4 June 2025)