Contextual Retrieval: RAG chunks with context and a local LLM

Improve the retrieval quality of a RAG system measurably? WZ-IT builds RAG systems on your own infrastructure, with a local model, hybrid search and traceable sources. The RAG Proof of Value measures a scoped knowledge collection against agreed reference questions, before and after two optimisation cycles. See the RAG Proof of Value · Internal AI assistants

A RAG system splits documents into passages, called chunks, and retrieves the matching ones for every question. Context gets lost along the way: a chunk reading "The company's revenue grew by 3% over the previous quarter" does not say which company or which quarter is meant. Contextual retrieval prepends a short, machine-generated context to every chunk and makes it findable again. Anthropic described the method and implemented it with Claude. This article explains it and shows how it works with a local model on vLLM or Ollama. As of September 2026.

Contents

• The problem: chunks without context
• How contextual retrieval works
• What Anthropic's measurements show
• The cost: one model call per chunk
• Prefix caching in vLLM as the local counterpart
• Ollama and llama.cpp
• Implementation with a local LLM
• Combining it with BM25 and reranking
• Limits and operational questions

The problem: chunks without context

A conventional RAG system processes the corpus in three steps (What is RAG?): split documents into chunks, turn each chunk into a vector with an embedding model, store the vectors in a database such as Qdrant or pgvector. Many systems add a lexical BM25 index, because vectors are poor at matching exact identifiers such as error codes, file numbers or part numbers.

Both indexes only see the chunk, though. What precedes it in the document, such as title, chapter, contracting party, scope or reference date, is missing from the chunk. Typical cases in business documents:

How to split documents so that such breaks occur less often is covered in Chunking for RAG. Contextual retrieval comes in afterwards: the split stays, and each chunk gets its context back.

How contextual retrieval works

Anthropic published the method in September 2024 (Anthropic, Introducing Contextual Retrieval). It consists of two parts: contextual embeddings and contextual BM25. In both cases, chunk-specific explanatory context is prepended to each chunk before it is embedded and written to the BM25 index.

A language model generates the context. It receives the whole document and the individual chunk and is asked to write a short context that situates the chunk within the document. Anthropic used this prompt with Claude 3 Haiku:

<document>
{{WHOLE_DOCUMENT}}
</document>
Here is the chunk we want to situate within the whole document
<chunk>
{{CHUNK_CONTENT}}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.

According to Anthropic, the resulting context is usually 50 to 100 tokens. In the source's example, "The company's revenue grew by 3% over the previous quarter." becomes a chunk prefixed with the information that it comes from an SEC filing on ACME Corp's performance in Q2 2023 and that the previous quarter's revenue was 314 million US dollars.

Related approaches are distinct from this: generic document summaries attached to chunks (according to Anthropic only very limited gains), hypothetical document embeddings and summary-based indexing (low performance in Anthropic's evaluation). The difference is that the context is generated for each chunk individually.

What Anthropic's measurements show

Anthropic measured across several knowledge domains (codebases, fiction, ArXiv papers, science papers). The metric is 1 minus recall@20, the share of relevant documents that fail to appear among the top 20 chunks. The averages refer to the best embedding configuration in the test, Gemini Text 004.

For reranking, Anthropic passed the top 150 from the initial retrieval to a reranker (Cohere) and handed the top 20 to the model. Further findings from the same source: embeddings plus BM25 beat embeddings alone, 20 chunks in the prompt worked better than 10 or 5, and the effects stack.

Two caveats belong to the assessment. The measurement comes from the vendor of the model used, and the datasets are in English. What transfers is the direction, not the number. Whether and how much your own corpus benefits can only be shown by measuring with your own reference questions, see Measuring RAG quality.

The cost: one model call per chunk

Contextual retrieval shifts compute into indexing. Every chunk triggers one model call, and every call contains the whole document. The search query itself triggers no additional model call.

Anthropic assumes 800-token chunks, 8,000-token documents, 50 tokens of instructions and 100 tokens of context per chunk, and arrives at a one-time cost of 1.02 US dollars per million document tokens with prompt caching. That figure is based on Claude 3 Haiku pricing at the time of publication.

With a local model there are no token prices; the cost shows up as GPU time. Using the same assumptions, per document (our own calculation):

Without a cache the model reads each document as many times as it has chunks. With a cache the document is processed once, and afterwards only the chunk and the instructions. Generating the contexts themselves (decoding) stays the same.

Prefix caching in vLLM as the local counterpart

vLLM provides automatic prefix caching (APC), the counterpart to the prompt caching of cloud APIs. APC stores the KV cache of processed requests. If a new request starts with the same prefix, vLLM skips computing the shared part (vLLM, Automatic Prefix Caching). The documentation names exactly the contextual retrieval pattern as a typical use case: many requests against the same long document.

Three rules for the pipeline follow from this:

1. The document goes at the start of the prompt. Anthropic's prompt is already built that way: document first, then the chunk, then the instructions. Everything that changes from chunk to chunk belongs after the document.
2. The prefix must be identical token for token. System prompt, chat template and document text stay the same for all chunks of one document. A chunk number or a timestamp before the document prevents any cache hit.
3. Group jobs by document. If the chunks of a document are processed together, its prefix is still in the cache. A queue mixed across the whole corpus risks the prefix being evicted before the next chunk arrives.

Whether the cache is effective is visible in the Prometheus metrics vllm:prefix_cache_queries and vllm:prefix_cache_hits (vLLM, Metrics). How vLLM works in general is explained in What is vLLM?.

Ollama and llama.cpp

Contextual retrieval can also be implemented with Ollama, but two points need attention.

The context window. By default Ollama uses a context window of 4,096 tokens (Ollama FAQ). A document of 8,000 tokens does not fit. The window is raised with the environment variable OLLAMA_CONTEXT_LENGTH or the API parameter num_ctx. Memory use scales with OLLAMA_NUM_PARALLEL times the context length; the default for parallel requests per model is 1.

Caching. Ollama does not document a configurable prefix cache like vLLM's. The llama.cpp server, whose technology Ollama originally built on, has the cache_prompt option (enabled by default): the prompt is compared with the previous request and only the differing remainder is processed again (llama.cpp server README). This helps with consecutive chunks of the same document, but it is tied to a slot and not comparable to vLLM's block-based cache.

For one-off indexing of a large corpus, vLLM is therefore usually the better choice; for small corpora and tests, Ollama is sufficient. The overall comparison of the inference servers is in vLLM vs. Ollama.

Implementation with a local LLM

The pipeline does not need a separate service. It is an additional step between chunking and embedding that runs against an OpenAI-compatible API.

Start a model with an OpenAI-compatible API in vLLM, here using a small instruct model under the Apache 2.0 licence as an example:

vllm serve Qwen/Qwen3-4B-Instruct-2507 --max-model-len 32768

The call per chunk. For non-English corpora, write the prompt in the language of the documents, so that the context uses the same technical terms as the later questions:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

def context(document: str, chunk: str) -> str:
    prompt = (
        f"<document>\n{document}\n</document>\n"
        "Here is the chunk we want to situate within the whole document\n"
        f"<chunk>\n{chunk}\n</chunk>\n"
        "Please give a short succinct context to situate this chunk within the "
        "overall document for the purposes of improving search retrieval of the "
        "chunk. Answer only with the succinct context and nothing else."
    )
    response = client.chat.completions.create(
        model="Qwen/Qwen3-4B-Instruct-2507",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=150,
        temperature=0,
    )
    return response.choices[0].message.content.strip()

On model choice: with Claude 3 Haiku, Anthropic deliberately used a small model. The task requires reading comprehension over long texts and short outputs, not deep reasoning. What matters is a context window that fits the longest documents and good command of the corpus language. Models with a thinking mode should run without it for this task, otherwise output length and runtime grow. Anthropic also recommends tailoring the prompt to the domain, for example with a glossary of terms that are only defined in other documents.

Which hardware suffices for which model is described in Sizing GPU & VRAM. Indexing typically runs on the same hardware as the chat service, for example overnight.

Combining it with BM25 and reranking

Contextual retrieval does not replace hybrid search; it improves both parts. The context often contains exactly the terms BM25 needs: company name, contract title, section number, product number. A chunk that previously only said "the company" can then also be found by name.

How hybrid search and reranking are built in detail is described in Hybrid search and reranking. Which embedding model suits German texts is compared in Embedding models for German.

For answer generation, Anthropic recommends passing the contextualized chunk to the model while marking what is context and what is the original text. The citation shown to the user always refers to the original text, because the generated context does not appear in the document.

Limits and operational questions

Small corpora do not need RAG. Anthropic names roughly 200,000 tokens (about 500 pages) as the threshold below which the entire corpus fits straight into the prompt. Locally, the threshold depends on the model's context window and GPU memory.

Long documents exceed the context window. A 300-page manual does not fit into every prompt. In that case, instead of the whole document, the surrounding chapter is passed, together with the title and table of contents. That deviates from the original method and should be measured.

Changes affect the whole document. A chunk's context depends on the entire document. If one section changes, the whole document is contextualized again. Indexing therefore has to version per document, not per chunk.

The context can be wrong. The model can situate a chunk incorrectly, for example assign it to the wrong version. Spot checks of the generated contexts belong in the acceptance test, and the user-facing view shows the original text.

Permissions also apply to the context. The context contains information from the whole document. It therefore inherits the document's access rights, just like the chunk (RAG with permissions). If several tenants share one vLLM server, cache_salt separates the caches.

Index size and chunk length grow. Every chunk becomes 50 to 100 tokens longer. That affects storage in the index and the number of tokens that go into the prompt with every answer.

What this means for your project

Contextual retrieval is an indexing step, not a change of product. It can be added to an existing pipeline, needs no cloud API and runs with a local model on the same hardware as the assistant, on an AI Cube in your own network or on a managed GPU server from WZ-IT. The cost lies in GPU time during indexing, and prefix caching in vLLM reduces it considerably.

Whether the step pays off is decided by a measurement: the same reference questions before and after contextualization, assessed with recall@k and answer quality. The RAG Proof of Value provides exactly this comparison of baseline and optimisation, with support, consulting and implementation by WZ-IT.

The basics are in What is RAG?, the step before in Chunking for RAG, the step after in Hybrid search and reranking. How to measure the effect reliably is shown in Measuring RAG quality, and which vector database carries hybrid search is compared in Qdrant vs. pgvector.

Sources

• Anthropic, Introducing Contextual Retrieval (19 Sep 2024)
• Anthropic Cookbook, Contextual Embeddings
• vLLM, Automatic Prefix Caching
• vLLM, Design: Automatic Prefix Caching
• vLLM, Engine arguments
• vLLM, Metrics
• vLLM, CacheConfig in the source code
• Ollama FAQ
• llama.cpp server README
• Hugging Face, Qwen3-4B-Instruct-2507
• Hugging Face, BGE-M3
• Hugging Face, bge-reranker-v2-m3
• Hugging Face, Qwen3-Reranker-0.6B