Hybrid search and reranking for RAG: BM25, vectors, cross-encoders

Search that finds file numbers and paraphrased questions alike? WZ-IT builds RAG systems with hybrid search, reranking and permissions on your own infrastructure and measures retrieval quality with your own questions. The RAG Sprint makes one priority knowledge source usable with defined reference questions, a baseline measurement and citations. Explore the RAG Sprint · Internal AI assistants

A RAG system answers only as well as its search finds the right passages. Pure vector search finds paraphrased questions but regularly misses exact identifiers such as file numbers, part numbers or legal sections. The answer is a two-stage search: hybrid search collects candidates from lexical and semantic retrieval, and a reranker then orders them by actual relevance. This article explains the building blocks, shows the implementation in Qdrant, OpenSearch and PostgreSQL and compares rerankers for self-hosting. As of September 2026.

Table of contents

• Why vector search alone is not enough
• The two retrieval paths: BM25 and dense retrieval
• Merging results: Reciprocal Rank Fusion
• BGE-M3: dense and sparse from one model
• Reranking with a cross-encoder
• Rerankers compared
• Implementation in Qdrant, OpenSearch and PostgreSQL
• German text: compounds and identifiers
• Latency and GPU requirements

Why vector search alone is not enough

An embedding model turns the question and each passage into vectors and measures the distance between them (What is RAG?). This works well when question and document say the same thing in different words: a question about "upkeep" finds the passage about "maintenance".

The weakness shows with strings whose meaning lies in their exact spelling. "Case 4 K 1234/24" and "Case 4 K 1243/24" sit close together in vector space but refer to two different proceedings. The same applies to part numbers, standard designations, legal sections, version numbers and error codes. Anthropic describes exactly this case: for a question about "Error code TS-999", an embedding model finds general error documentation, while BM25 matches the specific string (Anthropic, Contextual Retrieval).

Conversely, pure keyword search fails on paraphrases, synonyms and questions in everyday language. Neither path covers both cases. In company knowledge bases both occur constantly, often in the same question: "What does section 35 of the German Building Code say about outbuildings in undesignated outer areas?"

The two retrieval paths: BM25 and dense retrieval

BM25 scores a passage on three factors: how often a search term occurs in it, how rare the term is across the whole corpus (inverse document frequency, IDF) and how long the passage is. Qdrant lists k1 = 1.2 and b = 0.75 as default parameters (Qdrant, Full-Text Search). Rare terms such as a file number therefore carry a lot of weight, frequent words such as "application" very little.

Dense retrieval uses an embedding model that maps question and passage into the same vector space. Which model suits German text is compared in the article on embedding models for German.

Merging results: Reciprocal Rank Fusion

Both retrieval paths return a ranked list with scores that are not comparable. A BM25 score of 14.2 and a cosine similarity of 0.81 cannot be meaningfully added. There are two ways to solve this.

Rank-based: Reciprocal Rank Fusion (RRF). RRF ignores the scores and uses only the rank positions. Each document receives 1 / (k + rank) from every list, and the values are summed:

RRF(d) = Σ 1 / (k + rank_r(d))    over all result lists r

The method comes from Cormack, Clarke and Büttcher. In their SIGIR 2009 paper, k = 60 was fixed during a pilot investigation and not changed afterwards; the constant dampens the impact of outlier rankings (Cormack et al., Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods). A document ranked high in both lists wins; one that appears in only one list stays in play.

Score-based: normalisation. The scores of each list are brought onto a common scale and combined with weights. OpenSearch offers min_max, l2 and z_score normalisation for this (OpenSearch, Normalization processor), Qdrant offers Distribution-Based Score Fusion (DBSF), which uses the mean and standard deviation of each list (Qdrant, Hybrid Queries).

RRF is the robust default because nothing needs to be calibrated. The defaults differ between systems, however:

Sources: Qdrant, Hybrid Queries, OpenSearch, Score ranker processor. Anyone comparing results across systems or moving from one to another should therefore set k explicitly instead of relying on the default.

BGE-M3: dense and sparse from one model

BGE-M3 by BAAI produces three representations of a text in one pass: a dense vector with 1,024 dimensions, learned term weights as a sparse vector, and ColBERT-style multi-vectors. The model handles up to 8,192 tokens, supports more than 100 languages and is licensed under MIT.

The advantage: one model serves both retrieval paths and the pipeline stays lean. The model card explicitly recommends hybrid retrieval followed by a reranker. The difference from BM25: the sparse weights are learned and based on the model's tokenizer, not on whole words. Whether they match exact identifiers in your own corpus as reliably as a classic BM25 index with suitable analysis only a test can show. For corpora with many numbers and file references, an additional BM25 index is the safe choice.

Reranking with a cross-encoder

Embedding models are bi-encoders: question and passage are turned into vectors separately and compared only afterwards. This is fast, because the passage vectors are created once at indexing time. It is also imprecise, because the model never sees question and text together.

A cross-encoder reads question and passage as a pair and outputs a relevance score directly (BAAI, bge-reranker-v2-m3). It recognises whether a passage actually answers the question or merely uses the same terms. The price: every pair needs its own model pass, nothing can be precomputed.

This leads to the usual two-stage architecture:

1. Collect candidates. Hybrid search returns a few dozen hits per path, merged with RRF.
2. Reorder. The reranker scores these candidates and passes the best to the language model.
3. Set a threshold. If even the best candidate scores below a defined value, the system does not answer but reports a gap.

For scale: in its tests, Anthropic retrieved 150 candidates, had them reranked and passed the top 20 to the model. Combining contextual embeddings, BM25 and reranking reduced the top-20 retrieval failure rate from 5.7% to 1.9%, a reduction of 67%; without reranking it was 49% (Anthropic, Contextual Retrieval). How contextual embeddings are produced with a local model is explained in Contextual retrieval.

The limit matters: a reranker only reorders what the first stage found. If the right passage is missing from the candidates, it stays missing. That is why the recall of the hybrid search is checked first, then the ordering.

The threshold from step 3 is also the simplest safeguard against invented answers. bge-reranker-v2-m3 can map its scores to 0 to 1 via a sigmoid (option normalize=True in FlagEmbedding), which makes a fixed threshold manageable. The right value does not come from a data sheet but from an evaluation with questions to which the corpus deliberately contains no answer.

Rerankers compared

Self-hostable rerankers with multilingual support, as of September 2026:

The Qwen3 rerankers work differently from classic cross-encoders: they are language models that compute the probability of the answers "yes" and "no" for each pair. According to the model card, a task description (instruction) typically improves results by 1 to 5%.

The vendor figures from the Qwen3-Reranker-8B model card for the multilingual retrieval set MMTEB-R (top 100 candidates from Qwen3-Embedding-0.6B):

These are the vendor's figures on public benchmarks, not a statement about German administrative or domain-specific text. They indicate a direction but do not replace a test with your own questions. How such a test is set up is described in RAG evaluation.

Implementation in Qdrant, OpenSearch and PostgreSQL

Qdrant

Qdrant stores dense and sparse vectors in the same point. The Query API runs several subqueries (prefetch) and fuses them in the main query:

POST /collections/knowledge/points/query
{
  "prefetch": [
    { "query": { "indices": [1042, 88731], "values": [0.61, 0.44] },
      "using": "sparse", "limit": 50,
      "filter": { "must": [{ "key": "acl", "match": { "any": ["grp-legal"] } }] } },
    { "query": [0.012, -0.087, 0.143],
      "using": "dense", "limit": 50,
      "filter": { "must": [{ "key": "acl", "match": { "any": ["grp-legal"] } }] } }
  ],
  "query": { "rrf": { "k": 60 } },
  "limit": 30
}

The sparse part comes either from the BGE-M3 term weights or from BM25. Since version 1.15.2, Qdrant can convert text into BM25 sparse vectors itself (model qdrant/bm25); Qdrant computes the IDF component on the server, which requires the idf modifier on the sparse vector (Qdrant, Sparse Retrieval). The default BM25 text analysis in Qdrant is English; language, stemming and stop words are configurable (Qdrant, Full-Text Search). The application calls the reranker separately afterwards.

OpenSearch

OpenSearch comes with a full BM25 index and language analysis and combines it with a k-NN search via the hybrid query. Fusion is handled by a search pipeline:

PUT /_search/pipeline/rrf-pipeline
{
  "phase_results_processors": [
    { "score-ranker-processor": { "combination": { "technique": "rrf", "rank_constant": 60 } } }
  ]
}

GET /knowledge/_search?search_pipeline=rrf-pipeline
{
  "query": {
    "hybrid": {
      "queries": [
        { "match": { "text": "notice period framework agreement 2024-117" } },
        { "knn": { "embedding": { "vector": [0.012, -0.087, 0.143], "k": 50 } } }
      ]
    }
  }
}

A hybrid query combines up to five subqueries (OpenSearch, Hybrid query). Reranking is possible directly in the pipeline: since version 2.12, the rerank processor calls a cross-encoder model registered in OpenSearch (OpenSearch, Rerank processor).

PostgreSQL with pgvector

If you already run PostgreSQL, you can combine full-text search (tsvector) and pgvector in one query. The pgvector documentation explicitly names RRF or a cross-encoder for merging results. RRF can be written directly in SQL:

-- Column: tsv tsvector GENERATED ALWAYS AS (to_tsvector('german', text)) STORED, with a GIN index
WITH semantic AS (
  SELECT id, RANK() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks ORDER BY embedding <=> $1 LIMIT 50
),
lexical AS (
  SELECT id, RANK() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS rank
  FROM chunks, websearch_to_tsquery('german', $2) q
  WHERE tsv @@ q
  ORDER BY ts_rank_cd(tsv, q) DESC LIMIT 50
)
SELECT COALESCE(s.id, l.id) AS id,
       COALESCE(1.0 / (60 + s.rank), 0) + COALESCE(1.0 / (60 + l.rank), 0) AS rrf
FROM semantic s
FULL OUTER JOIN lexical l ON s.id = l.id
ORDER BY rrf DESC LIMIT 30;

One limitation: ts_rank and ts_rank_cd are not BM25. According to the PostgreSQL documentation they do not use any global information, so no inverse document frequency. For RRF, which only uses ranks, that is often sufficient. Real BM25 is provided by extensions such as pg_textsearch (PostgreSQL licence) or ParadeDB's pg_search (AGPL-3.0). pgvector itself supports sparse vectors via sparsevec with up to 1,000 non-zero elements, for example for BGE-M3 term weights.

The fundamental choice between a dedicated vector database and a Postgres extension is covered in Qdrant vs. pgvector. Regardless of the system, the permission filter belongs in every subquery, not after the fusion (RAG with permissions).

German text: compounds and identifiers

For German corpora, the text analysis of the lexical part determines quality. Three points come up regularly.

Compounds. "Instandhaltungsrahmenvertrag" (maintenance framework agreement) is a single term for BM25. A search for "Rahmenvertrag Instandhaltung" does not match it. OpenSearch offers the dictionary_decompounder and hyphenation_decompounder filters, which split compound words into their parts using a word list or hyphenation patterns and are explicitly intended for languages such as German (OpenSearch, Dictionary decompounder). In PostgreSQL, Ispell dictionaries can split compounds (PostgreSQL, Dictionaries); the bundled Snowball stemmer for german does not.

Identifiers. Standard tokenizers split "4 K 1234/24" or "A-4711-03" at slashes and hyphens into individual numbers. The search then matches every document in which "1234" and "24" appear anywhere. A proven approach is to detect identifiers by pattern during indexing and additionally store them unchanged in a dedicated field that is searched exactly or used as a filter.

Stemming and stop words. A German analyser reduces "Anträge", "Antrags" and "Antrag" to one stem and removes filler words. Without this setting, BM25 loses a lot of recall on German text. How documents are split before indexing so that passages keep their identifiers and headings is described in Chunking strategies for RAG.

Latency and GPU requirements

Hybrid search itself is cheap: both paths run in parallel on indexes, and the fusion is a sum over rank positions. The compute load arises in two places: embedding the question and reranking.

The reranker scores each candidate pair individually. Compute time therefore grows linearly with the number of candidates and with their length. The levers are the number of candidates after fusion, the maximum passage length and the model size.

The values follow from the parameter count times two bytes; activations and batching come on top. Two common servers are available: Hugging Face Text Embeddings Inference supports XLM-RoBERTa-based rerankers such as bge-reranker-v2-m3 and also runs on CPU. vLLM provides a rerank interface at /rerank, /v1/rerank and /v2/rerank, compatible with the Jina and Cohere rerank APIs (vLLM, Online Serving); the Qwen3 rerankers need an architecture override via hf_overrides at startup.

On a GPU server, the language model, the embedding model and the reranker share GPU memory. How much of it the language model itself needs is worked out in GPU and VRAM sizing for LLMs.

What this means for your project

Hybrid search is the sensible default for company knowledge bases as soon as identifiers, numbers or technical terms matter, which is almost always the case. Reranking pays off when the right passage is found but ends up too far down. Both can only be judged with your own reference questions: first measure whether the right passage is among the candidates, then whether it ranks at the top.

We build on what is already in operation: PostgreSQL with pgvector where Postgres is in place, Qdrant or OpenSearch where vector or full-text search is central. Models run locally on the AI Cube or on WZ-IT managed GPU servers with NVIDIA RTX PRO 4000 Blackwell (24 GB) or RTX PRO 6000 Blackwell (96 GB). Support, consulting and implementation by WZ-IT.

How RAG works in principle is explained in What is RAG?. The stages before the search are covered in Chunking strategies for RAG and Contextual retrieval, measuring the results in RAG evaluation. Which embedding models suit German text is compared in embedding models for German, and how permissions take effect before the search is shown in RAG with permissions.