RAG retrieval methods: dense vs BM25 vs SPLADE vs ColBERT

The retriever is the part of a RAG system that decides what the model gets to read. If it hands over the wrong chunks, no prompt and no model upgrade will save the answer. This post compares the retrieval methods you can choose from (dense vectors, BM25, SPLADE, ColBERT and hybrid combinations), what each one costs, and when you should skip retrieval and use a long context window instead.

It is the “which retriever” half of a pair. The “how do I build the whole pipeline” half, with chunking, contextual retrieval, reranking and citation checks in code, is in contextual retrieval: how to build a RAG pipeline that works. If the retrieval layer feeds an agent rather than a single answer, our guide to production Model Context Protocol deployments covers that side.

Which RAG retrieval method should you use?

Start with hybrid search: a dense embedding index and a BM25 index queried in parallel, merged with reciprocal rank fusion, then re-scored by a cross-encoder reranker. It handles paraphrased questions and exact identifiers, runs on common infrastructure, and is easy to measure. Move to SPLADE or ColBERT only when your evaluation set shows hybrid is falling short.

That recommendation rests on how each method fails, so here is the short version before the details.

Approach	Good at	Weak at	Storage	Effort to run
Dense bi-encoder	Paraphrase, synonyms, conceptual similarity	Exact strings, rare identifiers	One vector per chunk	Low
BM25	Exact terms, codes, names	Synonyms and paraphrase	Inverted index, small	Low
SPLADE v3	Exact terms plus learned expansion	Needs a model at index and query time	Sparse inverted index	Medium
ColBERT	Token-level matching on fine wording	Storage and tooling	One vector per token	High
Hybrid (dense + sparse + reranker)	Covers both failure types	Extra latency from the reranker	Two indexes	Medium

A hybrid pipeline routes queries through more than one retriever and merges the results before generation.

Why plain vector search falls short

The original RAG recipe was: chunk the documents, embed each chunk, store the vectors, and pull the nearest neighbours at query time. It demos well. In production it runs into three problems.

1. Exact strings get blurred. An embedding compresses a chunk into one fixed-size vector. Queries that hinge on precise tokens (“Policy 4.2.1”, ConnectionPoolTimeout, an ICD code) land near chunks that are about the same topic but contain the wrong clause.
2. Chunks lose their context. A chunk that says “the exception to this rule is…” means little without the chunk that defined the rule.
3. Questions and documents are written differently. “How do I fix the timeout error?” and “Connection timeout occurs when the socket pool is exhausted” are related, but they do not always sit close together in embedding space.

The first problem is what sparse retrieval fixes. The second is what contextual retrieval fixes, and it is covered in the companion build guide. The third is partly fixed by both, and partly by query rewriting.

Dense retrieval: the baseline

Dense bi-encoders such as OpenAI’s text-embedding-3-large, Cohere’s embed models, or open models like bge-m3 project queries and documents into one vector space. Retrieval is a nearest-neighbour search.

from openai import OpenAI
import numpy as np

client = OpenAI()

def embed(texts: list[str], model: str = "text-embedding-3-large") -> np.ndarray:
    response = client.embeddings.create(input=texts, model=model)
    return np.array([e.embedding for e in response.data])

query_vec = embed(["How do I configure retry logic?"])
doc_vecs = embed(chunks)  # pre-computed and stored in practice

# Cosine similarity (OpenAI embeddings are normalised, so a dot product works)
similarities = np.dot(doc_vecs, query_vec.T).flatten()
top_k_indices = np.argsort(similarities)[-10:][::-1]

Dense retrieval is the right first index for most corpora. It is forgiving about wording, and every vector database supports it. Its blind spot is the exact-match query, and that blind spot is large in technical, legal and medical text.

BM25: still the best exact-match retriever

BM25 is the term-weighting algorithm behind decades of search engines. It scores documents on the query terms they contain, weighted by how rare each term is. It needs no GPU and no model.

from rank_bm25 import BM25Okapi
import numpy as np

tokenized_corpus = [doc.split() for doc in chunks]
bm25 = BM25Okapi(tokenized_corpus)

query = "ConnectionPoolTimeout retry backoff"
scores = bm25.get_scores(query.split())
top_k = np.argsort(scores)[-10:][::-1]

On queries full of function names, error codes or configuration keys, BM25 tends to beat dense retrieval for an obvious reason: those strings are rare, so BM25 weights them heavily, while an embedding treats them as one small signal among many. It cannot see that “authentication failure” and “login error” mean the same thing. That gap is why you pair it with a dense index instead of choosing one. In production, use the BM25 built into your search engine (Elasticsearch, OpenSearch, Vespa) or vector database rather than rank_bm25, which keeps everything in memory.

SPLADE: sparse retrieval with learned expansion

SPLADE uses a transformer to produce a sparse vector over the vocabulary. Each dimension is a token, and the model can switch on related tokens that do not appear in the text. A passage about “authentication” can get weight on “login”, so you keep the exact-match behaviour of an inverted index and gain some of the synonym handling of dense models.

The current public checkpoint is naver/splade-v3, and its model card shows usage through Sentence Transformers:

from sentence_transformers import SparseEncoder

model = SparseEncoder("naver/splade-v3")

query_emb = model.encode_query(["how do I reset a login token"])
doc_embs = model.encode_document(chunks)

scores = model.similarity(query_emb, doc_embs)  # dot product

Two practical points. First, SPLADE vectors live happily in an inverted index (Elasticsearch, Lucene, Vespa, or the sparse-vector support in Qdrant and others), so teams that already run search do not need a new store. Second, read the license. The splade-v3 model card lists CC-BY-NC-SA-4.0, which rules out commercial use of those weights. That surprises people, and it is a good reason to start with BM25 and treat SPLADE as an upgrade you evaluate with legal sign-off.

ColBERT: token-level late interaction

ColBERT keeps one embedding per token for both query and document. A document’s score is the sum, over query tokens, of the best match among the document’s tokens (the MaxSim operation).

# Pseudocode for ColBERT MaxSim scoring
def colbert_score(query_embeddings, doc_embeddings):
    """
    query_embeddings: (num_query_tokens, dim)
    doc_embeddings: (num_doc_tokens, dim)
    """
    similarity_matrix = query_embeddings @ doc_embeddings.T  # (Q, D)
    max_similarities = similarity_matrix.max(dim=1).values   # (Q,)
    return max_similarities.sum()

Because individual token representations survive, ColBERT can separate “Python list comprehension” from “Python list methods” in cases where a single vector blurs them.

The cost is storage, and it scales with document length. The ColBERTv2 paper puts numbers on it: a 128-dimension token vector at 16-bit precision is 256 bytes, and ColBERTv2’s residual compression stores each vector as a 4-byte centroid ID plus a 1-bit or 2-bit residual, for 20 or 36 bytes per vector. Applying that to a 200-token chunk gives a feel for the trade-off. The per-token figures come from the paper; the multiplication is ours, and it ignores index overhead such as centroid tables.

Representation	Vectors per 200-token chunk	Bytes per vector	Approx. per 1,000 chunks
Dense, 1,536 dims, float32	1	6,144	6.1 MB
ColBERT, 128 dims, 16-bit	200	256	51 MB
ColBERTv2, 2-bit residuals	200	36	7.2 MB
ColBERTv2, 1-bit residuals	200	20	4 MB

So compressed ColBERT is not the storage monster it once was, but you need the PLAID-style engine or a vector database with multi-vector support to serve it, and fewer teams know how to operate that. My view: ColBERT earns its place when your evaluation shows hybrid missing fine-grained distinctions and the cost of a wrong answer is high (contracts, clinical guidance, financial filings). For documentation search and support bots, hybrid is usually enough.

Hybrid search: the default to start with

Hybrid search runs more than one retriever in parallel and merges their candidate lists. A two-stage layout is the common one:

1. Parallel retrieval: a dense index for meaning and BM25 or SPLADE for exact terms, each returning 20 to 50 candidates.
2. Reranking: a cross-encoder scores each query and candidate pair together and keeps the best handful.

from typing import NamedTuple

class RetrievalResult(NamedTuple):
    chunk_id: str
    score: float
    source: str

def hybrid_retrieve(
    query: str,
    dense_index,
    sparse_index,
    reranker,
    top_k: int = 10,
    candidates_per_source: int = 30,
) -> list[RetrievalResult]:
    # Stage 1: parallel retrieval
    dense_results = dense_index.search(query, k=candidates_per_source)
    sparse_results = sparse_index.search(query, k=candidates_per_source)

    # Merge ranked lists with reciprocal rank fusion (defined below)
    fused = reciprocal_rank_fusion([
        [r.chunk_id for r in dense_results],
        [r.chunk_id for r in sparse_results],
    ])
    candidate_ids = [chunk_id for chunk_id, _ in fused[: candidates_per_source]]

    # Stage 2: cross-encoder reranking
    return reranker.rerank(
        query=query,
        documents=[get_chunk_text(c) for c in candidate_ids],
        top_k=top_k,
    )

Reciprocal rank fusion

Dense cosine scores, BM25 scores and SPLADE dot products sit on different scales, so adding them directly is a calibration problem. Reciprocal rank fusion (RRF) avoids it by using ranks instead of scores. Each document gets 1 / (k + rank) from every list it appears in, and the sums are sorted. Cormack, Clarke and Buettcher introduced it at SIGIR 2009 with k = 60, a value they fixed in a pilot study and did not tune afterwards. Most engines still default to 60.

def reciprocal_rank_fusion(
    result_lists: list[list[str]],
    k: int = 60,
) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for result_list in result_lists:
        for rank, doc_id in enumerate(result_list):
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

The reranker

The reranker is where most of the precision comes from. A bi-encoder embeds query and document separately; a cross-encoder reads them together, so it can tell that a chunk mentions the right product but the wrong version. Hosted options include Cohere Rerank (the current API reference uses rerank-v4.0-pro in its examples) and open models such as the bge-reranker family that you can run yourself.

The best public evidence for stacking these stages is Anthropic’s contextual retrieval write-up. On its test sets, contextual embeddings cut the top-20 retrieval failure rate by 35%, adding contextual BM25 took the cut to 49%, and adding a reranker took it to 67%. Those are retrieval numbers on Anthropic’s datasets, not answer accuracy on yours, but the direction is consistent: each stage adds something the previous one missed.

Rerankers add latency because they run a full model over every candidate. Keep the candidate pool modest (Anthropic reranked 150 candidates down to 20), run self-hosted models on a GPU, and measure p95 latency with your real candidate counts before promising a response time.

Contextual retrieval works with every retriever

Contextual retrieval is an indexing step: before you embed a chunk or add it to BM25, you prepend a short, LLM-written note about where the chunk sits in its document. It is model-agnostic and composes with every method above. Anthropic estimated the one-time cost at $1.02 per million document tokens using prompt caching. Implementation details, including the prompt, are in the build guide, so I will not repeat them here.

A reference configuration

Here is how the pieces fit together as configuration. Treat the numbers as starting points to tune against your evaluation set, not recommendations backed by a benchmark.

# rag-pipeline-config.yaml
ingestion:
  chunking:
    strategy: "recursive_character"
    chunk_size: 512
    overlap: 64
    separators: ["\n## ", "\n### ", "\n\n", "\n", ". "]
  contextual_retrieval:
    enabled: true
    model: "claude-haiku-5-5"
    max_context_tokens: 100
  embedding:
    dense:
      model_id: "text-embedding-3-large"
    sparse:
      type: "bm25"            # or a learned sparse model, license permitting
  indexing:
    dense_store: "pgvector"   # or qdrant, pinecone, weaviate
    sparse_store: "elasticsearch"

retrieval:
  stage_1:
    dense_top_k: 40
    sparse_top_k: 40
    fusion: { method: "reciprocal_rank_fusion", k: 60 }
  stage_2:
    reranker: { model: "rerank-v4.0-pro", top_k: 20 }
  generation:
    model: "claude-sonnet-5-5"
    max_context_chunks: 8
    system_prompt_template: "answer_with_citations"

If you have not picked a vector store yet, our vector database comparison covers pgvector, Pinecone, Weaviate and Qdrant.

Evaluation results should feed back into retrieval settings, so regressions show up before users notice them.

Evaluate retrieval separately from generation

When a RAG answer is wrong, there are two possible culprits: the retriever did not find the right chunk, or the model misread it. You need to know which, because the fixes are different. The simplest way is a labelled set of questions, each tagged with the chunk IDs that contain the answer, and two metrics you can compute without any framework:

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    hits = len(set(retrieved[:k]) & relevant)
    return hits / len(relevant) if relevant else 0.0

def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    for i, chunk_id in enumerate(retrieved, start=1):
        if chunk_id in relevant:
            return 1.0 / i
    return 0.0

def evaluate_retriever(retrieve, eval_set, k: int = 20) -> dict:
    recalls, rrs = [], []
    for item in eval_set:  # {"question": str, "relevant_ids": set[str]}
        ids = [r.chunk_id for r in retrieve(item["question"])]
        recalls.append(recall_at_k(ids, item["relevant_ids"], k))
        rrs.append(reciprocal_rank(ids, item["relevant_ids"]))
    return {
        f"recall@{k}": sum(recalls) / len(recalls),
        "mrr": sum(rrs) / len(rrs),
    }

Run it on every retrieval change: a new embedding model, a different chunk size, a reranker swap. Embedding model upgrades are a classic source of silent regressions, because a new model can treat abbreviations or code identifiers differently from the old one.

For the generation side, frameworks such as Ragas provide LLM-judged metrics (faithfulness, response relevancy, context precision and context recall among them). Our guide to LLM evaluation with LangSmith and Braintrust covers how to run those in CI. Refresh the question set as the corpus changes. A stale test set gives false confidence.

RAG, long context or fine-tuning?

Retrieval is not the only way to get knowledge into a model, and sometimes it is the wrong one.

When long context is the better choice

Current frontier models accept very long prompts; the Claude models overview, for example, lists a 1M-token context window across the current lineup. Anthropic’s own contextual retrieval post says that if your knowledge base is under about 200,000 tokens (roughly 500 pages), you can often skip RAG, include everything in the prompt, and use prompt caching to keep it affordable.

Above that size, arithmetic takes over. At Claude Sonnet 5.5’s listed $2 per million input tokens, a 200,000-token prompt costs $0.40 in input per query before caching, while a RAG prompt of about 5,000 tokens costs $0.01. Prompt caching narrows the gap a lot for repeated prefixes (cache reads on Sonnet 5.5 are billed at 5% of the base input price), but it does not help once the corpus is too large to fit, and long prompts still take longer to process. Models also tend to be less precise at locating one fact in a very long context than in a short, focused one.

My rule of thumb: small and stable corpus, use long context with caching. Large or fast-changing corpus, use retrieval. In between, try both on your evaluation set and compare cost per correct answer.

When fine-tuning is the better choice

Choose RAG when:

• the knowledge changes weekly or faster
• answers need citations back to a source
• tenants must only see their own data
• the corpus is too large to bake into training data

Choose fine-tuning when:

• you need a specific tone, format or reasoning style
• the knowledge is stable
• you cannot afford retrieval latency

Many production systems use both: fine-tuning shapes how the model answers, retrieval supplies what it answers with. Our RAG vs fine-tuning decision guide goes deeper on that trade-off, and the piece on small language models in production covers when a smaller model plus retrieval beats a frontier model with a huge context window.

Recommendations

1. Start with hybrid search. Dense plus BM25 with RRF is cheap to build and covers both major failure types. Add a cross-encoder reranker when precision matters.
2. Build the evaluation set before tuning. A few hundred labelled questions and the recall@k harness above will tell you more than any benchmark table, including the ones in vendor blogs.
3. Treat SPLADE and ColBERT as upgrades to test, not defaults. SPLADE has a licensing question; ColBERT has an operations question.
4. Fix chunking before swapping retrievers. Contextual retrieval and structure-aware chunking often move recall more than a new retriever does.
5. Ask the model to cite and to admit uncertainty. Good retrieval does not stop a model from embellishing, so pair it with checks like those in mitigating LLM hallucination in production.

If the retrieval output feeds a multi-step agent instead of a single generation call, the orchestration layer needs the same evaluation discipline. Our agentic workflow orchestration guide covers that part of the system.