Contextual retrieval: how to build a RAG pipeline that works

Most disappointing RAG systems fail at retrieval, not generation. The model writes a fluent answer from the wrong chunk, and nobody notices until a customer gets told something that is not in the policy. Better embeddings and bigger models rarely fix that. Rebuilding the retrieval pipeline in stages usually does.

This is a build guide for that pipeline, in the order I would build it, with code for each stage. If you are still deciding which retriever to use (dense, BM25, SPLADE, ColBERT), read the companion piece, RAG retrieval methods compared, first. This post assumes hybrid search and focuses on making it work.

How do you build a RAG pipeline that works?

Chunk documents along their structure, then use an LLM to prepend a short context note to each chunk. Index the contextualized chunks twice, as dense vectors and in BM25, and merge both result lists with reciprocal rank fusion. Rerank the merged candidates, generate with citations, and verify each claim against the retrieved chunks before answering.

That is contextual retrieval plus the stages around it. The rest of this post builds each one.

Why naive RAG breaks

The original pattern looked like this:

User query → Embed → Vector search → Top K chunks → LLM → Answer

It works in demos and breaks in production, for four reasons:

1. Chunks lose context. Split a document into 500-token pieces and each piece forgets which document it came from, which section it was in, and what came before it.
2. Similarity is not relevance. “How do I cancel my subscription?” and “Our subscription plans offer flexibility” are close in embedding space. Answering with the second chunk is wrong.
3. No exact matching. Ask about “Policy 4.2.1” and vector search may return chunks about policy guidelines in general.
4. The top-K trap. Five chunks sounds like plenty until the answer is spread across fifteen chunks in three documents.

The 2024 response was a pile of patches: reranking, query expansion, hypothetical document embeddings, metadata filters, parent-child chunks. Each helped a little. Contextual retrieval goes after the first problem directly: every chunk should carry enough context to be useful on its own.

Step 1: chunk along document structure

Your chunking sets the ceiling for everything downstream. A reranker can only reorder the chunks it is given; it cannot repair a chunk that split a table in half.

Fixed-size chunking

Most tutorials start here:

# Avoid this in production
def fixed_chunk(text: str, chunk_size: int = 500) -> list[str]:
    words = text.split()
    return [
        " ".join(words[i:i + chunk_size])
        for i in range(0, len(words), chunk_size)
    ]

It cuts sentences in half, separates headings from the paragraphs under them, and splits tables across chunks. It is fine for a prototype and nothing else.

Recursive chunking

Recursive splitters try paragraph breaks first, then line breaks, then sentences, then words, so most chunks end at a natural boundary. In current LangChain the splitter lives in its own package, langchain-text-splitters:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " ", ""],
)

chunks = splitter.split_text(document_text)

Putting Markdown headings at the front of the separator list keeps sections together when the source has structure. This is the right default for most corpora.

Semantic chunking

Semantic chunking embeds each sentence and starts a new chunk where the similarity between neighbouring sentences drops:

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def semantic_chunk(sentences: list[str], threshold: float = 0.3) -> list[str]:
    """Split where similarity between adjacent sentences drops."""
    embeddings = model.encode(sentences, normalize_embeddings=True)
    chunks, current = [], [sentences[0]]

    for i in range(1, len(sentences)):
        sim = float(np.dot(embeddings[i - 1], embeddings[i]))
        if sim < threshold:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])

    chunks.append(" ".join(current))
    return chunks

It helps on long, loosely structured prose (transcripts, essays, support threads) where headings do not exist. On well-structured docs it often adds cost without beating a recursive splitter. The threshold needs tuning per corpus, so test it against your evaluation set before adopting it.

Comparing the options

Strategy	Respects structure	Indexing cost	Complexity	Use it for
Fixed size	No	Negligible	Trivial	Prototypes
Recursive	Mostly	Negligible	Low	Most corpora
Semantic	By meaning	One embedding per sentence	Medium	Unstructured prose
Contextual (any of the above plus context notes)	Yes, and each chunk is self-describing	One LLM call per chunk	High	Anything where wrong answers are costly

Contextual is not a replacement for the others. It is a layer you add on top of whichever splitter you pick.

Step 2: add context to every chunk

This is the step that gives contextual retrieval its name. The technique was published by Anthropic in September 2024: before indexing a chunk, ask an LLM to write a short note that situates it within the whole document, and prepend the note to the chunk. A chunk that said “Revenue grew 3% over the previous quarter” becomes one that also says which company and which quarter.

The prompt below follows the one in Anthropic’s write-up. The document goes in its own content block with cache_control, so every chunk after the first reads the document from the prompt cache instead of paying for it again. That caching is what makes the cost manageable.

import anthropic

client = anthropic.Anthropic()

CONTEXT_PROMPT = """Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the overall
document for the purposes of improving search retrieval of the chunk. Answer
only with the succinct context and nothing else."""


def situate_chunk(document: str, chunk: str) -> str:
    response = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=1024,
        output_config={"effort": "low"},  # short, simple task
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": f"<document>\n{document}\n</document>",
                    "cache_control": {"type": "ephemeral"},
                },
                {"type": "text", "text": CONTEXT_PROMPT.format(chunk=chunk)},
            ],
        }],
    )
    return next(b.text for b in response.content if b.type == "text")


def contextualize(document: str, chunks: list[str]) -> list[str]:
    """Return chunks with their context note prepended, ready to index."""
    return [f"{situate_chunk(document, c)}\n\n{c}" for c in chunks]

A few practical notes:

• Process all chunks of one document back to back, so the cached document prefix is still warm.
• Very short documents fall under the minimum cacheable prompt length and will not cache. That is fine; they are cheap anyway.
• Store the context note and the raw chunk separately. Index the combined text, but show users the raw chunk in citations.
• Anthropic used Claude 3 Haiku for its cost estimate of $1.02 per million document tokens, assuming 800-token chunks, 8,000-token documents and 100-token context notes. Any small, cheap model works. The model ID above is the current Haiku from the Claude models overview.

The cost is paid once per document version, at indexing time, and every later query benefits.

Step 3: index twice and search with hybrid fusion

Vector search and BM25 fail in opposite ways. Vector search misses exact strings (policy numbers, product names, error codes). BM25 misses synonyms and paraphrase. So index the contextualized chunks both ways and merge the results.

Anthropic’s numbers show why: contextual embeddings alone cut top-20 retrieval failures by 35%, and adding a BM25 index built over the same contextualized chunks (what they call contextual BM25) took the cut to 49%.

Qdrant can hold both vector types in one collection and fuse them server side. This follows the pattern in Qdrant’s hybrid query docs:

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")


def hybrid_search(query: str, top_k: int = 50) -> list[dict]:
    dense_vector = embed_query(query)              # list[float]
    indices, values = sparse_encode(query)         # BM25 or another sparse encoder

    response = client.query_points(
        collection_name="docs",
        prefetch=[
            models.Prefetch(query=dense_vector, using="dense", limit=top_k),
            models.Prefetch(
                query=models.SparseVector(indices=indices, values=values),
                using="sparse",
                limit=top_k,
            ),
        ],
        query=models.RrfQuery(rrf=models.Rrf(k=60)),  # reciprocal rank fusion
        limit=top_k,
    )

    return [
        {
            "chunk_id": point.id,
            "text": point.payload["text"],
            "source": point.payload["source"],
            "score": point.score,
        }
        for point in response.points
    ]

Reciprocal rank fusion merges the two lists by rank rather than raw score, which sidesteps the fact that cosine similarity and BM25 scores are on different scales. The retrieval methods comparison covers RRF, the k = 60 constant, and when SPLADE or ColBERT are worth considering instead of plain BM25.

Retrieve generously at this stage. Anthropic found 20 final chunks performed better than 5 or 10, and fetched 150 candidates for the reranker to choose from.

Step 4: rerank the candidates

Hybrid search gets the right chunk into the candidate list. A reranker gets it to the top. A cross-encoder reads the query and each candidate together, so it can catch distinctions a bi-encoder misses, like a chunk about the right feature but the wrong product tier.

In Anthropic’s tests, reranking on top of contextual embeddings and contextual BM25 took the cut in retrieval failures from 49% to 67%. That was the largest single step in their stack.

Cohere’s hosted reranker is the easiest to try. The current API reference uses rerank-v4.0-pro in its examples, and each result carries the index of the document in your input list plus a relevance_score:

import cohere

co = cohere.ClientV2()  # reads CO_API_KEY from the environment


def rerank(query: str, candidates: list[dict], top_n: int = 20) -> list[dict]:
    response = co.rerank(
        model="rerank-v4.0-pro",
        query=query,
        documents=[c["text"] for c in candidates],
        top_n=top_n,
    )
    return [
        {**candidates[r.index], "relevance_score": r.relevance_score}
        for r in response.results
    ]

The options, qualitatively:

Reranker	Hosting	Latency	Good fit
Cohere Rerank	API	Network round trip plus model time	Most teams, quickest to ship
Open cross-encoders (e.g. bge-reranker)	Self-hosted, ideally on GPU	Depends on hardware and candidate count	Data that cannot leave your network
ColBERT as a reranker	Self-hosted	Higher setup cost	Long documents where exact wording matters
None	n/a	None	Very cost-sensitive, low-stakes search

My view: unless you are extremely cost-sensitive, use one. Measure the latency it adds with your real candidate count before you promise a response time, because reranking 150 candidates is a different cost from reranking 30.

Step 5: analyse the query before searching

Most pipelines search for the user’s words exactly as typed. A short analysis step first can extract identifiers for BM25, metadata filters, and alternative phrasings:

async def analyze_query(query: str) -> dict:
    """Understand the query before searching."""
    return await call_llm_json(f"""Analyze this search query:

Query: "{query}"

Return JSON with:
- intent: "factual", "comparison", "procedural", "exploratory" or "troubleshooting"
- entities: specific names, numbers or codes mentioned
- keywords: important terms for keyword search
- expanded_queries: 2-3 alternative phrasings that might match relevant documents
- filters: any metadata filters (date ranges, document types, categories)
""")

# Query: "What's our refund policy for enterprise customers who signed after January 2025?"
# {
#   "intent": "factual",
#   "entities": ["enterprise", "January 2025"],
#   "keywords": ["refund policy", "enterprise customers"],
#   "expanded_queries": [
#     "enterprise refund policy terms",
#     "refund conditions for enterprise tier",
#     "enterprise customer agreement refund clause"
#   ],
#   "filters": {"document_type": ["policy", "agreement"], "date_after": "2025-01-01"}
# }

Run the expanded queries through hybrid search in parallel and merge everything before reranking. Apply the filters before the search, not after. A question about “2025 Q3 revenue” should never see a 2023 report.

Step 6: generate with citations, then verify them

Ask the model to cite chunk IDs for every claim. Then check the citations, because a model can cite a real chunk and still say something the chunk does not support.

async def verify_citations(response: str, source_chunks: list[str]) -> dict:
    """Check that each claim in the response is supported by a source."""
    verification = await call_llm_json(f"""
You are a fact-checker. Compare the response against the source documents.

RESPONSE:
{response}

SOURCE DOCUMENTS:
{format_sources(source_chunks)}

For each factual claim in the response:
1. Is it directly supported by a source? (SUPPORTED / UNSUPPORTED / PARTIALLY_SUPPORTED)
2. If supported, which source?
3. If unsupported, is it a reasonable inference or a likely hallucination?

Return JSON: {{"claims": [{{"claim": str, "status": str, "source": str | null}}]}}
""")

    unsupported = [c for c in verification["claims"] if c["status"] == "UNSUPPORTED"]

    if unsupported:
        return await regenerate_without_unsupported(response, source_chunks, unsupported)

    return {"response": response, "verified": True}

It costs one extra LLM call per query, and a small model is usually enough for the check. For user-facing systems I would not skip it. It is one technique among several covered in mitigating LLM hallucination in production.

The full reference stack

Here is the whole pipeline in one place. It uses Qdrant, but the choice between Qdrant, pgvector, Pinecone and Weaviate matters less than getting the stages right; our vector database comparison helps if you are still choosing.

Ingestion:
  1. Parse documents (Unstructured or Docling for PDFs and complex layouts)
  2. Chunk along structure (recursive splitter; semantic for unstructured prose)
  3. Contextualize each chunk with a small Claude model and prompt caching
  4. Embed twice: dense vectors + sparse (BM25) over the contextualized text
  5. Upsert into Qdrant with both named vectors and the metadata

Query:
  1. Analyse the query (intent, entities, filters, expansions)
  2. Hybrid search per expanded query, RRF fusion, ~150 candidates
  3. Rerank to the top ~20
  4. Assemble context (deduplicate, order by document position)
  5. Generate with citations
  6. Verify citations against the source chunks

What the published numbers say

The best public data on this approach is still Anthropic’s contextual retrieval write-up. Measured as the top-20 retrieval failure rate across its test datasets:

Configuration	Failure rate	Reduction vs baseline
Standard embeddings	5.7%
Contextual embeddings	3.7%	35%
Contextual embeddings + contextual BM25	2.9%	49%
Reranked contextual embeddings + contextual BM25	1.9%	67%

These are retrieval numbers, not end-to-end answer accuracy, and they come from Anthropic’s datasets, not yours. The shape is what matters: each stage stacks on the one before it, and no single stage does most of the work. Your own evaluation set will tell you how large each gain is on your data.

When to skip RAG

RAG is not always the answer.

1. The whole knowledge base fits in the prompt. Anthropic’s guidance is that below about 200,000 tokens (around 500 pages), you can often include everything in the prompt and use prompt caching instead of building retrieval. The retrieval methods comparison works through the cost arithmetic for larger corpora.
2. Single-document questions. If users ask about one document at a time, pass that document in.
3. The model already knows the answer. A general programming assistant does not need retrieval over textbooks it was trained on.
4. Live data. Stock prices, inventory, weather and order status need API calls or tools, not a document index.

A simple decision path:

Do you have private or recent documents the model hasn't seen?
  └── No → Skip RAG. Prompt the model, or fine-tune for style.
  └── Yes →
      Does the whole corpus fit comfortably under ~200K tokens?
        └── Yes → Put it in the prompt with prompt caching. Revisit if it grows.
        └── No  → Build the retrieval pipeline above.

If you are weighing retrieval against fine-tuning, our RAG vs fine-tuning decision guide covers that trade-off.

Common mistakes

1. Not evaluating retrieval separately from generation. You need to know whether the wrong chunk was retrieved or the right chunk was misread. The fixes have nothing in common.
2. One embedding model for every content type. Code, legal text, chat logs and technical docs embed differently. Test the model on your actual data before committing.
3. Ignoring metadata. Titles, dates, authors and document types are cheap, precise filters.
4. Never updating the index. Documents change. You need incremental indexing, and contextual retrieval means re-running the context step when a document changes, since the notes depend on the whole document.
5. No evaluation set. Build 100 to 200 questions with known answers and the chunks that contain them, and run them on every pipeline change. Our guide to LLM evaluation with LangSmith and Braintrust covers the tooling, and responses you serve repeatedly can be cached.

The order to build it in

1. Recursive chunking and vector search. Get the basics working end to end.
2. BM25 alongside the vectors, merged with RRF. The biggest gain for the effort in most corpora.
3. A reranker. Cohere’s API is the quickest to integrate.
4. An evaluation set, before going further, so you can measure the next steps.
5. Contextual chunking. Now you can check whether the indexing cost pays off on your data.
6. Query analysis and expansion, kept only if the evaluation set improves.
7. Citation verification, for anything user-facing.

Measure every step. If a step does not improve results on your data, drop it. The stack is a starting point for iteration, not a fixed prescription.

Beyond the basic pipeline

Three directions are worth watching once the pipeline above is solid. Agentic retrieval lets a model decide which searches to run and when to search again, which suits research-style questions; the patterns in agentic AI workflow orchestration apply. Graph-based retrieval adds entity and relationship structure on top of vector search, which helps with heavily interconnected enterprise data. Multimodal retrieval indexes tables, diagrams and images alongside text, which matters in manufacturing and engineering documentation where the answer is often in a figure.

If your RAG system is underperforming, or you want a second opinion on a retrieval design before you build it, get in touch with CODERCOPS.