Basic RAG — chunk, embed, retrieve top-K, generate — works well for straightforward document Q&A. But production RAG systems encounter harder queries that basic retrieval handles poorly: questions that span multiple documents, questions where the most semantically similar chunks aren’t the most relevant, questions requiring temporal or structural reasoning, and ambiguous queries where the user’s intent is unclear. Advanced RAG techniques address these failure modes. This guide covers the ones that consistently improve quality in production systems.
Reranking: Getting Better Relevance After Retrieval
Basic RAG ranks retrieved chunks by embedding cosine similarity — the distance between the query vector and each chunk vector in embedding space. Embedding models are trained for broad semantic similarity, not for precise relevance to a specific question. A cross-encoder reranker takes the query and each candidate chunk together as input and produces a more accurate relevance score, at higher computational cost.
The workflow: retrieve 20-30 candidates using fast embedding similarity, then rerank with a cross-encoder to select the top 3-5 for the LLM context. The extra reranking step typically adds 50-200ms of latency (on CPU with a small cross-encoder) and meaningfully improves retrieval precision, particularly for complex or specific questions:
pip install sentence-transformers
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def retrieve_and_rerank(query: str, n_retrieve: int = 20,
n_final: int = 5) -> list[str]:
# Step 1: fast ANN retrieval (broad candidates)
results = collection.query(
query_embeddings=[embed(query)],
n_results=n_retrieve
)
candidates = results["documents"][0]
# Step 2: rerank with cross-encoder
pairs = [(query, chunk) for chunk in candidates]
scores = reranker.predict(pairs)
# Step 3: return top n_final by reranker score
ranked = sorted(zip(scores, candidates), reverse=True)
return [chunk for _, chunk in ranked[:n_final]]The ms-marco-MiniLM-L-6-v2 cross-encoder is small (22MB), fast on CPU, and significantly improves precision over embedding similarity alone on passage retrieval tasks. For higher quality at greater latency cost, try cross-encoder/ms-marco-electra-base or BGE reranker models.
HyDE: Hypothetical Document Embeddings
HyDE (Hypothetical Document Embeddings) inverts the retrieval problem: instead of embedding the query and matching to document chunks, generate a hypothetical document that would answer the query and embed that. Hypothetical documents are more similar in embedding space to real answer-containing chunks than bare questions are, because they share vocabulary, phrasing, and semantic content with the target retrieval results.
def hyde_retrieve(query: str, n: int = 5) -> list[str]:
# Step 1: generate a hypothetical answer
hypo_response = ollama.chat(
model="llama3.1",
messages=[{"role": "user", "content":
f"Write a short paragraph that would directly answer this question, "
f"as if from a relevant document. Question: {query}"
}],
options={"temperature": 0.3, "num_predict": 200}
)
hypothetical_doc = hypo_response["message"]["content"]
# Step 2: embed the hypothetical document (not the query)
hypo_embedding = embed(hypothetical_doc)
# Step 3: retrieve using the hypothetical embedding
results = collection.query(
query_embeddings=[hypo_embedding],
n_results=n
)
return results["documents"][0]HyDE is particularly effective for question-answer style RAG where queries are short and documents are long — the vocabulary gap between a brief question and a detailed answer is large, and HyDE bridges it by expanding the query into answer-space. The downside: one additional LLM call before retrieval, adding 1-3 seconds of latency. Use HyDE when retrieval precision is more important than latency.
Parent-Child Chunking: Precise Retrieval, Rich Context
A common problem in RAG: small chunks retrieve precisely but lack surrounding context for the LLM to generate a good answer. Large chunks provide rich context but retrieve imprecisely. Parent-child chunking solves this by indexing small child chunks for retrieval but passing the surrounding parent chunk (or full section) to the LLM as context:
from langchain_text_splitters import RecursiveCharacterTextSplitter
def build_parent_child_index(text: str, source: str):
# Split into large parent chunks
parent_splitter = RecursiveCharacterTextSplitter(
chunk_size=1500, chunk_overlap=100)
parent_chunks = parent_splitter.split_text(text)
# Split each parent into smaller child chunks for indexing
child_splitter = RecursiveCharacterTextSplitter(
chunk_size=300, chunk_overlap=30)
for p_idx, parent in enumerate(parent_chunks):
children = child_splitter.split_text(parent)
collection.add(
ids=[f"{source}_p{p_idx}_c{c_idx}" for c_idx in range(len(children))],
embeddings=[embed(c) for c in children],
documents=children,
metadatas=[{"source": source, "parent_id": p_idx,
"parent_text": parent} for _ in children]
)
def retrieve_parent_context(query: str, n: int = 3) -> list[str]:
results = collection.query(
query_embeddings=[embed(query)],
n_results=n * 3, # retrieve more children to find n unique parents
include=["documents", "metadatas"]
)
# Deduplicate by parent_id and return parent texts
seen_parents = set()
parent_texts = []
for meta in results["metadatas"][0]:
if meta["parent_id"] not in seen_parents:
seen_parents.add(meta["parent_id"])
parent_texts.append(meta["parent_text"])
if len(parent_texts) == n:
break
return parent_textsFigure 1 — Advanced RAG Techniques: Impact vs Complexity
Query Expansion and Multi-Query Retrieval
Single queries often miss relevant content because the vocabulary doesn’t match the document’s phrasing. Query expansion generates multiple semantically related queries and retrieves for each, then deduplicates results:
def multi_query_retrieve(query: str, n_variants: int = 3,
n_per_query: int = 5) -> list[str]:
# Generate query variants
response = ollama.chat(
model="llama3.1",
messages=[{"role": "user", "content":
f"Generate {n_variants} different ways to ask this question, "
f"one per line, no numbering:\n\n{query}"
}],
options={"temperature": 0.7, "num_predict": 200}
)
variants = [query] + [
v.strip() for v in response["message"]["content"].split("\n")
if v.strip()
][:n_variants]
# Retrieve for each variant, deduplicate
seen = set()
all_chunks = []
for q in variants:
results = collection.query(
query_embeddings=[embed(q)],
n_results=n_per_query
)
for chunk in results["documents"][0]:
if chunk not in seen:
seen.add(chunk)
all_chunks.append(chunk)
return all_chunks[:n_per_query * 2] # return top combined resultsRAPTOR: Hierarchical Summarisation for Multi-Document Synthesis
RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) builds a hierarchical index of summaries: chunk documents, embed and cluster the chunks, summarise each cluster with an LLM, embed the summaries, cluster those, and repeat. The resulting tree allows retrieval at different levels of abstraction — specific details from leaf chunks, broader themes from cluster summaries. RAPTOR significantly improves RAG performance on questions that require synthesising information across many documents, where basic chunk retrieval returns fragments without the connecting narrative. The trade-off is indexing cost: building the RAPTOR tree requires many LLM calls (one per cluster at each level) and substantially more indexing time. For document collections where users frequently ask broad synthesis questions (“summarise the key themes across all these documents”), RAPTOR is worth the investment. For targeted fact retrieval, basic chunking with reranking is sufficient and much faster to implement.
Contextual Retrieval: Anthropic’s Approach
Contextual retrieval (introduced by Anthropic in 2024) addresses a specific problem: chunks retrieved out of context lose the surrounding information needed to understand them. A chunk that says “the policy was updated in March” is useless without knowing which policy. Contextual retrieval prepends a short context summary to each chunk before embedding it, making each chunk self-contained. The context is generated by an LLM: for each chunk, pass the full document and the chunk to the LLM and ask it to write a one-sentence context summary. Then concatenate the context and the chunk, and embed the combined text. This adds significant indexing cost (one LLM call per chunk) but improves retrieval performance substantially on documents with heavy co-reference and context-dependent content. The technique is most impactful on documents like meeting transcripts, legal filings, and technical manuals where chunks frequently reference entities defined elsewhere in the document.
RAG Fusion: Combining Multiple Retrieval Strategies
RAG fusion combines multiple retrieval signals using Reciprocal Rank Fusion (RRF) — the same technique used for hybrid search. Run the query through multiple retrieval strategies (embedding similarity, BM25 keyword search, HyDE, different chunking sizes), get a ranked list from each, and merge the rankings using RRF to produce a final combined ranking. The merged ranking surfaces chunks that rank well across multiple strategies, which tends to be more reliable than any single strategy alone. The implementation is straightforward: run each retrieval strategy, collect (chunk, rank) pairs from each, apply RRF scoring (1 / (60 + rank) per strategy), sum scores across strategies for each unique chunk, and sort by total score. The computational cost is the sum of all retrieval strategies plus the merge step — typically adding 50-200ms depending on how many strategies you run.
Practical Advice: Which Techniques to Try First
When basic RAG is not meeting quality requirements, try techniques in this order. First, add a cross-encoder reranker — this is the highest-impact, lowest-effort improvement available and works for almost every RAG use case. If retrieval is still missing relevant content, switch to parent-child chunking — the precise retrieval with rich context benefit is consistent and the implementation is not complex. If queries are broad or ambiguous, add multi-query expansion — the extra LLM call is worth it for open-ended questions. Only reach for HyDE, RAPTOR, or contextual retrieval if the simpler techniques are insufficient — they add meaningful complexity and cost that most RAG systems do not need. Track the quality impact of each change with a test set of representative question-answer pairs before deciding whether to keep the added complexity. Techniques that look theoretically powerful sometimes underperform on specific document types or query distributions — the benchmark on your actual workload is the only reliable signal.
RAG Evaluation Metrics: Measuring What Matters
Knowing whether your RAG improvements are actually improving quality requires systematic measurement. The key metrics for RAG evaluation: Context precision measures whether the retrieved chunks are actually relevant to the question — a high retrieval count with many irrelevant chunks is low precision. Computed by judging (manually or with an LLM judge) whether each retrieved chunk is relevant. Context recall measures whether the retrieved chunks contain all the information needed to answer the question — low recall means relevant content was missed. Requires knowing the ground-truth answer. Answer faithfulness measures whether the generated answer is grounded in the retrieved context — unfaithful answers make claims not supported by the retrieved content. Answer relevancy measures whether the generated answer actually addresses the question asked. The RAGAS library (pip install ragas) implements these metrics and can evaluate a RAG pipeline automatically using an LLM as a judge — pass it your question, context, and generated answer, and it returns scores for each metric. Building a small evaluation dataset of 20-50 representative question-answer pairs and running RAGAS evaluations after each pipeline change gives you a principled signal for whether changes are actually improving quality rather than just changing it.
The Compound Effect of Combined Techniques
Advanced RAG techniques are most powerful in combination. A production RAG system might use parent-child chunking for the index, retrieve 20 candidates, rerank with a cross-encoder to select the top 5 parent chunks, and pass those to the LLM. The parent-child design improves retrieval precision, the reranker further refines relevance, and the larger parent context gives the LLM what it needs to generate a complete, accurate answer. Each technique addresses a different failure mode, and combining complementary ones produces quality that no single technique achieves alone. Start simple, measure carefully, and add techniques only when evaluation shows clear quality gains — the overhead of complexity is only worth paying when the quality improvement is real and measured on your actual workload.