Bi-encoder retrieval (the kind used in standard vector search) is fast but imprecise — it compresses each document into a fixed-size vector and compares queries and documents independently. Cross-encoders are different: they look at the query and document together, producing a much more accurate relevance score at the cost of speed. Adding a cross-encoder reranking step after bi-encoder retrieval gives you the best of both: fast candidate retrieval followed by precise relevance scoring on a small shortlist. This guide shows exactly how to implement it.
Bi-Encoder vs Cross-Encoder: The Core Trade-Off
A bi-encoder embeds the query and each document independently into the same vector space, then ranks by cosine similarity. This is fast because document embeddings are precomputed and stored — ranking 100,000 documents at query time is just a vector similarity computation. The downside is accuracy: embedding a document in isolation loses context about how it relates to a specific query. A cross-encoder takes the query and document together as a single input and outputs a relevance score. This is far more accurate — the model sees the relationship between query and document — but cannot be precomputed, so it must run inference for every (query, document) pair at query time. Cross-encoders on 10-50 candidates take 1-3 seconds; running them on 100,000 documents is not feasible.
The Two-Stage Architecture
The standard solution is a two-stage pipeline: retrieve 50-100 candidates quickly with the bi-encoder (vector search), then rerank those candidates with the cross-encoder. You only pay the cross-encoder cost on a small shortlist, not the full collection. The top 5-10 after reranking are then passed to the LLM for generation. This consistently outperforms single-stage retrieval on standard RAG benchmarks (BEIR, MS-MARCO) by 5-15% on MRR@10.
from sentence_transformers import CrossEncoder
import chromadb
# Initialize
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_collection("documents")
def rerank_retrieve(query: str, n_candidates: int = 50, top_k: int = 5) -> list[dict]:
# Stage 1: fast bi-encoder retrieval
results = collection.query(
query_texts=[query],
n_results=n_candidates
)
candidates = [
{"text": doc, "metadata": meta}
for doc, meta in zip(
results["documents"][0],
results["metadatas"][0]
)
]
# Stage 2: cross-encoder reranking
pairs = [[query, c["text"]] for c in candidates]
scores = cross_encoder.predict(pairs)
# Sort by cross-encoder score, return top_k
ranked = sorted(
zip(scores, candidates),
key=lambda x: x[0],
reverse=True
)
return [{"score": float(s), **c} for s, c in ranked[:top_k]]
Choosing a Cross-Encoder Model
The sentence-transformers library has several cross-encoder checkpoints trained on MS-MARCO (passage retrieval) and other datasets. For general-purpose RAG, cross-encoder/ms-marco-MiniLM-L-6-v2 is the best starting point — it is fast (6-layer model), accurate, and widely benchmarked. For domain-specific applications, fine-tuned cross-encoders on domain data outperform general models significantly; the BEIR benchmark provides task-specific leaderboards for guidance. For multilingual content, use cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 which handles 13 languages. All of these run locally via sentence-transformers — no API calls required.
Integrating with a LangChain RAG Pipeline
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import OllamaEmbeddings
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder
import ollama
# Set up base retriever
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma(persist_directory="./chroma_db", embedding_function=embeddings)
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 50})
# Wrap with cross-encoder reranker
model = HuggingFaceCrossEncoder(model_name="cross-encoder/ms-marco-MiniLM-L-6-v2")
compressor = CrossEncoderReranker(model=model, top_n=5)
reranking_retriever = ContextualCompressionRetriever(
base_compressor=compressor,
base_retriever=base_retriever
)
def rag_with_reranking(query: str) -> str:
# Retrieves 50, reranks to 5, generates
docs = reranking_retriever.invoke(query)
context = "
".join(d.page_content for d in docs)
response = ollama.chat(
model="llama3.1",
messages=[
{"role": "system", "content": "Answer based on the context provided."},
{"role": "user", "content": f"Context:
{context}
Question: {query}"}
]
)
return response["message"]["content"]
Performance Impact
Reranking adds 0.5-2 seconds per query on CPU (the MiniLM-L-6 model is small and fast). On GPU it is under 200ms. For most RAG applications this is acceptable — generation takes 5-30 seconds anyway, so a 1-second reranking step is not the bottleneck. If latency is critical, reduce the candidate pool (try 20 instead of 50) or batch reranking requests. The quality improvement is most pronounced for queries that require precise semantic understanding — factual Q&A, technical documentation lookup, legal or medical search — and less impactful for keyword-heavy queries that bi-encoder retrieval already handles well.
Figure 1 — Two-stage retrieval: bi-encoder speed + cross-encoder precision
Fine-Tuning a Cross-Encoder on Your Data
A general cross-encoder trained on MS-MARCO works well for general text. For specialized domains — legal, medical, scientific, code — fine-tuning on domain-specific query-document pairs improves performance substantially. You need positive pairs (query, relevant document) and negative pairs (query, irrelevant document). The sentence-transformers library makes fine-tuning straightforward with the CrossEncoderTrainer class. Even 500-1000 labeled pairs typically produce a meaningful improvement over the general model for domain-specific retrieval tasks. Label your pairs using an LLM (GPT-4o or Claude to generate synthetic relevance labels) combined with a small manual review — this produces training data at low cost.
Reranking vs Retrieval Tuning
Reranking is not the only way to improve retrieval quality. Better chunking, better embedding models, hybrid search (BM25 + vector), and query expansion all improve retrieval independently. Reranking is most effective when the bi-encoder retrieval is already returning the right documents in the top 50 but ranking them poorly. If the relevant document is not in the top 50 at all, reranking cannot help — in that case, fix retrieval (improve chunking, try hybrid search, adjust chunk overlap). Use reranking as a refinement step on top of a working bi-encoder retrieval pipeline, not as a substitute for getting retrieval fundamentals right.
Summary
Cross-encoder reranking is one of the highest-leverage improvements you can make to a RAG pipeline. The pattern is simple — retrieve 50 with vector search, rerank to 5 with a cross-encoder, generate with the top 5 — and the implementation takes about 20 lines of code using sentence-transformers. The quality improvement is consistent and meaningful for most retrieval tasks. Run it locally alongside Ollama for a fully local, no-API-cost retrieval and generation stack that rivals cloud-hosted RAG in accuracy.
Batching for Efficiency
Cross-encoders accept batched input, so you can score all candidate pairs in a single inference call rather than looping. The sentence-transformers predict() method already handles batching internally — pass all pairs at once rather than in a loop, and set batch_size explicitly to control memory usage. On CPU, a batch of 50 pairs with MiniLM-L-6 takes about 1-1.5 seconds total. If you are running reranking as part of a web application, consider running it in a thread pool executor (FastAPI + asyncio.run_in_executor) to avoid blocking the event loop while the cross-encoder runs synchronously.
# Efficient batched reranking
import asyncio
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(max_workers=2)
async def async_rerank(query: str, candidates: list[str]) -> list[float]:
pairs = [[query, c] for c in candidates]
loop = asyncio.get_event_loop()
scores = await loop.run_in_executor(
executor,
lambda: cross_encoder.predict(pairs, batch_size=32)
)
return scores.tolist()
When Reranking Is Not Worth It
Reranking adds latency and complexity. Skip it when: your queries are short, keyword-heavy lookups where BM25 or exact-match retrieval already performs well; your document collection is small (under 1,000 documents) and bi-encoder retrieval is already precise; your application has strict latency requirements under 500ms end-to-end; or your evaluation shows that standard vector retrieval already places the relevant chunk in the top 3 consistently. Run an offline evaluation against a set of labeled query-document pairs before adding reranking — if your bi-encoder MRR@5 is already above 0.85, the marginal gain from reranking may not justify the added latency. Add it where the data shows it helps, not as a default for every RAG pipeline.
Evaluating Reranking Gains on Your Dataset
Before committing to a reranking step, measure whether it actually helps on your queries. Build a small offline test set: 50-100 questions where you know the correct answer and which document contains it. Run your pipeline twice — once with and once without reranking — and measure MRR@5 (Mean Reciprocal Rank at 5) and Recall@5. If your bi-encoder already places the correct chunk in position 1 or 2 the vast majority of the time, reranking adds latency without meaningful quality gain. The cases where reranking consistently helps are: complex multi-clause questions, paraphrase-heavy queries where the vocabulary in the question differs significantly from the vocabulary in the relevant document, and domain-specific content where the general embedding model underperforms.
Reciprocal Rank Fusion as a Cheaper Alternative
If cross-encoder latency is unacceptable and you are running hybrid search (BM25 + vector), Reciprocal Rank Fusion (RRF) is a lightweight reranking alternative that combines the two ranked lists without any inference cost. RRF scores each document as the sum of 1/(rank + k) across both retrievers, where k is typically 60. It consistently outperforms either retriever alone and adds microseconds rather than seconds of latency. Use RRF as the default fusion strategy for hybrid search, and reserve cross-encoder reranking for the final stage if quality still needs improvement after RRF.
def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[str]:
"""Fuse multiple ranked lists using RRF. Each list contains doc IDs in rank order."""
scores = {}
for ranked in ranked_lists:
for rank, doc_id in enumerate(ranked):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (rank + k)
return sorted(scores, key=scores.get, reverse=True)
# Example: fuse BM25 and vector search results
bm25_results = ["doc_3", "doc_1", "doc_7", "doc_2"]
vector_results = ["doc_1", "doc_3", "doc_9", "doc_4"]
fused = reciprocal_rank_fusion([bm25_results, vector_results])
# Returns docs ranked by combined RRF score
The Reranking Stack in Practice
A production retrieval stack typically layers multiple improvements. Start with good chunking and a quality embedding model — these affect the entire retrieval pipeline. Add hybrid search (BM25 + vector + RRF) to improve recall on keyword-heavy and paraphrase-heavy queries. Add cross-encoder reranking as the final precision step on the shortlist. Each layer improves different failure modes: chunking affects what can be retrieved, hybrid search improves recall, RRF improves fusion, and cross-encoding improves final ranking accuracy. Not every pipeline needs all four layers — evaluate each addition against your specific queries and latency requirements before adding it. The right stack is the simplest one that meets your quality bar.
Choosing K: How Many Candidates to Retrieve
The number of candidates retrieved before reranking (K) is a key parameter to tune. More candidates give the cross-encoder more material to work with, increasing the chance that the truly relevant document is in the pool — but they also increase reranking latency linearly. The practical sweet spot is 30-50 candidates for most applications. Below 20, retrieval recall suffers and the cross-encoder cannot compensate for documents that were never in the pool. Above 80, you pay significant latency for diminishing returns. The right K depends on how spread out your relevant content is: collections with a few highly relevant documents benefit from smaller K; collections where relevant information is distributed across many chunks benefit from larger K. Measure recall@K at different values on a labelled test set to find the knee of the curve for your specific collection before deploying.