Bi-encoder retrieval — the kind used in standard vector search — is fast but imprecise. It compresses each document into a fixed-size vector and scores query-document similarity independently, without either knowing anything about the other. Cross-encoders flip this: they see the query and document together and output a single relevance score. That joint view is far more accurate, but it cannot be precomputed, which makes it too slow to run over an entire collection at query time. The solution is a two-stage pipeline: retrieve a shortlist quickly with the bi-encoder, then rerank that shortlist precisely with the cross-encoder. This guide shows exactly how to set it up.
Bi-Encoder vs Cross-Encoder: The Core Trade-Off
A bi-encoder embeds the query and each document independently into the same vector space, then ranks by cosine similarity. This is fast because document embeddings are precomputed and stored — ranking 100,000 documents at query time is just a vector lookup. The downside is accuracy: encoding a document in isolation loses context about how it relates to a specific query. A cross-encoder takes the query and document together as a single input and outputs a relevance score. It is far more accurate — the model sees the relationship directly — but must run inference for every (query, document) pair at query time. Running a cross-encoder over 100,000 documents is not feasible; running it over 50 candidates takes 1-2 seconds.
The Two-Stage Architecture
Retrieve 50-100 candidates quickly with the bi-encoder, then rerank those candidates with the cross-encoder, then pass the top 5-10 to the LLM for generation. You only pay the cross-encoder cost on a small shortlist. This consistently outperforms single-stage retrieval on standard benchmarks (BEIR, MS-MARCO) by 5-15% on MRR@10.
from sentence_transformers import CrossEncoder
import chromadb
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_collection("documents")
def rerank_retrieve(query: str, n_candidates: int = 50, top_k: int = 5) -> list[dict]:
# Stage 1: fast bi-encoder retrieval
results = collection.query(query_texts=[query], n_results=n_candidates)
candidates = [
{"text": doc, "metadata": meta}
for doc, meta in zip(results["documents"][0], results["metadatas"][0])
]
# Stage 2: cross-encoder reranking
pairs = [[query, c["text"]] for c in candidates]
scores = cross_encoder.predict(pairs)
ranked = sorted(zip(scores, candidates), key=lambda x: x[0], reverse=True)
return [{"score": float(s), **c} for s, c in ranked[:top_k]]
Choosing a Cross-Encoder Model
For general-purpose RAG, cross-encoder/ms-marco-MiniLM-L-6-v2 is the best starting point — it is fast (6-layer model), well-benchmarked, and requires no additional setup. For multilingual content, use cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 which handles 13 languages. For domain-specific applications, a fine-tuned cross-encoder on domain data will outperform the general model significantly. All of these run locally via sentence-transformers with no API calls.
Integrating with a LangChain RAG Pipeline
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import OllamaEmbeddings
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder
import ollama
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma(persist_directory="./chroma_db", embedding_function=embeddings)
base_retriever = vectorstore.as_retriever(search_kwargs={"k": 50})
model = HuggingFaceCrossEncoder(model_name="cross-encoder/ms-marco-MiniLM-L-6-v2")
compressor = CrossEncoderReranker(model=model, top_n=5)
reranking_retriever = ContextualCompressionRetriever(
base_compressor=compressor, base_retriever=base_retriever
)
def rag_with_reranking(query: str) -> str:
docs = reranking_retriever.invoke(query)
context = "
".join(d.page_content for d in docs)
response = ollama.chat(model="llama3.1", messages=[
{"role": "system", "content": "Answer based on the context provided."},
{"role": "user", "content": f"Context:
{context}
Question: {query}"}
])
return response["message"]["content"]
Batching for Efficiency
Cross-encoders accept batched input, so pass all pairs at once rather than looping. The sentence-transformers predict() method handles batching internally — set batch_size explicitly to control memory usage. On CPU, a batch of 50 pairs with MiniLM-L-6 takes about 1-1.5 seconds total. If you are running reranking in a FastAPI application, run it in a thread pool executor to avoid blocking the event loop:
import asyncio
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(max_workers=2)
async def async_rerank(query: str, candidates: list[str]) -> list[float]:
pairs = [[query, c] for c in candidates]
loop = asyncio.get_event_loop()
scores = await loop.run_in_executor(
executor, lambda: cross_encoder.predict(pairs, batch_size=32)
)
return scores.tolist()
Reciprocal Rank Fusion as a Cheaper Alternative
If cross-encoder latency is unacceptable and you are running hybrid search (BM25 + vector), Reciprocal Rank Fusion (RRF) is a lightweight reranking alternative that combines the two ranked lists without any inference cost. RRF scores each document as the sum of 1/(rank + k) across both retrievers, where k is typically 60. It consistently outperforms either retriever alone and adds microseconds rather than seconds of latency. Use RRF as the default fusion strategy for hybrid search, and add cross-encoder reranking only if quality still needs improvement after RRF.
def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[str]:
scores = {}
for ranked in ranked_lists:
for rank, doc_id in enumerate(ranked):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (rank + k)
return sorted(scores, key=scores.get, reverse=True)
fused = reciprocal_rank_fusion([bm25_results, vector_results])
Choosing K: How Many Candidates to Retrieve
The practical sweet spot is 30-50 candidates. Below 20, retrieval recall suffers and the cross-encoder cannot compensate for documents that were never in the pool. Above 80, you pay significant latency for diminishing returns. Measure recall@K at different values on a labelled test set to find the knee of the curve for your specific collection — collections where relevant information is spread across many chunks need larger K than collections with a few highly relevant documents.
Figure 1 — Two-stage retrieval: bi-encoder speed combined with cross-encoder precision
Evaluating Whether Reranking Actually Helps
Before committing to reranking, measure whether it helps on your specific queries. Build a test set of 50-100 questions where you know the correct answer and which document contains it. Run your pipeline with and without reranking and compare MRR@5 (Mean Reciprocal Rank at 5) and Recall@5. If your bi-encoder already places the correct chunk in position 1 or 2 consistently, reranking adds latency without meaningful quality gain. Reranking helps most for complex multi-clause questions, paraphrase-heavy queries where vocabulary differs significantly between query and document, and domain-specific content where the general embedding model underperforms.
Fine-Tuning a Cross-Encoder on Your Data
A general cross-encoder trained on MS-MARCO works well for general text. For specialized domains — legal, medical, scientific, code — fine-tuning on domain-specific query-document pairs improves performance substantially. You need positive pairs (query, relevant document) and negative pairs (query, irrelevant document). The sentence-transformers library makes fine-tuning straightforward with the CrossEncoderTrainer class. Even 500-1,000 labeled pairs typically produce a meaningful improvement. Label your pairs using an LLM to generate synthetic relevance judgments, then do a small manual review pass — this produces training data at low cost and reasonable quality.
When Reranking Is Not Worth It
Reranking adds latency and complexity — skip it when your queries are short keyword lookups where BM25 already performs well, your collection is small (under 1,000 documents) and bi-encoder retrieval is already precise, your application has strict sub-500ms latency requirements, or your evaluation shows the bi-encoder is already placing the relevant chunk in the top 3 consistently. Reranking is a refinement on top of a working retrieval pipeline, not a substitute for getting retrieval fundamentals right. If the relevant document is not in the top 50 at all, a cross-encoder cannot help — fix retrieval first.
The Full Retrieval Stack in Practice
A production retrieval stack typically layers multiple improvements, each addressing a different failure mode. Good chunking affects what can be retrieved at all — the right answer needs to fit within a single chunk for retrieval to find it. Hybrid search (BM25 + vector + RRF) improves recall on keyword-heavy and paraphrase-heavy queries that pure vector search handles poorly. Cross-encoder reranking improves final ranking accuracy on the shortlist that reaches the LLM. Not every pipeline needs all three layers — evaluate each addition against your specific queries and latency budget. The right stack is the simplest one that meets your quality bar, not the most technically sophisticated one.
Performance on Real Hardware
To set accurate expectations: on a machine with a consumer NVIDIA GPU (RTX 3080 or equivalent), cross-encoder reranking on 50 candidate pairs with MiniLM-L-6 takes under 200ms. On an M2/M3 MacBook Pro using Metal, it takes 300-600ms. On CPU-only, 1-2 seconds. These numbers are well within acceptable range for most RAG applications where LLM generation already takes 5-30 seconds. The latency cost of reranking is almost never the bottleneck — if your application is slow, the bottleneck is generation, not reranking. Measure before optimising, and do not skip reranking for performance reasons until you have profiled the actual query latency breakdown.
Connecting Reranking to Your Evaluation Workflow
Add a before/after reranking comparison to your offline evaluation harness. Run the same set of labelled test queries through your pipeline with reranking disabled and then with it enabled, comparing MRR@5 and Recall@5 for both. This gives you a clean measurement of the quality delta on your actual queries rather than relying on published benchmark numbers that may not reflect your domain. Store both results in your evaluation changelog. If reranking improves MRR@5 by more than 5 points on your test set, it is worth the latency cost; if the improvement is under 2 points, the overhead may not be justified. Let your own data make the decision rather than defaulting to “reranking is always better.”
Cross-encoder reranking is one of the highest-leverage improvements you can make to a RAG pipeline. The pattern is simple — retrieve 50 with vector search, rerank to 5 with a cross-encoder, generate with the top 5 — and the implementation takes about 20 lines using sentence-transformers. Run it locally alongside Ollama for a fully local, no-API-cost stack. Evaluate on a labelled test set first to confirm it improves your specific queries, add it if it does, and reach for RRF or fine-tuning if you need to go further.