Standard RAG retrieves once and generates once. Corrective RAG (CRAG) and Self-RAG add a step that standard pipelines skip entirely: evaluating whether the retrieved content is actually good enough before generating. Both architectures recognise that retrieval sometimes fails — the retrieved chunks are irrelevant, outdated, or insufficient — and build in mechanisms to detect and correct those failures rather than silently generating a poor answer. This guide explains both approaches and shows how to implement them with local models via Ollama.
The Problem Standard RAG Ignores
In standard RAG the pipeline is deterministic: embed query, retrieve top-K, generate answer. If retrieval returns poor chunks — because the query is ambiguous, the document collection lacks relevant content, or embedding similarity misfires — the LLM generates a bad answer without any signal that retrieval failed. There is no mechanism for the model to say “I do not trust this context” or “I need more information.” CRAG and Self-RAG both add that mechanism, in different ways.
Corrective RAG (CRAG)
CRAG (Shi et al., 2024) adds a retrieval evaluator that scores whether the retrieved documents are relevant to the query. Based on the score, CRAG takes one of three actions: Correct (retrieved content is relevant — proceed to generate), Incorrect (irrelevant — discard and search the web instead), or Ambiguous (partially relevant — extract the relevant portions and optionally supplement with web search). The original paper uses a fine-tuned evaluator model, but in practice a simple LLM prompt works well enough for most applications.
import ollama
def evaluate_retrieval(query: str, chunks: list[str], model: str = "llama3.1") -> str:
"""Returns: correct / incorrect / ambiguous"""
context = "
---
".join(chunks[:3])
response = ollama.chat(
model=model,
messages=[{"role": "user", "content":
f"Question: {query}
Retrieved passages:
{context}
"
f"Are these passages relevant to the question? "
f"Reply with exactly one word: CORRECT, INCORRECT, or AMBIGUOUS."
}],
options={"temperature": 0, "num_predict": 10}
)
verdict = response["message"]["content"].strip().upper()
if "CORRECT" in verdict: return "correct"
if "INCORRECT" in verdict: return "incorrect"
return "ambiguous"
The Full CRAG Pipeline
With the evaluator in place, the complete pipeline branches based on the verdict. If retrieval is correct, generate directly. If incorrect, fall back to a web search or broader knowledge source. If ambiguous, extract only the relevant sentences and optionally supplement with a targeted search.
def crag_pipeline(query: str, vectorstore, model: str = "llama3.1") -> str:
chunks = vectorstore.similarity_search(query, k=5)
chunk_texts = [c.page_content for c in chunks]
verdict = evaluate_retrieval(query, chunk_texts, model)
if verdict == "correct":
context = "
".join(chunk_texts)
elif verdict == "incorrect":
context = web_search_fallback(query)
else:
context = extract_relevant_strips(query, chunk_texts, model)
context += "
" + web_search_fallback(query, num_results=2)
response = ollama.chat(model=model, messages=[
{"role": "system", "content": "Answer based on the provided context. If the context is insufficient, say so."},
{"role": "user", "content": f"Context:
{context}
Question: {query}"}
])
return response["message"]["content"]
Self-RAG: The Reflection Token Approach
Self-RAG (Asai et al., 2023) takes a different approach. Rather than a separate evaluator, Self-RAG trains the LLM to generate special reflection tokens inline: [Retrieve] (should I retrieve?), [IsRel] (is this chunk relevant?), [IsSup] (does my generation stay faithful?), and [IsUse] (is my output useful?). The model self-critiques at each step without an external judge. True Self-RAG requires a specifically fine-tuned model. With general-purpose models, you can approximate it by prompting the model to reflect on each step explicitly.
def self_rag_approximate(query: str, vectorstore, model: str = "llama3.1") -> dict:
# Step 1: does the model need to retrieve?
retrieve_check = ollama.chat(model=model, messages=[{"role": "user", "content":
f"Question: {query}
Can you answer this from training knowledge alone, "
f"or do you need to retrieve context? Reply: RETRIEVE or NO_RETRIEVE."
}], options={"temperature": 0, "num_predict": 20})
needs_retrieval = "RETRIEVE" in retrieve_check["message"]["content"].upper()
context = None
if needs_retrieval:
chunks = vectorstore.similarity_search(query, k=4)
relevant = []
for chunk in chunks:
rel = ollama.chat(model=model, messages=[{"role": "user", "content":
f"Query: {query}
Passage: {chunk.page_content[:500]}
Relevant? RELEVANT or NOT_RELEVANT."
}], options={"temperature": 0, "num_predict": 10})
if "RELEVANT" in rel["message"]["content"].upper():
relevant.append(chunk.page_content)
context = "
".join(relevant) if relevant else None
messages = [{"role": "user", "content":
(f"Context:
{context}
" if context else "") + f"Question: {query}
Answer:"
}]
answer = ollama.chat(model=model, messages=messages)["message"]["content"]
# Step 3: faithfulness check
faithful = True
if context:
faith = ollama.chat(model=model, messages=[{"role": "user", "content":
f"Context: {context[:800]}
Answer: {answer}
Is this answer supported? SUPPORTED or NOT_SUPPORTED."
}], options={"temperature": 0, "num_predict": 10})
faithful = "SUPPORTED" in faith["message"]["content"].upper()
return {"answer": answer, "used_retrieval": needs_retrieval, "faithful": faithful}
CRAG vs Self-RAG: When to Use Each
Figure 1 — CRAG vs Self-RAG across key dimensions
CRAG is easier to implement with any LLM and works well when retrieval failures are the primary problem — ambiguous queries or sparse document collections. Self-RAG (or its approximation) is better when faithfulness is the bottleneck — you want the model to flag when its answer goes beyond what the retrieved context supports. The approximation shown above works with any instruction-following model; a true Self-RAG requires a specifically fine-tuned model like the self-rag-llama3 variants on Hugging Face.
LangGraph for CRAG: Stateful Pipelines
If you are already using LangGraph, CRAG fits naturally as a conditional edge in a retrieval graph. Define retrieve, evaluate, and generate nodes with conditional routing based on the verdict. LangGraph’s state machine handles the branching cleanly and makes the pipeline easy to visualise and extend.
from langgraph.graph import StateGraph, END
from typing import TypedDict
class RAGState(TypedDict):
query: str; chunks: list[str]; verdict: str; answer: str
def retrieve_node(s): s["chunks"] = vectorstore.similarity_search(s["query"], k=5); return s
def evaluate_node(s): s["verdict"] = evaluate_retrieval(s["query"], s["chunks"]); return s
def generate_node(s):
ctx = "
".join(s["chunks"]) if s["verdict"] != "incorrect" else web_search_fallback(s["query"])
s["answer"] = ollama.chat(model="llama3.1", messages=[
{"role": "user", "content": f"Context:
{ctx}
Question: {s['query']}"}
])["message"]["content"]
return s
graph = StateGraph(RAGState)
graph.add_node("retrieve", retrieve_node)
graph.add_node("evaluate", evaluate_node)
graph.add_node("generate", generate_node)
graph.add_edge("retrieve", "evaluate")
graph.add_conditional_edges("evaluate", lambda s: s["verdict"],
{"correct": "generate", "incorrect": "generate", "ambiguous": "generate"})
graph.add_edge("generate", END)
graph.set_entry_point("retrieve")
crag_app = graph.compile()
When CRAG Outperforms Standard RAG
CRAG provides the biggest gains when your document collection has uneven coverage — some topics well-represented, others sparse or missing. Standard RAG always returns top-K regardless of relevance; CRAG’s evaluator catches failures before generation, directing the query to a web fallback or flagging insufficient context. Benchmarks from the original CRAG paper show 15-25% accuracy improvements on open-domain QA tasks, with the largest gains on queries where the local collection lacks relevant content. Log the evaluator verdict for every production query — over time this data shows exactly which topics need better document coverage and where retrieval is consistently failing.
Latency and Production Trade-Offs
Adding evaluation steps increases latency. A single CRAG evaluation call on a 7B model adds 2-4 seconds per query. The Self-RAG approximation with 3-4 reflection calls adds 8-15 seconds. For interactive applications, this is often too slow — consider running evaluations asynchronously or returning a preliminary answer quickly and updating it if the evaluation triggers a correction. For batch processing — document Q&A, research workflows, automated analysis — the latency cost is entirely acceptable and the quality improvement is worth it.
A practical two-model approach: run the evaluator on a small fast model (1.5B, low temperature) and the generator on your main model (7B+). The evaluator only needs to classify into three buckets, not generate nuanced prose — a 1.5B model handles this reliably and adds under a second of latency. The generator quality is unchanged. This keeps the total overhead manageable for most interactive use cases while retaining the quality benefit of CRAG’s branching logic.
Agentic RAG: Combining Both Approaches
Production RAG systems increasingly combine CRAG-style retrieval evaluation with Self-RAG-style generation reflection in an agentic loop that can re-retrieve with a reformulated query if the first attempt scores poorly. The basic loop: retrieve, evaluate retrieval quality, optionally reformulate and re-retrieve, generate, evaluate faithfulness, optionally regenerate. Each evaluation step uses a cheap fast model (small model, temperature 0), while generation uses your main model. This keeps total latency manageable while adding meaningful quality checks at both retrieval and generation stages. Start with the simpler CRAG pattern, measure quality gains, and add the generation reflection layer only if faithfulness remains a problem after retrieval quality is addressed.
Implementation Tips That Actually Matter
A few details make the difference between a CRAG implementation that works reliably and one that produces inconsistent verdicts. Keep the evaluation prompt short and deterministic — set temperature to 0 and limit token output to 10-20 tokens to ensure consistent CORRECT/INCORRECT/AMBIGUOUS classification. Evaluate only the top 3 retrieved chunks rather than all K; the most relevant chunks appear first and evaluating more adds latency without proportional benefit. Cache evaluation results keyed by (query hash, top chunk hashes) — the same query against the same documents always gets the same verdict, so caching eliminates redundant evaluator calls in systems with query repetition. For the web fallback, DuckDuckGo’s zero-cost instant answer API handles most factual queries; SerpAPI or Brave Search provide fuller results for research-style queries.
Tracking Retrieval Quality Over Time
Once CRAG is in production, log the evaluator verdict for every query. Over time this data is genuinely valuable: you can see which query patterns consistently produce INCORRECT verdicts (your document collection lacks coverage there), which produce AMBIGUOUS (your chunking may be too coarse), and which produce CORRECT (the pipeline is working). This verdict log directly informs what documents to add, how to refine chunking for underperforming topics, and where the evaluator prompt may itself be miscalibrated. Treat it as a product feedback signal, revisit it monthly, and use it to drive systematic improvements rather than reacting to user complaints. The combination of automated quality checking at query time and verdict-based monitoring over time gives you a self-improving retrieval system rather than a static one.
Starting Small: The One-Prompt Upgrade
If you take one thing from this article, it is this: adding a single evaluation call before generation is a low-risk, high-reward upgrade to any RAG pipeline. You do not need LangGraph, you do not need a fine-tuned evaluator model, and you do not need to restructure your pipeline. Just add the evaluate_retrieval() function from this article, wrap your existing generation call in the three-branch conditional, and log the verdicts. That alone gets you most of the CRAG benefit — the web fallback for incorrect retrievals and the relevant-sentence extraction for ambiguous ones are enhancements you can layer in once the basic evaluator is running and you have data on how often each verdict fires in your specific application. Small, measurable, incremental.
CRAG is the practical starting point — one extra prompt, no new models, immediate improvement for queries that fall outside your document collection’s coverage. Self-RAG’s reflection approach pays off when generation faithfulness is the bottleneck, particularly in high-stakes domains where hallucinated answers are costly. Both run entirely locally with Ollama, giving you quality safeguards without any cloud dependency or per-token evaluation cost. Start with CRAG, measure whether it improves your specific pipeline, and layer in Self-RAG-style faithfulness checks only where the data shows they are needed.