Standard RAG retrieves once and generates once. Corrective RAG (CRAG) and Self-RAG add a critical missing step: evaluating whether the retrieved content is actually good enough before generating an answer. Both architectures recognise that retrieval sometimes fails — the retrieved chunks are irrelevant, out of date, or insufficient — and build in mechanisms to detect and correct these failures. This guide explains both approaches and shows how to implement them with local models.
The Core Problem: Retrieval Is Not Always Reliable
In standard RAG, the pipeline is deterministic: embed query, retrieve top-K, generate answer. If retrieval returns poor chunks — because the query is ambiguous, the document collection lacks relevant content, or embedding similarity misfires on this particular query — the LLM generates a poor answer or hallucinates without any signal that retrieval failed. The LLM has no mechanism to flag “I don’t trust the retrieved context” or “I need more information before answering.” CRAG and Self-RAG both address this by making retrieval quality an explicit checkpoint in the pipeline.
Corrective RAG (CRAG)
CRAG (Shi et al., 2024) adds a retrieval evaluator — a lightweight model that scores whether the retrieved documents are relevant to the query. Based on this score, CRAG takes one of three actions: Correct (retrieved content is relevant — proceed to generate), Incorrect (retrieved content is irrelevant — discard and search the web instead), or Ambiguous (content is partially relevant — extract the relevant portions and supplement with web search). The original CRAG paper uses a small fine-tuned retrieval evaluator model, but in practice you can use your main LLM as the evaluator with a simple relevance-checking prompt. This is slower but requires no additional model and works well enough for most applications.
import ollama
def evaluate_retrieval(query: str, chunks: list[str],
model: str = "llama3.1") -> str:
"""Evaluate whether retrieved chunks are relevant. Returns: correct/incorrect/ambiguous."""
context = "\n\n---\n\n".join(chunks[:3]) # evaluate top 3
response = ollama.chat(
model=model,
messages=[{"role": "user", "content":
f"Evaluate whether these retrieved passages are relevant to answer the question.\n\n"
f"Question: {query}\n\n"
f"Retrieved passages:\n{context}\n\n"
f"Are these passages relevant to the question? "
f"Reply with exactly one word: CORRECT, INCORRECT, or AMBIGUOUS."
}],
options={"temperature": 0, "num_predict": 10}
)
verdict = response["message"]["content"].strip().upper()
if "CORRECT" in verdict: return "correct"
if "INCORRECT" in verdict: return "incorrect"
return "ambiguous"
def extract_relevant_strips(query: str, chunks: list[str],
model: str = "llama3.1") -> str:
"""Extract only the relevant portions from ambiguous chunks."""
context = "\n\n---\n\n".join(chunks)
response = ollama.chat(
model=model,
messages=[{"role": "user", "content":
f"Extract only the sentences or phrases from these passages that are "
f"directly relevant to this question. Remove irrelevant content.\n\n"
f"Question: {query}\n\nPassages:\n{context}\n\nRelevant extracts:"
}],
options={"temperature": 0.1, "num_predict": 500}
)
return response["message"]["content"]
def crag_retrieve_and_answer(query: str, collection,
web_search_fn=None) -> str:
"""Full CRAG pipeline with corrective retrieval."""
# Step 1: retrieve from vector store
results = collection.query(
query_embeddings=[embed(query)], n_results=5
)
chunks = results["documents"][0]
# Step 2: evaluate retrieval quality
verdict = evaluate_retrieval(query, chunks)
if verdict == "correct":
context = "\n\n---\n\n".join(chunks)
elif verdict == "incorrect":
# Retrieval failed: fall back to web search if available
if web_search_fn:
context = web_search_fn(query)
else:
context = "No relevant information found in the knowledge base."
else: # ambiguous
# Extract relevant portions and supplement
extracted = extract_relevant_strips(query, chunks)
supplement = web_search_fn(query) if web_search_fn else ""
context = extracted + ("\n\n" + supplement if supplement else "")
# Step 3: generate answer from (corrected) context
response = ollama.chat(
model="llama3.1",
messages=[{"role": "user", "content":
f"Answer based on the context below.\n\nContext:\n{context}\n\nQuestion: {query}"
}],
options={"temperature": 0.1}
)
return response["message"]["content"]Self-RAG: Teaching the Model to Decide When to Retrieve
Self-RAG (Asai et al., 2023) takes a different approach: rather than a separate evaluator model, Self-RAG fine-tunes the generative model itself to output special reflection tokens that signal whether retrieval is needed, whether retrieved documents are relevant, and whether the generated output is supported by the retrieved evidence. The model learns to be its own judge of retrieval quality. Self-RAG tokens include: [Retrieve] (should I retrieve?), [Relevant] / [Irrelevant] (is this document relevant?), [Fully supported] / [Partially supported] / [No support] (is my output supported by the document?), and [Utility: 1-5] (how useful is the retrieved content). Using a true Self-RAG model requires a fine-tuned version that understands these tokens. However, the Self-RAG logic can be approximated with standard LLMs using prompt engineering:
def self_rag_answer(query: str, collection, model: str = "llama3.1") -> dict:
"""Approximate Self-RAG using prompting with reflection steps."""
# Step 1: should we retrieve at all?
retrieve_decision = ollama.chat(
model=model,
messages=[{"role": "user", "content":
f"Does answering this question require looking up specific facts from a document,\n"
f"or can you answer from general knowledge? Question: {query}\n\n"
f"Reply with RETRIEVE or NO_RETRIEVE."
}],
options={"temperature": 0, "num_predict": 10}
)["message"]["content"].upper()
if "NO_RETRIEVE" in retrieve_decision:
# Answer from general knowledge
answer = ollama.chat(
model=model,
messages=[{"role": "user", "content": query}],
options={"temperature": 0.3}
)["message"]["content"]
return {"answer": answer, "retrieved": False, "supported": "N/A"}
# Step 2: retrieve
chunks = collection.query(
query_embeddings=[embed(query)], n_results=5
)["documents"][0]
context = "\n\n---\n\n".join(chunks)
# Step 3: generate answer with support check
response = ollama.chat(
model=model,
messages=[{"role": "user", "content":
f"Context:\n{context}\n\nQuestion: {query}\n\n"
f"Answer the question using the context. After your answer, on a new line "
f"write 'SUPPORT: FULL', 'SUPPORT: PARTIAL', or 'SUPPORT: NONE' "
f"to indicate how well the context supports your answer."
}],
options={"temperature": 0.1}
)["message"]["content"]
# Parse support level
lines = response.strip().split("\n")
support_line = next((l for l in reversed(lines) if l.startswith("SUPPORT:")), "SUPPORT: UNKNOWN")
answer_text = "\n".join(l for l in lines if not l.startswith("SUPPORT:"))
return {"answer": answer_text, "retrieved": True,
"supported": support_line.replace("SUPPORT:", "").strip()}