What Is LLM Hallucination?
Hallucination refers to LLM outputs that are factually incorrect, fabricated, or unsupported by the model’s input context — presented with the same confident tone as accurate information. The term covers a wide range of failure modes: citing research papers that do not exist, stating incorrect facts with high confidence, generating plausible-sounding but wrong statistics, and in RAG applications, producing answers that contradict or are not grounded in the retrieved documents. Hallucination is not random noise — it tends to cluster around specific conditions, which makes it partially predictable and partially preventable.
The appropriate response to hallucination risk depends heavily on your application’s stakes. A creative writing assistant can tolerate some factual looseness. A medical information system cannot. A legal document summariser where a wrong clause interpretation has contractual consequences requires near-zero hallucination tolerance. Understanding your application’s hallucination risk profile — what types of outputs it produces, what the consequences of errors are, and how often users will independently verify answers — should drive how much engineering investment you put into hallucination detection and mitigation.
Why Models Hallucinate
Understanding the mechanisms behind hallucination helps in designing better mitigations. The most common causes are: training data gaps (the model has seen little or no accurate information about the topic and extrapolates plausibly but wrongly), context-answer mismatch in RAG (the model generates from training knowledge rather than retrieved context), long-range dependencies in long context (the model loses track of constraints established early in a long prompt), and sycophancy (the model tells the user what they seem to want to hear rather than what is accurate). Each of these has different mitigation strategies.
Detecting Hallucination: The Three Approaches
LLM-as-judge evaluation uses a second model to check whether a response is supported by the source context. This is the most flexible approach and works for both RAG and non-RAG applications:
import anthropic, json
client = anthropic.Anthropic()
FAITHFULNESS_PROMPT = """You are evaluating whether an AI response is faithful to its source context.
Context provided to the AI:
{context}
AI Response:
{response}
Evaluate: Is every factual claim in the response directly supported by the context above?
Does the response avoid adding information not present in the context?
Return JSON only:
{{"faithful": true/false, "score": 0.0-1.0, "unsupported_claims": ["claim1", "claim2"], "explanation": "..."}}"""
def check_faithfulness(context: str, response: str) -> dict:
result = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=512,
messages=[{"role": "user", "content": FAITHFULNESS_PROMPT.format(
context=context, response=response
)}]
)
return json.loads(result.content[0].text)
result = check_faithfulness(
context="The company was founded in 2015 and has 200 employees.",
response="The company was founded in 2015 and has over 500 employees."
)
print(result)
# {"faithful": false, "score": 0.4, "unsupported_claims": ["over 500 employees"], ...}Self-Consistency Checking
Self-consistency exploits the observation that correct answers tend to be stable across multiple model runs while hallucinated answers tend to vary. Generate the same query multiple times at temperature > 0 and flag responses where the answers differ significantly:
from collections import Counter
def self_consistency_check(query: str, n_samples: int = 5, model: str = "claude-haiku-4-5-20251001") -> dict:
responses = []
for _ in range(n_samples):
resp = client.messages.create(
model=model, max_tokens=256, temperature=0.7,
messages=[{"role": "user", "content": query}]
)
responses.append(resp.content[0].text.strip())
# Check agreement — for factual questions, ask the judge to extract key claims
extract_prompt = f"Extract the single key factual claim from this response as a short phrase: {responses[0]}"
claims = []
for r in responses:
claim_resp = client.messages.create(
model=model, max_tokens=64,
messages=[{"role": "user", "content": f"Extract the main factual claim in 5 words or less: {r}"}]
)
claims.append(claim_resp.content[0].text.strip().lower())
most_common, count = Counter(claims).most_common(1)[0]
agreement_rate = count / n_samples
return {"agreement_rate": agreement_rate, "consensus": most_common,
"is_consistent": agreement_rate >= 0.6, "all_claims": claims}Self-consistency is particularly useful for factual Q&A where you can define what “agreement” means for the answer type. For yes/no questions or numeric answers, agreement is straightforward. For longer responses, you need to extract the key claims first. The main drawback is cost — running N samples per query multiplies API costs by N, so reserve self-consistency for high-stakes queries rather than applying it universally.
RAG-Specific Hallucination Detection
In RAG applications, the primary hallucination risk is the model generating from training knowledge rather than retrieved context. Three signals indicate this is happening. First, the response contains information not present in any retrieved chunk. Second, the response contradicts information in the retrieved chunks. Third, the response is confident about specific details (dates, numbers, names) that were absent from the context. LlamaIndex and LangChain’s evaluation modules both include faithfulness evaluators that check for these patterns automatically:
from llama_index.core.evaluation import FaithfulnessEvaluator, BatchEvalRunner
evaluator = FaithfulnessEvaluator(llm=Settings.llm)
# Evaluate a single response
response = query_engine.query("What were our Q3 revenues?")
result = evaluator.evaluate_response(response=response)
print(f"Faithful: {result.passing} | Score: {result.score}")
print(f"Feedback: {result.feedback}")
# Batch evaluation for monitoring
runner = BatchEvalRunner({"faithfulness": evaluator}, workers=8)
eval_results = await runner.aevaluate_queries(
query_engine,
queries=test_questions # list of production query samples
)
# Aggregate scores
faithfulness_scores = [r.score for r in eval_results["faithfulness"]]
print(f"Mean faithfulness: {sum(faithfulness_scores)/len(faithfulness_scores):.2f}")For production RAG systems, run faithfulness evaluation on a 5–10% sample of live queries and track the mean score over time. A falling faithfulness score typically indicates one of three things: the document corpus has developed gaps relative to what users are asking about (retrieval is returning less relevant chunks), the system prompt has changed in a way that reduces context grounding, or query complexity has increased beyond what the current chunking strategy handles well.
Reducing Hallucination: Prompt Engineering
Prompt design is the first line of defence against hallucination and costs nothing to implement. Several prompt techniques reliably reduce hallucination rates:
Explicit uncertainty instructions. Telling the model to say “I don’t know” rather than guessing significantly reduces confident hallucination. Add to your system prompt: “If the answer is not clearly supported by the provided context, say ‘I don’t have enough information to answer that’ rather than guessing.”
Source citation requirements. Requiring citations forces the model to anchor each claim to a specific source chunk. Claims without a citable source are implicitly flagged. “For every factual statement, cite the specific document and section it comes from in brackets.”
Chain-of-thought verification. Asking the model to verify its reasoning before committing to an answer — “Before answering, check whether each key fact in your response appears in the context. If any fact does not, remove it or note that it cannot be verified from the provided materials.” — significantly reduces hallucination on structured reasoning tasks.
Scope constraints. Explicitly delimiting what the model should answer using only the provided context, and what it should refuse. “Answer only using the documents provided. Do not use your general knowledge to supplement information that is absent from the documents.”
Reducing Hallucination: Model Selection and Temperature
Model choice significantly affects hallucination rates. Models with better instruction-following tend to hallucinate less on RAG tasks because they more reliably follow the “answer only from context” instruction. Claude models are generally noted for strong instruction adherence and good calibration (expressing uncertainty when uncertain). GPT-4o has excellent grounding in structured contexts. Smaller models (7B–13B) hallucinate more frequently than frontier models, particularly on complex or ambiguous queries — if hallucination rate is critical, this is one of the strongest arguments for using a larger model.
Temperature should be zero or near-zero for factual RAG applications. Temperature above 0 introduces randomness that manifests as creative but inaccurate responses. Set temperature to 0 for any application where factual accuracy matters and creative variation is unwanted. Reserve temperature > 0 for explicitly creative or brainstorming use cases.
Hallucination Guardrails in Production
For high-stakes applications, implement a response validation layer that blocks low-confidence or unfaithful responses before they reach the user:
from dataclasses import dataclass
@dataclass
class ValidatedResponse:
response: str
faithfulness_score: float
passed_validation: bool
fallback_message: str = ""
def validated_rag_response(query: str, min_faithfulness: float = 0.80) -> ValidatedResponse:
rag_response = query_engine.query(query)
retrieved_context = "
".join([n.text for n in rag_response.source_nodes])
faith_result = check_faithfulness(
context=retrieved_context,
response=rag_response.response
)
if faith_result["score"] >= min_faithfulness:
return ValidatedResponse(
response=rag_response.response,
faithfulness_score=faith_result["score"],
passed_validation=True
)
else:
return ValidatedResponse(
response="",
faithfulness_score=faith_result["score"],
passed_validation=False,
fallback_message="I found some information that may be relevant, but I am not confident enough in its accuracy to share it directly. Please consult the source documents or speak with a subject matter expert."
)The fallback message matters — it should be honest about uncertainty without being so vague that it is useless. Tune the faithfulness threshold based on your application’s stakes: 0.70 for general information, 0.85 for financial or legal content, 0.95 for safety-critical applications.
Measuring Hallucination in Production
Track hallucination as a first-class production metric alongside latency and cost. The key metrics are: faithfulness rate (what fraction of responses pass your faithfulness check), unfaithful claim types (what categories of information does the model most commonly hallucinate — useful for improving the RAG corpus to cover gaps), and hallucination-triggered fallback rate (how often does the validation layer block a response — a high rate indicates the RAG retrieval quality needs improvement). Alert when any of these metrics move significantly from their baseline — a sudden increase in hallucination rate often indicates a retrieval problem or a prompt change that reduced context grounding. Treating hallucination as an operational metric rather than an inherent property of LLMs is the shift in mindset that enables systematic improvement over time.
Retrieval Quality as a Hallucination Root Cause
A frequently underappreciated source of RAG hallucination is poor retrieval rather than poor generation. If the retrieved chunks do not contain the information needed to answer the query, the LLM must choose between saying “I don’t know” and generating from training knowledge — and many models default to the latter even when instructed otherwise. Improving retrieval quality is often more effective at reducing hallucination than improving the generation prompt.
Measure retrieval quality separately from generation quality. For a set of test questions with known answers, check whether the answer appears in the top-K retrieved chunks before the LLM even sees them. If the answer is absent from the retrieved context, no amount of generation-side hallucination mitigation will help — the information simply is not there. Common retrieval quality improvements: reduce chunk size to increase semantic specificity, use hybrid search (vector + BM25) for queries with specific identifiers, add more documents to cover gaps in the corpus, and tune the embedding model to your domain if retrieval relevance is consistently poor on domain-specific terminology.
Long-Context Hallucination: The Lost-in-the-Middle Problem
Research has demonstrated that LLMs perform significantly worse at using information in the middle of long contexts compared to information at the beginning or end — the “lost-in-the-middle” problem. In RAG applications with many retrieved chunks, relevant information in chunks 4–8 of a 10-chunk context is more likely to be ignored or misremembered than information in chunks 1–2 or 9–10. Practical mitigations: rerank retrieved chunks by relevance to the query before inserting them into context (putting the most relevant first and last), reduce the number of chunks retrieved to the minimum that consistently contains the answer, and use reranking models (Cohere Rerank, cross-encoder rerankers) that better identify the single most relevant chunk rather than relying solely on embedding similarity for ordering.
Hallucination in Agentic Systems
Agentic systems that take real-world actions introduce a qualitatively different hallucination risk. An agent that hallucinates a database query can corrupt data. An agent that hallucinates a tool call argument can trigger an unintended API action. An agent that hallucinates progress on a multi-step task can skip steps that were never actually completed. For agentic systems, hallucination detection must happen at the action level, not just the response level. Verify tool call arguments against schemas before executing them. Log the state of the world before and after each action and verify it matches the agent’s internal model. Require explicit human confirmation for any irreversible action where the agent’s grounding cannot be independently verified. The cost of hallucination in agentic systems is higher than in conversational systems — a wrong answer can be corrected, a deleted database record or sent email cannot be.
The Honest Answer on Hallucination
No current technique eliminates hallucination entirely. Retrieval augmentation reduces it significantly compared to open-ended generation. Faithfulness evaluation and response gating reduce the rate of unfaithful responses reaching users. Better prompting reduces the rate at which models confabulate. Model selection reduces the baseline rate for a given task type. But some fraction of responses will still contain inaccuracies — the question is whether that fraction is acceptable for your use case and whether you have the detection and escalation mechanisms to catch the ones that matter. Building users’ appropriate calibration — helping them understand that LLM outputs should be verified for high-stakes decisions rather than trusted uncritically — is as important as the technical mitigations. A system with 98% faithfulness and users who verify important outputs is safer than a system with 99.5% faithfulness and users who treat every output as authoritative.