Why LLM Evaluation Is Hard
Evaluating a language model is fundamentally different from evaluating a traditional software system. A classifier has a ground-truth label for every input — you measure accuracy against it. An LLM can produce dozens of valid responses to the same prompt, making “correct” a genuinely ambiguous concept. How do you measure whether a response is helpful, factually grounded, appropriately concise, and on-brand — simultaneously, at scale?
The gap between “the model sometimes gives good answers in a demo” and “the model reliably does what I need in production” is an evaluation problem. Teams that skip rigorous evaluation ship regressions they do not detect, over-index on anecdotal impressions, and make model or prompt changes without knowing whether they helped or hurt. Evaluation is the discipline that closes that gap.
The Three Layers of LLM Evaluation
A complete evaluation strategy operates at three levels. Benchmark evaluation runs the model against standardised datasets — MMLU, HellaSwag, TruthfulQA, HumanEval — to establish baseline capability. This is useful for comparing models but tells you little about how a model will perform on your specific task. Task-specific evaluation tests the model on examples drawn from your actual use case, with metrics calibrated to what matters for your application. This is where most of the practical value is. Production monitoring tracks model behaviour on live traffic — response quality, latency, error rates, user feedback signals — and alerts you when something degrades. All three layers are necessary; no single one is sufficient.
Evaluation Metrics: Choosing What to Measure
Exact match and F1 work for extractive tasks where the answer is a specific span of text. They are too strict for open-ended generation. BLEU and ROUGE measure n-gram overlap between generated text and reference answers. They correlate poorly with human judgement for most modern LLM tasks and should generally be avoided unless you are specifically working on translation or summarisation where they were designed. Semantic similarity embeds both the generated response and a reference answer, then measures cosine similarity — more robust than n-gram metrics because it captures meaning rather than surface form. LLM-as-judge uses a capable model to evaluate responses against a rubric. It scales to open-ended tasks where n-gram metrics fail, and when calibrated against human labels it can match human inter-annotator agreement on many tasks. It is the dominant approach for evaluating open-ended generation in 2026. Task-specific metrics — retrieval precision and recall for RAG systems, code execution pass rate for code generation, tool call accuracy for agents — are often the most informative of all because they measure what you actually care about rather than a proxy.
LLM-as-Judge: Implementation and Pitfalls
LLM-as-judge evaluations work by prompting a capable model with the input, the generated response, and a scoring rubric, then asking it to rate the response:
import anthropic, json
client = anthropic.Anthropic()
JUDGE_PROMPT = """You are evaluating an AI assistant's response.
User question: {question}
Assistant response: {response}
Reference answer: {reference}
Score the response on these dimensions (1-5 each):
1. Accuracy: Does it correctly answer the question?
2. Completeness: Does it cover all key points?
3. Clarity: Is it clearly written and easy to understand?
4. Groundedness: Are claims supported by the reference material?
Respond with JSON only:
{{"accuracy": N, "completeness": N, "clarity": N, "groundedness": N, "reasoning": "brief explanation"}}"""
def judge_response(question: str, response: str, reference: str) -> dict:
result = client.messages.create(
model="claude-opus-4-6",
max_tokens=256,
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
question=question, response=response, reference=reference
)}]
)
return json.loads(result.content[0].text)Known pitfalls to account for: LLM judges have position bias (preferring the first of two options when doing pairwise comparisons), verbosity bias (preferring longer responses even when shorter ones are better), and self-enhancement bias (models rating outputs from similar models more favourably). Mitigate these by randomising response order in pairwise comparisons, including a brevity criterion in your rubric, and validating judge scores against human labels on a sample of your eval set before trusting them at scale.
Evaluation Frameworks: The Main Options
Ragas is the most widely used framework for evaluating RAG pipelines. It provides metrics for faithfulness (are claims grounded in the retrieved context?), answer relevancy (does the response address the question?), context precision (are the retrieved chunks actually relevant?), and context recall (did retrieval surface the right content?). All metrics are automated using LLM-as-judge under the hood, making them applicable without human-labelled ground truth:
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset
data = {
"question": ["What is the capital of France?"],
"answer": ["Paris is the capital of France."],
"contexts": [["France is a country in Western Europe. Its capital is Paris."]],
"ground_truth": ["Paris"]
}
dataset = Dataset.from_dict(data)
results = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])
print(results)DeepEval provides a broader set of metrics beyond RAG — including G-Eval (a customisable LLM-as-judge framework), hallucination detection, toxicity scoring, and bias detection. It integrates with pytest, making it easy to add LLM evaluations to your existing CI/CD pipeline so failing evals block deployment automatically.
LangSmith from LangChain is a platform that captures traces of every LLM call in your application, lets you annotate and evaluate them, and provides dashboards for tracking quality metrics over time. It is particularly useful for teams that want evaluation integrated with their observability infrastructure rather than as a separate offline process.
Promptfoo is a CLI-first evaluation tool that runs test cases against prompt templates and compares outputs across model versions. It is particularly useful for prompt regression testing — ensuring that a prompt change does not break existing cases while fixing the target issue.
Building an Eval Dataset
The quality of your evaluation is only as good as your eval dataset. Deliberately include cases where the system has failed in the past, edge cases at the boundary of your use case, and adversarial inputs designed to probe specific weaknesses. A representative eval set should feel slightly hard — if your model aces it easily, it is not discriminating enough to be useful.
Mix easy cases (where any reasonable approach should succeed) with hard ones (where only a well-optimised system will). Easy cases detect catastrophic regressions. Hard cases measure genuine improvements. Get real data from production wherever possible — synthetic eval sets generated by asking an LLM to create test cases tend to be easier and less diverse than real user queries. Even 100–200 carefully labelled examples from real usage are more valuable than 1,000 synthetically generated ones.
Version your eval set alongside your model and prompt configurations. When you update the eval set — adding new cases, fixing labelling errors — record what changed and why. This makes it possible to understand whether a performance change is due to a model improvement or a change in what you are measuring.
Continuous Evaluation in CI/CD
The most impactful shift in LLM evaluation practice is moving from periodic manual review to automated continuous evaluation integrated with your deployment pipeline. Every pull request that changes a prompt, updates a model version, or modifies retrieval logic should trigger an eval run. The results gate deployment — if answer relevancy drops by more than 5% or faithfulness falls below threshold, the change does not ship without human review.
This requires fast evals. A full eval suite that takes 30 minutes to run will be skipped in practice. Design your CI eval suite to run a small, high-coverage set of 50–100 cases in under five minutes, and run the comprehensive suite less frequently — nightly or on release candidates. The fast suite catches obvious regressions; the comprehensive suite catches subtle degradations that only show up at scale.
Evaluating Agentic Systems
Evaluating agents is harder than evaluating single-turn completions because you need to assess multi-step behaviour, not just the final output. Did the agent choose the right tools? Did it use them in the right order? Did it correctly interpret intermediate results? Did it know when to stop?
Three evaluation approaches work for agents. Trajectory evaluation records the full sequence of tool calls and intermediate states, then evaluates whether the path taken was reasonable — not just whether the final answer was correct. Outcome evaluation checks the end result against a ground truth, ignoring the path — useful when multiple valid paths exist. Simulation-based evaluation runs the agent against a simulated environment where tool responses are controlled, making it possible to test specific scenarios deterministically without depending on live external systems.
For most production agent evaluations, a combination of outcome evaluation for basic correctness and trajectory spot-checking for debugging is a practical starting point. Full trajectory evaluation at scale is expensive and best reserved for critical paths or after detecting a regression in outcome metrics.
Treating Evaluation as Engineering
Treating evaluation as a first-class engineering concern — with the same rigour you apply to unit testing, performance benchmarking, or security scanning — is the difference between teams that ship reliable LLM features and those that are perpetually firefighting quality issues in production. The infrastructure investment is real but modest: a versioned eval dataset, a small set of automated metrics, and a CI integration. The payoff is the ability to iterate on models, prompts, and retrieval logic with confidence rather than anxiety, knowing that regressions will be caught before they reach users.
Human Evaluation: When and How to Use It
Automated metrics are fast and cheap but imperfect. Human evaluation is slow and expensive but remains the gold standard for assessing whether a model is actually useful to real people. The practical approach is to use automated metrics for continuous monitoring and regression detection, and human evaluation for high-stakes decisions: choosing between two substantially different models, validating a new task type before launch, or investigating a complex failure pattern that automated metrics are not capturing.
When running human evaluation, a few practices make results more reliable. Use relative preference judgements — “which response do you prefer?” — rather than absolute quality scores, since people are much more consistent at comparing two things than rating one thing on an absolute scale. Blind evaluators to which model or prompt produced each response to eliminate bias. Use multiple evaluators per example and measure inter-annotator agreement — low agreement is a signal that your evaluation task is not well-defined, not that the model is ambiguous. Define clear evaluation criteria in advance and give evaluators concrete examples of what each rating level looks like, rather than leaving them to interpret the rubric independently.
Connecting Evaluation to Business Outcomes
Evaluation metrics are proxies for what you ultimately care about: does the application create value for users and the business? The most important validation step — often skipped — is confirming that your automated metrics actually correlate with the outcomes that matter. A model that scores well on faithfulness and answer relevancy should also produce fewer user complaints, higher task completion rates, and better retention than one that scores poorly. If you cannot establish that correlation, you may be optimising for metrics that do not reflect real-world quality.
Instrument your application to capture implicit user signals alongside explicit evaluation: time spent reading a response, whether the user asked a follow-up question, whether they gave a thumbs up or thumbs down, whether they completed the task they came to do. These signals are noisier than structured evaluation but reflect actual user experience at production scale. Over time, they let you validate that your evaluation metrics are pointing in the right direction and catch the occasional case where a high-scoring model is somehow still producing poor user experiences in ways your eval set does not capture.
The teams building the most reliable LLM applications in production are not the ones with the best models — they are the ones with the most rigorous evaluation culture. Start small, instrument everything, and let data drive your decisions rather than intuition about what the model can or cannot do.