How to Evaluate a RAG Pipeline: RAGAS, the RAG Triad, and a Production Checklist

by

Building a RAG pipeline is straightforward. Knowing whether it actually works — and what specifically is broken when it does not — requires systematic evaluation. RAG systems can fail in at least four distinct ways: irrelevant retrieval, faithfulness failures where the LLM ignores the retrieved context, knowledge gaps where the right context is missing, and presentation failures where the answer is correct but unclear. Standard end-to-end accuracy metrics collapse all four into a single number, making it impossible to know which component to fix. This guide covers three frameworks that diagnose each failure mode separately: RAGAS, the RAG Triad, and a production readiness checklist.

RAGAS: Automated RAG Evaluation

RAGAS (RAG Assessment) is an open-source library that evaluates RAG pipelines using an LLM judge. It measures four core metrics: faithfulness (does the answer stay within the retrieved context?), answer relevancy (does the answer address the question?), context precision (are the most relevant chunks ranked first?), and context recall (did retrieval find all the information needed to answer fully?). You do not need human annotators for every run — though a ground truth set is required to compute recall and answer correctness.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

eval_data = {
    "question": ["What is the context window of Llama 3.1?"],
    "answer": ["Llama 3.1 supports a 128k token context window."],
    "contexts": [["Llama 3.1 has a context window of 128,000 tokens..."]],
    "ground_truth": ["128k tokens"],
}

results = evaluate(
    Dataset.from_dict(eval_data),
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(results)
# {'faithfulness': 0.95, 'answer_relevancy': 0.88, 'context_precision': 0.82, 'context_recall': 0.79}

Running RAGAS with a Local LLM Judge

RAGAS defaults to OpenAI as the evaluator. Swap in a local Ollama model to keep evaluation free and private — use at least a 13B model for reliable scoring, since smaller models produce inconsistent verdicts on nuanced faithfulness checks.

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_community.llms import Ollama
from langchain_community.embeddings import OllamaEmbeddings

local_llm = LangchainLLMWrapper(Ollama(model="llama3.1:8b"))
local_embeddings = LangchainEmbeddingsWrapper(OllamaEmbeddings(model="nomic-embed-text"))

results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    llm=local_llm,
    embeddings=local_embeddings,
)

Interpreting RAGAS Scores

RAGAS scores run from 0 to 1. Here are rough production-readiness thresholds and what low scores mean in practice:

  • Faithfulness below 0.75 — the LLM is generating from parametric memory rather than retrieved context. Tighten the system prompt: explicitly instruct the model to only use provided context and to refuse if the context is insufficient.
  • Context precision below 0.60 — the most relevant chunks are not ranked first. Fix with cross-encoder reranking or a better embedding model.
  • Context recall below 0.60 — retrieval is missing relevant documents entirely. Increase K, improve chunking, or switch to a domain-specific embedding model.
  • Answer relevancy below 0.70 — the model is responding to something slightly different from the actual question. Add query transformation or clarification steps before retrieval.

These thresholds are guidelines, not hard rules. Medical and legal applications need faithfulness above 0.95; creative or exploratory Q&A can tolerate lower scores. Always validate RAGAS scores against human judgment on a sample of your specific queries — LLM judges can be miscalibrated on domain-specific content that differs from their training distribution.

The RAG Triad

The RAG Triad, popularised by TruLens, distils evaluation to three binary-ish questions. A good RAG response must pass all three:

  • Context Relevance — is the retrieved context relevant to the question?
  • Groundedness — is the answer grounded in the retrieved context?
  • Answer Relevance — does the answer actually address the question?

The triad’s value is in how it isolates failures. A response can be grounded and answer-relevant but based on irrelevant context — a retrieval failure. Or relevant and context-appropriate but not grounded — a faithfulness failure. End-to-end accuracy misses both; the triad catches them separately. Use it as a quick mental checklist when manually reviewing outputs, and as the basis for automated checks when you want a simpler scoring model than the full RAGAS suite.

Figure 1 — The RAG Triad: a good response must pass all three checks

RAG Triad — All Three Must Pass Context Relevance Ground- edness Answer Relevance

What to Evaluate First

If you are just starting to evaluate a RAG pipeline, the order matters. Focus on faithfulness first — it is the most critical metric and the easiest to improve with prompt changes. Once faithfulness is above 0.85, measure context recall — low recall means your retriever is missing relevant documents, which is the hardest failure to fix and usually requires improving chunking or the embedding model. Context precision and answer relevancy are refinements on top of a working foundation. This sequencing tracks how most RAG pipelines actually fail: generation goes off-piste first, then retrieval gaps emerge as quality expectations rise.

Production RAG Checklist

Automated metrics catch quality failures but miss operational and reliability issues. Run through these before declaring a pipeline production-ready:

Figure 2 — Production RAG readiness: retrieval, generation, and operations

RETRIEVAL ✓Chunk size evaluated (128–512 tokens) and tuned for your content type ✓Hybrid search (BM25 + vector) tested against vector-only baseline ✓Context recall measured: does top-K actually contain the answer? ✓Out-of-scope queries handled gracefully (“I don’t know” path) GENERATION ✓Faithfulness score above 0.85 on a labelled eval set ✓System prompt explicitly says “only use provided context” ✓Temperature set to 0 or low for factual Q&A ✓Hallucination rate monitored on a held-out test set OPERATIONS ✓Latency measured at P50 and P95 under realistic concurrent load ✓Document index refresh process defined, tested, and scheduled ✓Query logs captured for ongoing evaluation sampling ✓Regression tests run on every document collection update

Building an Evaluation Harness

Rather than evaluating interactively, build a small offline harness that runs a fixed query set and writes scores to a file. This lets you compare pipeline versions, catch regressions, and track quality trends without manual review on every change.

import json, datetime
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset

def run_evaluation(pipeline_fn, test_cases: list[dict], run_name: str) -> dict:
    questions, answers, contexts_list = [], [], []
    for case in test_cases:
        answer, contexts = pipeline_fn(case["question"])
        questions.append(case["question"])
        answers.append(answer)
        contexts_list.append(contexts)

    dataset = Dataset.from_dict({
        "question": questions, "answer": answers, "contexts": contexts_list,
        "ground_truth":  for c in test_cases],
    })
    results = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])
    output = {"run_name": run_name, "timestamp": datetime.datetime.now().isoformat(),
              "scores": dict(results), "n_cases": len(test_cases)}
    with open(f"eval_{run_name}.json", "w") as f:
        json.dump(output, f, indent=2)
    return output

Keep the test set small and fast: 20-50 queries covering your most common patterns and known edge cases, runnable in under five minutes. Store results in version control so you can diff scores across commits. If faithfulness drops more than five points from baseline, investigate before deploying.

Continuous Evaluation in Production

Offline tests on a fixed set are necessary but not sufficient. Production queries differ from your test set in ways you cannot fully anticipate. Build a lightweight online evaluation loop alongside your harness:

  • Sample 1–5% of production queries randomly
  • Run RAGAS faithfulness and answer relevancy on the sampled responses
  • Alert if rolling 7-day scores drop below your thresholds
  • Store query logs, retrieved contexts, and generated answers in a database — this creates a growing pool of real examples to label and feed back into your offline test set

Add your RAGAS evaluation script to CI so it runs automatically on every pull request touching retrieval or generation. Set pass/fail thresholds — PR fails if faithfulness drops below 0.80 or context recall drops more than 10 points from baseline. Connect the online sampling evaluation to your alerting system so the on-call engineer is notified immediately if production scores fall. This turns RAG quality from a periodic concern into a continuous engineering practice.

When Automated Metrics Are Not Enough

RAGAS is scalable but imperfect. LLM judges can be miscalibrated on domain-specific jargon, may score a confidently wrong but fluent answer highly, and can be inconsistent across runs. For high-stakes applications — legal, medical, financial — supplement automated scoring with periodic human review. Develop a simple rubric: relevance (1-5), faithfulness (1-5), completeness (1-5), clarity (1-5). Reviewing 20-30 queries per week against this rubric builds a ground truth signal that automated metrics cannot replace, and helps you detect when RAGAS scores have drifted away from what actually matters to your users.

Setting Up a Regression Test Suite

Every RAG pipeline change risks breaking something that was previously working. A regression test suite — a fixed set of queries with expected answers that runs before every deployment — catches these regressions before they reach users. Keep it small and fast: 20-50 queries covering the most common patterns and known edge cases, runnable in under five minutes. Store results in version control alongside your pipeline code so you can compare scores across commits. If a score drops significantly — more than five points on any RAGAS metric — investigate before deploying. Even a one-line prompt change can tank faithfulness on specific query types, and the regression suite is the only way to catch that before it affects real users.

Pair the regression suite with a changelog. Every time you make a retrieval or generation change, record what changed, why, and what the before/after RAGAS scores were. After a few months, this log becomes genuinely useful: you can see which changes actually improved quality, which looked good on paper but hurt specific metrics, and which query types have been chronically underserved. It turns evaluation from a gate into a learning system.

RAGAS, the RAG Triad, and a production checklist are complementary — not redundant. RAGAS quantifies the four core failure modes automatically. The Triad gives you a clear mental model for diagnosing which component failed. The checklist catches operational gaps that automated metrics miss entirely. Run all three before declaring a pipeline production-ready, keep running RAGAS offline after every significant change, and treat the production sampling loop as a first-class engineering concern rather than an afterthought. That combination gives you reliable, measurable quality from day one through the full life of the pipeline.

Leave a Comment