Evaluate and Debug RAG Pipelines with Ragas

What you'll build

A small evaluation harness that scores your RAG pipeline's outputs on three Ragas metrics (faithfulness, context precision, answer relevancy) and tells you whether a bad answer came from the retriever or the generator.

Prerequisites

• Python 3.10–3.13. Verified on 3.13; Ragas declares support for 3.9+.
• Ragas 0.4.3 and openai 1.109.1, the current stable releases this was tested against. The code below uses the ragas.metrics.collections API introduced in the 0.3/0.4 line. It will not run on ragas 0.2.x.
• langchain-community pinned below 0.4 (see step 1; ragas 0.4.3 breaks with 0.4.x).
• An OpenAI API key with a small amount of credit. The judge calls in this tutorial use gpt-4o-mini and text-embedding-3-small; a full run costs about a cent.
• Commands are for macOS/Linux. On Windows, activate the venv with .venv\Scripts\activate instead.

A note on how this works: these metrics are LLM-judged. Ragas sends your question, contexts, and answer to a judge model that extracts claims and checks them. That's why you need an API key even though you're evaluating outputs you already have.

1. Set up the project

mkdir rag-eval && cd rag-eval
python3 -m venv .venv && source .venv/bin/activate
pip install ragas==0.4.3 openai "langchain-community<0.4"
export OPENAI_API_KEY="sk-..."

The langchain-community<0.4 pin matters. Ragas 0.4.3 imports a module that langchain-community 0.4.x removed, and without the pin the install succeeds but every import fails (see Troubleshooting).

2. Capture pipeline outputs as an eval dataset

Ragas evaluates records with four fields: user_input (the question), retrieved_contexts (the chunks your retriever returned), response (what your generator produced), and reference (a known-good answer, needed only for context precision). In a real app you'd log these from your pipeline; here we hand-craft three samples with known failure modes so you can see each metric react.

Save as samples.json:

[
  {
    "id": "good",
    "user_input": "What port does PostgreSQL listen on by default?",
    "retrieved_contexts": [
      "PostgreSQL listens on TCP port 5432 by default. The port is set by the port parameter in postgresql.conf.",
      "To change the listen address, edit the listen_addresses parameter in postgresql.conf and restart the server."
    ],
    "response": "PostgreSQL listens on port 5432 by default. You can change it via the port parameter in postgresql.conf.",
    "reference": "PostgreSQL's default port is 5432, configured by the port parameter in postgresql.conf."
  },
  {
    "id": "hallucinated-answer",
    "user_input": "How do I enable WAL archiving in PostgreSQL?",
    "retrieved_contexts": [
      "To enable WAL archiving, set wal_level to replica or higher and set archive_mode to on in postgresql.conf.",
      "The archive_command parameter defines the shell command used to copy a completed WAL segment to archive storage."
    ],
    "response": "Set wal_level to replica and archive_mode to on in postgresql.conf. Note that archive_mode defaults to on since PostgreSQL 16, so most installations already archive WAL automatically.",
    "reference": "Enable WAL archiving by setting wal_level to replica or higher, archive_mode to on, and configuring archive_command in postgresql.conf."
  },
  {
    "id": "bad-retrieval",
    "user_input": "What is the maximum identifier length in PostgreSQL?",
    "retrieved_contexts": [
      "VACUUM reclaims storage occupied by dead tuples and is run automatically by the autovacuum daemon.",
      "The pg_dump utility creates logical backups of a single PostgreSQL database in script or archive formats."
    ],
    "response": "PostgreSQL identifiers are limited to 63 bytes by default, set by the NAMEDATALEN constant at compile time.",
    "reference": "The maximum identifier length in PostgreSQL is 63 bytes, determined by NAMEDATALEN minus one."
  }
]

Sample two retrieves the right chunks but the generator invents a claim (archive_mode does not default to on). Sample three answers correctly from the model's own knowledge while the retriever returned junk, a failure that stays invisible until the model gets a question it can't answer from memory.

3. Write the scoring script

Save as eval_rag.py:

import asyncio
import json

from openai import AsyncOpenAI
from ragas.embeddings.base import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import AnswerRelevancy, ContextPrecision, Faithfulness

with open("samples.json") as f:
    samples = json.load(f)


async def main():
    client = AsyncOpenAI()  # reads OPENAI_API_KEY
    llm = llm_factory("gpt-4o-mini", client=client)
    embeddings = embedding_factory(
        "openai", model="text-embedding-3-small", client=client
    )

    faithfulness = Faithfulness(llm=llm)
    context_precision = ContextPrecision(llm=llm)
    answer_relevancy = AnswerRelevancy(llm=llm, embeddings=embeddings)

    print(f"{'sample':<22}{'faithfulness':>14}{'ctx_precision':>15}{'relevancy':>11}")
    for s in samples:
        faith, precision, relevancy = await asyncio.gather(
            faithfulness.ascore(
                user_input=s["user_input"],
                response=s["response"],
                retrieved_contexts=s["retrieved_contexts"],
            ),
            context_precision.ascore(
                user_input=s["user_input"],
                reference=s["reference"],
                retrieved_contexts=s["retrieved_contexts"],
            ),
            answer_relevancy.ascore(
                user_input=s["user_input"],
                response=s["response"],
            ),
        )
        print(
            f"{s['id']:<22}{faith.value:>14.2f}"
            f"{precision.value:>15.2f}{relevancy.value:>11.2f}"
        )


asyncio.run(main())

Each metric only sees the fields it needs, which is the point: faithfulness checks the response against the retrieved contexts, context precision checks the contexts against the reference, and answer relevancy ignores the contexts entirely. The three metrics for one sample run concurrently via asyncio.gather since they're independent API calls.

4. Read the scores like a debugger

Each metric isolates one component, so a low score points at a specific fix:

• Low faithfulness, high context precision: the generator is inventing claims the context doesn't support. Tighten the prompt ("answer only from the provided context") or use a stronger generation model.
• Low context precision: the retriever is the problem. Look at chunk size, embedding model, top-k, or add a reranker. Faithfulness may still be high if the model faithfully summarizes the wrong chunks.
• Low answer relevancy: the response dodges the question. Usually a prompt problem (over-long boilerplate answers) or the query needs rewriting before retrieval.

High relevancy with low faithfulness and low precision is the dangerous quadrant: confident, on-topic answers built from nothing.

Verify it works

python eval_rag.py

Expected output (judge-based scores wobble a few points between runs; the pattern is what you're checking):

sample                  faithfulness  ctx_precision  relevancy
good                            1.00           1.00       0.97
hallucinated-answer             0.67           1.00       0.93
bad-retrieval                   0.00           0.00       0.91

The run takes 20–60 seconds. Each sample's scores match its planted failure: the hallucinated answer loses a third of its faithfulness score (one unsupported claim out of three), and the bad-retrieval sample shows 0.00 precision while relevancy stays high because the answer itself reads fine.

Troubleshooting

ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai' on any ragas import. Ragas 0.4.3 depends on langchain-community without an upper bound, and 0.4.x removed this module. Fix: pip install "langchain-community<0.4" (resolves to 0.3.31).

OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable. The key isn't in the environment of the shell running the script. Re-run export OPENAI_API_KEY="sk-..." in the same terminal, or pass AsyncOpenAI(api_key=...) explicitly.

ValueError: Collections metrics only support modern embeddings. Found: LangchainEmbeddingsWrapper. You called the deprecated no-argument embedding_factory() from an older tutorial. Metrics from ragas.metrics.collections require the modern interface: embedding_factory("openai", model="text-embedding-3-small", client=client).

TypeError: AnswerRelevancy.__init__() missing 1 required positional argument: 'embeddings'. Unlike the other two metrics, AnswerRelevancy embeds generated questions to compare against the original, so it needs both llm= and embeddings=.

Next steps

Add ContextRecall and FactualCorrectness (both in ragas.metrics.collections, both need reference) to catch retrievers that miss chunks rather than fetch wrong ones. Replace the hand-written samples with real traces logged from your pipeline, or generate a synthetic test set from your documents with ragas.testset.TestsetGenerator. Once scores are stable, wire eval_rag.py into CI and fail the build when faithfulness drops below your baseline, so a prompt tweak can't silently reintroduce hallucinations.