RAG evaluation: testing retrieval and grounding separately

A wrong answer from a document-grounded application has two possible causes: the right passage never reached the model, or the model misused the passage it had. This article covers the metrics and test design that separate the two. Research figures carry a numbered source; directory counts are computed from published records at build time.

Why retrieval and generation are scored separately

RAG combines a model's parametric memory with a non-parametric memory, typically a document index searched at query time [1]. The RAGAS authors list the dimensions to evaluate as the retriever's ability to find "relevant and focused context passages", the model's faithful use of them, and the quality of the generated answer [2]. One end-to-end score mixes these causes, and each cause has a different fix.

RAG: Retrieval-augmented generation: the model answers from documents fetched at query time.

Barnett et al. describe seven failure points from three case studies in research, education and biomedicine [13]. Two sit in retrieval: the answer is in a document that "did not rank highly enough to be returned", or a retrieved document did not make it into the context [13]. Four sit in generation: the answer is in the context but not extracted, the format is wrong, or the answer is incomplete or pitched at the wrong level of detail [13].

How recall@k, MRR and nDCG are defined

All three metrics need relevance labels: for each test query, the passages that contain the answer.

• Recall@k. NIST's trec_eval tool defines recall as "relevant retrieved / relevant", measured at a cutoff k in the ranking [5]. The dense passage retrieval paper uses a related measure, top-k retrieval accuracy: the share of questions for which the top k passages contain an answer [8].
• Mean reciprocal rank (MRR). Each query scores the reciprocal of the rank of the first correct result, or 0 if no correct result was returned, and MRR is the mean over queries [4]. The TREC-8 report notes that the measure gives no credit for retrieving more than one correct answer [4].
• nDCG@k. Discounted cumulative gain is a weighted sum of relevance grades, with a weight that decreases with rank; the logarithmic discount is the standard form [6]. Normalised DCG divides by the DCG of the ideal ranking, so it runs from 0 to 1 [6]. trec_eval computes it following Järvelin and Kekäläinen's 2002 definition [5].

Each metric answers a different question. Recall@k, measured at the k the application actually passes to the model, tells you whether the answer reached the model at all. MRR suits queries with a single relevant passage [5], while nDCG handles graded labels, which is why the BEIR benchmark reports nDCG@10 [7]. Order matters downstream too: model performance was often highest when relevant information appeared at the beginning or end of a long context [12].

Relevance labels are a proxy. Salemi and Zamani found that retrieval metrics based on query-document relevance labels show "a small correlation" with downstream RAG performance [9]. Their method gives the model each retrieved document on its own and uses the score of the resulting output as that document's relevance label [9].

How faithfulness and answer relevance are measured

In RAGAS, an answer is faithful when "the claims that are made in the answer can be inferred from the context" [2]. The score is the number of supported statements divided by the number of statements in the answer [2]. An unsupported statement is a hallucination relative to the retrieved context, even when it happens to be true in the world.

hallucination: Model output that is fluent but not supported by facts or sources.

Answer relevance in RAGAS deliberately ignores factuality and penalises answers that are incomplete or carry redundant information [2]. Context relevance is the share of context sentences needed to answer the question, which penalises padded retrieval [2]. On the WikiEval set, RAGAS agreed with human annotators in 95% of pairwise faithfulness comparisons, 78% for answer relevance and 70% for context relevance [2]. Those rates argue for human review of a sample, especially on the two relevance scores.

ARES scores the same three dimensions with small fine-tuned judges and corrects their errors with a small human-labelled set, using "only a few hundred human annotations during evaluation" [3]. The calibration steps for any model judge are set out in how to evaluate LLM applications.

How to check citation accuracy

Liu, Zhang and Liang define two checks. Citation recall asks whether every statement is fully supported by its citations; citation precision asks whether every citation supports its statement [10]. In their audit of four generative search engines, 51.5% of generated sentences were fully supported by citations and 74.5% of citations supported their sentence [10]. On one long-form dataset in the ALCE benchmark, the models with the highest scores still lacked complete citation support 50% of the time [11].

Score both directions per sentence. Also check that every cited identifier exists in the retrieved set for that query, since a citation to a document that was never retrieved cannot support anything. Report citation precision and recall separately: an application can cite few sources accurately or many sources loosely.

How chunking and index changes cause regressions

Chunking sets what a passage can contain. Barnett et al. report that "if chunks are too small certain questions cannot be answered, if the chunks are too long then the answers include generated noise" [13]. They also note that "changing the embedding strategy requires re-indexing all chunks" [13], so an embedding change rebuilds the whole index at once.

Treat each change to the chunker, embedding model, reranker, document set, top-k setting or query rewriter as a release candidate, in the same way as a prompt change in prompt and model regression testing. Run the retrieval suite before and after, and list the queries whose relevant passage dropped out of the top k. A rise in mean nDCG can hide a set of queries that lost their only relevant passage. Run the generation suite as well, because a reranker change also moves passages within the context, and position affects how well models use them [12].

How to build the test set

Start from real queries: search logs, support tickets and pilot transcripts. Freeze them with a snapshot of the document set as a versioned golden set, and label the relevant passages for each query, with grades where some passages answer fully and others only partly. Include questions the documents cannot answer, since missing content is a named failure point and the expected behaviour is a refusal [13].

golden set: A fixed set of inputs with approved outputs used for regression checks.

Labels built from one retriever's results undercount what other retrievers find. In BEIR, one dense retriever's nDCG@10 rose from 0.654 to 0.735 after annotators judged the first 10 results that had never been labelled [7]. Pool candidates from more than one retrieval method before labelling, and label new unjudged results when the retriever changes.

Synthetic questions generated from the documents add coverage, but a human-labelled sample is what anchors the scores [3]. Barnett et al. conclude that "validation of a RAG system is only feasible during operation" [13], so add sampled production queries to the set each cycle.

A RAG evaluation checklist

1. 01

   Label relevance

   For each test query, list the passages that answer it, with grades, pooled from more than one retrieval method.

2. 02

   Score retrieval

   Compute recall@k at the k passed to the model, plus MRR or nDCG@10, per query and as a mean.

3. 03

   Score faithfulness

   Split each answer into statements and record the share supported by the retrieved context.

4. 04

   Score relevance

   Score answer relevance and context relevance, and have humans review a sample of both.

5. 05

   Check citations

   Compute citation recall and citation precision per sentence and confirm every cited source was retrieved.

6. 06

   Test unanswerable questions

   Confirm the application declines when the documents hold no answer.

7. 07

   Diff on every index change

   Rerun both suites after any chunker, embedding, reranker, top-k or document set change and list queries that regressed.

8. 08

   Refresh from production

   Add sampled production queries and every fixed incident to the golden set.

Retrieved documents are also an attack surface, since they can carry instructions the model may follow; prompt injection and LLM security testing covers that layer. After release, quality checks continue as monitoring and drift work.

Which listed vendors state RAG pipeline validation

14 published entries state RAG pipeline validation for LLM applications on their AI page. The table counts entries for each related service.

Each count reports what a vendor's own AI page states; it is not a finding about delivery. The LLM applications page lists the entries, and the LLM application testing checklist places retrieval checks among the other pre-release checks.

Sources

[1] Lewis, P. et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." arXiv (NeurIPS 2020), 2020. https://arxiv.org/abs/2005.11401

[2] Es, S., James, J., Espinosa-Anke, L., Schockaert, S. "Ragas: Automated Evaluation of Retrieval Augmented Generation." arXiv (EACL 2024), 2023. https://arxiv.org/abs/2309.15217

[3] Saad-Falcon, J., Khattab, O., Potts, C., Zaharia, M. "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems." arXiv (NAACL 2024), 2023. https://arxiv.org/abs/2311.09476

[4] Voorhees, E. M. "The TREC-8 Question Answering Track Report." National Institute of Standards and Technology, 1999. https://trec.nist.gov/pubs/trec8/papers/qa_report.pdf

[5] National Institute of Standards and Technology. "trec_eval: measure definitions (recall, ndcg_cut, recip_rank)." GitHub, usnistgov/trec_eval, main branch, read 2026. https://github.com/usnistgov/trec_eval

[6] Wang, Y., Wang, L., Li, Y., He, D., Chen, W., Liu, T.-Y. "A Theoretical Analysis of NDCG Type Ranking Measures." arXiv (COLT 2013), 2013. https://arxiv.org/abs/1304.6480

[7] Thakur, N. et al. "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models." arXiv (NeurIPS 2021 Datasets and Benchmarks), 2021. https://arxiv.org/abs/2104.08663

[8] Karpukhin, V. et al. "Dense Passage Retrieval for Open-Domain Question Answering." arXiv (EMNLP 2020), 2020. https://arxiv.org/abs/2004.04906

[9] Salemi, A., Zamani, H. "Evaluating Retrieval Quality in Retrieval-Augmented Generation." arXiv (SIGIR 2024), 2024. https://arxiv.org/abs/2404.13781

[10] Liu, N. F., Zhang, T., Liang, P. "Evaluating Verifiability in Generative Search Engines." arXiv (Findings of EMNLP 2023), 2023. https://arxiv.org/abs/2304.09848

[11] Gao, T., Yen, H., Yu, J., Chen, D. "Enabling Large Language Models to Generate Text with Citations." arXiv (EMNLP 2023), 2023. https://arxiv.org/abs/2305.14627

[12] Liu, N. F. et al. "Lost in the Middle: How Language Models Use Long Contexts." arXiv (TACL 2024), 2023. https://arxiv.org/abs/2307.03172

[13] Barnett, S. et al. "Seven Failure Points When Engineering a Retrieval Augmented Generation System." arXiv (CAIN 2024), 2024. https://arxiv.org/abs/2401.05856