What is retrieval evaluation?
Retrieval evaluation is the measurement of whether the correct passages were returned for a question, judged independently of the answer written from them. It requires a question set with the correct passage identified in advance. Without one, an organisation has no way to know whether a change to the index helped or harmed.
Why answer quality cannot tell you whether retrieval improved
Two measurements are routinely reported as one, and the consequence is that nobody can say what a change accomplished. A reviewer reading answers scores fluency alongside coverage, both of which a capable model produces over mediocre passages. When the index is then altered, the answer scores move for reasons no one can attribute. Separating retrieval evaluation from answer evaluation is what makes the work iterative rather than a sequence of guesses.
- A capable model writes a plausible answer from passages that were wrong.
- Changes to passage sizing or embeddings can only be compared on retrieval metrics.
- A question set with labelled passages is reusable across every index version.
- Retrieval metrics localise a regression to the step that caused it.
How retrieval evaluation works
- Questions are collected from real user traffic rather than written by the team.
- A subject matter expert identifies the passages genuinely answering each question.
- Retrieval returns a ranked list for every question in the set.
- Rank-aware and coverage metrics are computed against the labelled passages.
- Results are reported per question type and per source rather than as one average.
- The set is refreshed as the corpus changes, with the labels re-verified.
How retrieval evaluation is evaluated
A retrieval evaluation is only as good as its question set, so the set is examined as closely as the results. Report the following for each measurement:
- recall at the context budget actually used in production
- mean reciprocal rank with the question count behind it
- metrics reported per source and per question type
- share of questions drawn from real user traffic
- age of the labels against the current version of the corpus
Limitations, risks and common failure modes
- Question sets written by the implementation team encode what the system already handles.
- Labels identify a document when the metric is defined on passages.
- A single average conceals one source on which retrieval never works.
- The evaluation set is never refreshed, so improvements stop being real.