What is faithfulness?
Faithfulness is whether an answer represents its retrieved sources accurately. It is independent of whether the answer reads well or tells the reader what they hoped for. An unfaithful answer may be entirely fluent and may even be true, while asserting something the retrieved passages do not support.
Why a true answer can still be an unfaithful one
Faithfulness and correctness are measured as though they were one property, and separating them is what makes a retrieval failure diagnosable. An answer that is correct because the model remembered something from training, where the retrieved passages said nothing of the kind, has produced the right outcome by a route the organisation cannot repeat or defend. Evaluators who score only for correctness will approve that answer and discover the problem on the question where the memory is wrong. Faithfulness asks a narrower question, which is whether the answer follows from the evidence placed in front of it.
- Faithfulness isolates the retrieval failure from the question of factual truth.
- An answer true by recall rather than by retrieval cannot be cited or audited.
- Unfaithful summaries frequently soften a condition the source states firmly.
- The measurement can be automated, so it suits continuous monitoring in production.
How faithfulness works
- The answer is decomposed into individual claims that can be judged separately.
- Each claim is paired with the retrieved passages available when it was generated.
- A judge determines whether those passages support the claim or contradict it.
- Claims unsupported by any passage are counted against the answer.
- The judge is itself validated against human adjudication on a subsample.
- Scores are tracked per question type rather than as a single corpus average.
How faithfulness is evaluated
Faithfulness is scored at the level of the claim, since an answer is rarely wholly supported or wholly invented. Examine the following on production traffic:
- proportion of claims supported by the retrieved passages
- rate of claims contradicting a retrieved passage
- agreement between the automated judge and human adjudication
- faithfulness on questions where retrieval returned weak candidates
- distribution of unsupported claims across question types
Limitations, risks and common failure modes
- Faithfulness is scored by a model from the same family as the one generating.
- An answer faithful to a superseded document scores perfectly and misleads anyway.
- Whole-answer scoring conceals one unsupported clause inside a sound paragraph.
- Claims are decomposed inconsistently, so scores move when the decomposition changes.
- Evaluation sets are refreshed rarely, so the score drifts away from live traffic.