Groundedness

Compass

The measurable degree to which an answer is supported by the documents retrieved for it.

What is groundedness?

Groundedness is the degree to which an answer is supported by the documents retrieved for it, expressed as a measurement rather than an assurance. It is scored on a scale and monitored over time, broken down by question type. A system with no groundedness figure has an opinion about its own reliability.

Why reliability has to be a number somebody owns

Boards and risk committees ask whether an answering system can be trusted, and the useful reply is a figure with a method behind it. Groundedness provides one, which converts a conversation about confidence into a conversation about a threshold and a trend. The discipline this imposes is uncomfortable, because the figure will fall after a corpus change or a model upgrade, and somebody then owns the fall. Organisations that decline to measure it are left defending a claim they have no way to substantiate.

  • A measured figure can be compared across versions, where an assurance cannot.
  • Monitoring over time reveals degradation that acceptance testing never sees.
  • A threshold makes a release decision explicit rather than a matter of impression.
  • Per-segment scores locate the question types the system should refuse.

How groundedness works

  • A representative question set is assembled from real traffic and labelled.
  • Each answer is scored for support against the passages retrieved for it.
  • The scorer is calibrated against human adjudication on a subsample.
  • Scores are aggregated overall and broken down by question type.
  • A threshold is agreed, below which the system refuses rather than answers.
  • The measurement repeats on a schedule and after any corpus or model change.

How groundedness is evaluated

Groundedness is itself evaluated on whether the score would detect a real degradation. Examine the following alongside every reported figure:

  • agreement between the automated scorer and human adjudication
  • score variance across repeated measurement of the same set
  • coverage of the question set against live traffic distribution
  • sensitivity of the score to a deliberately degraded retrieval configuration
  • trend in the figure across corpus and model versions

Limitations, risks and common failure modes

  • A single corpus average conceals a question type that is never grounded.
  • The scorer and the generator share a model family, so their errors correlate.
  • Evaluation sets age, and the score measures a traffic pattern that has moved.
  • A figure measured once at acceptance is quoted to a board years afterwards.
  • Thresholds are lowered to accommodate a release rather than to reflect risk.

Related terms

Refusal and abstentionThe system declining to answer when the evidence is not there, which is a feature rather than a shortfall.Retrieval augmented generation (RAG)Answering from documents retrieved at the moment of the question, so the answer reflects your material rather than the model's memory.Answer attributionNaming the source behind each part of an answer rather than the answer as a whole.Citation groundingAttaching every statement in an answer to the passage it came from, so a reader can check it.FaithfulnessWhether an answer represents its sources accurately, as distinct from whether it is agreeable or well written.HallucinationA fluent statement that is unsupported by any source, which is the failure mode that matters most in a regulated setting.

All 315 terms