What is a hallucination?
A hallucination is a fluent statement that no retrieved source supports. It arrives in the same register as a correct answer, with the same apparent confidence, which is what makes it dangerous rather than merely incorrect. In a regulated setting the harm lies in the statement being acted upon before anyone thinks to check it.
Also called: Confabulation
Why fluency is the property that makes an invented answer harmful
A model that answered badly in an obvious way would be a manageable problem. What an organisation actually deploys is a system whose invented answers are indistinguishable in tone from its correct ones, so the reader has no signal to act on. In a bank that means a relationship manager quoting a limit nobody set; in a hospital it means a dosage that appears in no protocol. The engineering response is to remove the conditions under which the system answers from nothing, rather than to improve the wording of the instruction.
- An invented answer carries the same confidence as a correct one.
- Readers cannot detect the failure without the source beside the statement.
- Retrieval reduces the opportunity, and it does not remove it.
- The failure is bounded by refusal, which has to be a permitted outcome.
How a hallucination works
- Generation is restricted to the passages retrieved for the question.
- Every statement is required to carry a citation into those passages.
- Statements without supporting evidence are withheld rather than returned.
- The system refuses where retrieval returns nothing of sufficient quality.
- Answers are scored for support against their sources on live traffic.
- Unsupported statements are sampled and examined for their common cause.
How a hallucination is evaluated
Hallucination is measured by checking answers against the evidence available at the time rather than against the world. Examine the following on live traffic:
- rate of statements unsupported by any retrieved passage
- refusal rate on questions the corpus cannot answer
- proportion of invented statements carrying a plausible citation
- unsupported statement rate by question type and by language
- agreement between automated detection and human review
Limitations, risks and common failure modes
- Refusal is treated as a product failure, so the system is tuned to always answer.
- An invented statement acquires a citation to the nearest retrieved passage.
- Detection is performed on a curated set that excludes the hardest questions.
- Readers stop checking citations once the system has been right for a while.
- A corpus gap is reported as a model problem and addressed by prompt changes.