What is an evaluation harness?
An evaluation harness is the repeatable test suite a change must pass before reaching production. It holds a fixed set of cases with their expected outcomes, and it reports where a new version performs worse than the current one. The harness is what makes a release decision evidential rather than a matter of impression.
Also called: Eval harness, Evaluation suite
Why a change needs more evidence than a demonstration
In the absence of a harness, a change is approved because somebody tried a few prompts and the answers looked better. That judgement rests on the cases the author happened to think of, which are rarely the cases that fail. A harness replaces the impression with a comparison against a fixed set, and it records that comparison so a release decision has something behind it. The regression a harness catches most often is the one introduced while repairing something else.
- A fixed case set makes two versions comparable in a way that trials cannot.
- The cases that matter most are the ones nobody would think to try.
- A recorded comparison turns a release into a decision with evidence attached.
- Regression appears most often in behaviour nobody intended to change.
How an evaluation harness works
- Cases are assembled from real traffic, including the failures already seen.
- Each case states its input and the property the output must satisfy.
- Scoring is automated where possible and reviewed by a person where it is impractical.
- The current version and the candidate are assessed on the same set.
- Differences are reported per case so that a regression is identifiable.
- The suite is extended with every incident, which makes it harder to pass over time.
How an evaluation harness is evaluated
A harness is itself assessed on whether it would stop a change that should be stopped. Examine the following over recent releases:
- coverage of the behaviours the system is relied upon for
- proportion of past incidents represented as cases
- share of cases scored automatically rather than by hand
- sensitivity to a deliberately introduced regression
- time to complete a full comparison between two versions
Limitations, risks and common failure modes
- Cases are drawn from easy traffic, so the suite passes almost everything.
- The harness is extended only once a release has already failed in production.
- Automated scoring rewards a surface property instead of the behaviour intended.
- Thresholds are relaxed under schedule pressure with no record of the decision.
- A passing suite is reported as evidence of quality beyond what it measures.