What is judge calibration?
Judge calibration is the practice of testing an automated scorer against human decisions. The comparison uses cases the scorer has not previously been assessed on, repeated at a stated cadence rather than once at acceptance. It establishes where the automated judgement and the human standard part company, and by how much.
Who checks the thing doing the checking
Nobody asks who checked the checker until an outcome is disputed, and by then the answer is historical. An automated scorer accepted on a single comparison exercise carries that one figure for years. Model upgrades and rubric revisions arrive in the meantime and quietly invalidate it. Calibration treats the scorer as an instrument that drifts, measured again at intervals on cases nobody has tuned against.
- A scorer accepted once is quoted long after the cohort behind the figure has changed.
- Calibration on unseen cases prevents a figure flattered by the examples used in tuning.
- Repeated measurement produces a time series, which exposes drift a single comparison cannot.
- The human reference itself moves, so both sides of the comparison need measuring.
How judge calibration works
- A holdout sample is reserved that no prompt or rubric revision has been tuned against.
- Human reviewers score that sample independently, with no sight of the automated result.
- Agreement is computed per dimension as well as at the level of the outcome.
- Disagreements are examined individually to establish which judgement was the correct one.
- Human agreement on the same sample is measured as the ceiling for comparison.
- The exercise is repeated after every model version change and at a fixed interval.
How judge calibration is evaluated
Judge calibration is itself assessed on whether it would detect a scorer that has gone wrong. Examine the following on the reserved sample rather than on tuning data:
- chance corrected agreement between the scorer and human reviewers per dimension
- human agreement on the same sample as the comparison ceiling
- size of the reserved sample and the way it was drawn
- elapsed time since the last calibration exercise
- change in agreement across the two most recent model versions
Limitations, risks and common failure modes
- Calibration samples are frequently drawn from the pool used to tune the prompt.
- A scorer is calibrated at acceptance and then upgraded without the figure being recomputed.
- Agreement is measured on the outcome only, hiding a dimension the scorer never reaches.
- The human reference degrades over a cycle while the scorer stays constant.