What is normalisation?
Normalisation is the adjustment of scores to remove systematic differences between the reviewers or panels that produced them. A panel scoring consistently low is corrected relative to the others, so that a candidate's outcome does not depend on allocation. The adjustment is derived from the distributions themselves, which means it rests on an assumption about the work each panel received.
Also called: Score normalisation
Why an adjustment can conceal the problem it corrects
The cost of normalising badly falls entirely on individuals, in a way that aggregate reporting never shows. An adjustment assumes each panel received work of comparable quality, and where allocation was never random that assumption is wrong. Correcting a panel that was simply strict is reasonable; correcting a panel that genuinely received weaker submissions penalises everyone in it. An organisation that normalises without testing allocation has traded a visible inconsistency for an invisible one.
- Without adjustment a candidate's outcome depends on which panel received the file.
- The adjustment is only valid where allocation to panels was genuinely random.
- Normalisation is a policy decision, so it requires an owner and a record.
- An adjusted score is harder to explain to the applicant it was applied to.
How normalisation works
- Allocation to reviewers or panels is checked for randomness before anything is adjusted.
- Score distributions are compared across panels on dimensions and on totals.
- Anchor submissions scored by every panel establish how much of the difference is severity.
- An adjustment is derived from that common set rather than from the full distribution.
- Adjusted and original scores are both retained against the decision record.
- Outcome rates by group are rechecked afterwards, since a correction can move them.
How normalisation is evaluated
Normalisation is assessed by whether the adjustment corrects severity without erasing real differences. Examine the following before anything is applied to live decisions:
- evidence that allocation to panels was random
- spread of panel means on the common anchor set
- share of the difference between panels attributable to severity
- change in each group's outcome rate after adjustment
- number of candidates whose outcome the adjustment reverses
Limitations, risks and common failure modes
- Panels receive work by subject area, which makes the randomness assumption false.
- An adjustment derived once is applied for several cycles without rechecking.
- Only the adjusted score is retained, so the original judgement cannot be recovered.
- Candidates are told their score without being told that it was adjusted.