What is inter-rater reliability?
Inter-rater reliability is the degree to which two reviewers independently reach the same judgement on the same submission. It is a property of the assessment process rather than of either reviewer, since low reliability usually indicates the criteria are ambiguous. It is reported with a statistic that accounts for chance agreement, commonly Cohen's kappa.
Also called: Inter-rater agreement
Why reliability is the precondition for every other claim
Accuracy and reliability are frequently treated as one property, and they can move in opposite directions. A process can be highly reliable and consistently wrong, since two reviewers trained on the same flawed standard will agree with each other all day. What reliability establishes is that a score means something independent of who produced it, which is the precondition for any claim about fairness. Without it, an applicant's outcome depends on queue allocation, which is indefensible in front of anyone.
- Low reliability means an applicant's outcome depends on which reviewer received the file.
- It is the ceiling on any concordance figure an automated score can claim.
- Poor reliability points at ambiguous criteria rather than at careless reviewers.
- Measuring it repeatedly during a cycle detects drift that a single reading cannot.
How inter-rater reliability works
- A subset of submissions is allocated to two reviewers independently.
- Reviewers score without visibility of each other's judgement or of any recommendation.
- Paired scores are assembled per dimension and for the overall outcome.
- A chance corrected statistic is calculated with its confidence interval.
- Dimensions with the weakest agreement are examined for ambiguous wording.
- The exercise is repeated mid-cycle on live submissions rather than only at calibration.
How inter-rater reliability is evaluated
Reliability is reported per dimension, since an acceptable overall figure can hide one unusable criterion. Examine the following for each dimension in turn:
- chance corrected agreement per dimension with its interval
- proportion of submissions double scored during the cycle
- agreement between each reviewer and the panel consensus
- change in agreement between the opening and closing weeks
- raw agreement alongside the corrected figure for context
Limitations, risks and common failure modes
- Double scoring is allocated to the easiest submissions because capacity is limited.
- Reviewers discuss a submission before scoring, which inflates the figure.
- An acceptable overall figure conceals a dimension on which nobody agrees.
- Reliability is measured at calibration and reported as the figure for the whole cycle.