What is concordance?
Concordance is the share of cases where an automated score and a human reviewer reach the same conclusion. It is normally reported at the level of the decision, since that is what an applicant experiences, and also per dimension to locate disagreement. The human judgement serves as the reference, with all the limits that implies.
Also called: Agreement rate
Why concordance is a claim about the humans as well as the model
The first question an evaluation committee asks about an automated score is how often it agrees with their own reviewers. It is the right question with a trap inside it, because concordance is bounded by how consistently those reviewers agree with each other. A system that agrees with a panel more often than the panel agrees with itself is being measured against a moving reference. Concordance is therefore reported alongside inter-rater reliability, or it says less than it appears to.
- Concordance translates model performance into a figure a review committee understands.
- It is capped by the consistency of the human reference it is measured against.
- Per dimension concordance locates exactly where the model and the panel part company.
- A high figure on everyday cases can conceal poor agreement on borderline ones.
How concordance works
- A sample of submissions is scored by the automated process and by human reviewers independently.
- The sample is drawn to include borderline cases in proportion to their real frequency.
- Agreement is computed at the decision level and again for each dimension.
- Disagreements are examined individually to establish which judgement was correct.
- Human inconsistency on the same sample is measured as the ceiling for comparison.
- The figure is recomputed after any change to the rubric or the model version.
How concordance is evaluated
Concordance is only interpretable with its sample and its human ceiling stated. Report the following together for each comparison:
- decision level concordance with the sample size behind it
- concordance per dimension rather than on the outcome alone
- inter-rater reliability on the same sample as the comparison ceiling
- concordance restricted to cases near the threshold
- direction of disagreement, meaning whether the model scores higher or lower
Limitations, risks and common failure modes
- Samples drawn from everyday cases exclude the borderline decisions that matter most.
- Concordance is quoted without the human agreement figure that bounds it.
- A model that reproduces a panel's existing bias scores highly on concordance.
- The figure is measured once at acceptance and never recomputed after a rubric change.