Pairwise comparison

Prism

Judging two submissions against each other, which people do more reliably than assigning absolute scores.

What is pairwise comparison?

Pairwise comparison asks a reviewer which of two submissions is stronger, rather than what each one is worth on a scale. People answer the comparative question far more consistently, because it requires no shared interpretation of a numeric point. Many such judgements are then combined into a ranking by a statistical model.

Also called: Paired comparison

Why reviewers answer a comparison better than a scale

Ask a panel to agree what a seven means and the discussion lasts an afternoon; ask which of two proposals is stronger and the panel agrees quickly. Pairwise comparison exploits that difference, and it pays for the gain in volume, since a field of two hundred cannot be exhaustively paired. The practical method samples comparisons adaptively, concentrating them where the ranking is still uncertain. What the organisation gains in reliability it gives up in explanation, because a position in a ranking is harder to justify to one applicant than a score against a criterion.

  • A comparative judgement needs no shared interpretation of a point on a scale.
  • Agreement between reviewers is typically higher than it is for absolute scoring.
  • The number of possible pairs grows faster than any panel can read.
  • A ranking position is harder to explain to an applicant than a dimension score.

How pairwise comparison works

  • Reviewers are presented with two submissions and asked which better meets the criterion.
  • Each judgement is recorded with its reviewer and the pair that produced it.
  • Comparisons are sampled adaptively towards pairs where the ranking remains uncertain.
  • A statistical model converts the accumulated judgements into a quality estimate.
  • Reviewer severity and inconsistency are estimated from that same model.
  • The resulting order is checked against any absolute scores already held.

How pairwise comparison is evaluated

Pairwise comparison is assessed on the stability of the ranking and on what it cost to reach. Examine the following as comparisons accumulate:

  • consistency of each reviewer's judgements, including reversals on the same pair
  • number of comparisons required before the ranking stabilises
  • width of the uncertainty interval around each submission's position
  • agreement between the comparative ranking and independent absolute scores
  • reviewer time per comparison against time per scored submission

Limitations, risks and common failure modes

  • The comparison count required for a large field exceeds the panel time available.
  • Adaptive sampling concentrates attention near the threshold and thins it elsewhere.
  • A ranking carries no absolute standard, so a weak field still produces a top position.
  • Inconsistent judgements accumulate quietly and the model absorbs them as noise.

Related terms

Ranking and ratingTwo different questions, one asking which is better and the other asking how good each one is.Anchor exampleA worked example showing what a given score looks like, so reviewers are calibrated against something concrete.Calibration sessionA meeting where reviewers score the same submissions and reconcile their differences before the real review begins.Cut-off thresholdThe score at which a decision changes, and the justification for placing it where it was placed.RubricThe stated criteria a submission is scored against, written down before any scoring starts.Structured evaluationScoring against defined criteria with the reasoning recorded, as opposed to an overall impression.

All 315 terms