Moderation

Prism

A second reading applied to a sample or to borderline cases, to confirm the standard held.

What is moderation?

Moderation is a second reading of work that has already been scored. It is applied to a sample of the cohort or to the cases closest to a boundary. Its purpose is to confirm that the agreed standard held in practice rather than to rescore the cohort. A moderator may adjust an individual score or refer a wider pattern back to the panel.

Also called: Second marking

What a moderation sample can and cannot detect

Moderation becomes the only available control the moment volume makes double scoring every submission impossible, which is almost immediately. A sample is therefore doing the work of an assurance function, and the way it is drawn decides what it can detect. Samples chosen for convenience, meaning whatever is already to hand, confirm that the standard held among the easiest cases. The useful sample is weighted towards boundaries and towards the reviewers whose agreement figures are weakest.

  • Full double scoring is unaffordable, so the sample carries the whole assurance claim.
  • A sample drawn at random misses the boundary cases where errors change outcomes.
  • Moderation findings are the evidence that a published standard was applied as stated.
  • A pattern found in moderation should change the rubric rather than only the score.

How moderation works

  • The sampling rule is fixed in advance, covering a share of every reviewer's work.
  • Borderline cases within a stated margin of the threshold are included in full.
  • A second reviewer scores the sampled work without sight of the original score.
  • Differences beyond a stated tolerance are discussed against the criterion wording.
  • Individual adjustments are recorded with a reason, separately from the original score.
  • Systematic differences are reported back as a rubric or calibration issue.

How moderation is evaluated

Moderation is judged on what it finds and on whether the sample could have found it. Examine the following across a completed cycle:

  • share of each reviewer's work included in the sample
  • coverage of cases within the borderline margin
  • proportion of moderated scores adjusted, by reviewer
  • size of the adjustments relative to the scale in use
  • number of rubric changes arising from moderation findings

Limitations, risks and common failure modes

  • Moderation is applied to whatever work is nearest to hand rather than to a defined sample.
  • A moderator who outranks the original reviewer imposes a standard instead of testing one.
  • Adjusted scores overwrite the original, so the disagreement disappears from the record.
  • Findings are recorded against individual reviewers when the rubric is the cause.

Related terms

ConcordanceThe share of cases where an automated score and a human reviewer arrive at the same conclusion.Score driftThe standard moving over a long review cycle, so that the first submission and the four hundredth are judged differently.Grant evaluationAssessing funding applications against published criteria, where every unsuccessful applicant reads the standard applied.Cohen's kappaA statistic for agreement between two raters that accounts for the agreement chance alone would produce.Inter-rater reliabilityThe degree to which two reviewers independently reach the same judgement on the same submission.NormalisationAdjusting for systematic differences between reviewers or panels so that scores remain comparable.

All 315 terms