What is a score distribution?
A score distribution is the shape formed by all the scores awarded on a dimension or on a total. It shows which points of the scale reviewers actually use and how far submissions separate from one another. A distribution concentrated on two adjacent points indicates a dimension recording almost nothing.
What the shape of the scores says about the rubric
A rubric that discriminates poorly costs an organisation twice, first in the places awarded to the wrong submissions and then in the appeals it cannot answer. The distribution is where that failure is visible months before an appeal arrives, and it is almost never examined. Five point scales commonly collapse onto two values, at which point the weighting of the dimension is irrelevant because the dimension separates nobody. A healthy shape uses the range, with density falling away at the extremes rather than piling up against them.
- A dimension that uses two points of five is separating almost nothing.
- Weighting has no effect on a dimension whose scores barely vary.
- Clustering at the threshold means small scoring errors reverse real outcomes.
- Comparing the shape against the previous cycle exposes inflation early.
How a score distribution works
- Scores are examined per dimension rather than only on the total.
- The frequency of each point on the scale is counted for the cohort.
- Density within one point of the threshold is measured separately.
- The shape is compared against the previous comparable cohort.
- Distributions are also compared between reviewers to locate severity differences.
- A dimension with no spread is redefined or merged before the next cycle.
How a score distribution is evaluated
A distribution is read for spread and for where its mass falls relative to the threshold. Examine the following for every dimension:
- frequency with which each point on the scale is used
- spread of scores within each dimension
- density of submissions within one point of the threshold
- difference in shape between reviewers scoring the same pool
- shift in the distribution against the previous cycle
Limitations, risks and common failure modes
- Reviewers avoid the ends of a scale, which halves its effective range.
- A tidy symmetrical shape is reported as evidence that the rubric is sound.
- Distributions are examined on totals, hiding a dimension with no spread at all.
- Inflation across cycles makes a fixed threshold progressively easier to reach.