What is reviewer throughput?
Reviewer throughput is the number of submissions a reviewer completes in a given period at a stated standard of review. The standard is part of the measure, since throughput can always be increased by reviewing less carefully. It is reported per reviewer and per submission type, because a mixed caseload makes an average uninformative.
Why throughput without a quality standard is a meaningless number
A capacity plan asks how many submissions a reviewer can complete in a day, and the honest answer is a range with a quality figure attached to each end. Reporting the number alone invites the organisation to manage it, and throughput responds immediately to pressure because the fastest available saving is a shallower reading. The effect appears months later as an appeal rate, by which time the improvement has been reported to a board. Throughput and agreement therefore have to be measured over the same period and presented together.
- Throughput rises whenever review becomes shallower, so it moves under pressure.
- A figure quoted without its quality standard cannot be compared between periods.
- Mixed caseloads make an average throughput figure uninformative.
- Capacity planning depends on throughput, so the figure drives staffing decisions.
How reviewer throughput works
- Review activity is instrumented per submission with start and completion times.
- Submissions are classified by type, so throughput is reported within comparable work.
- A quality standard is defined, commonly agreement with a second independent reviewer.
- A sample of each reviewer's work is double reviewed within the same period.
- Throughput is reported per reviewer alongside the agreement figure for that sample.
- Any change in throughput is examined against agreement before it is accepted.
How reviewer throughput is evaluated
Throughput is only interpretable alongside the quality of the decisions produced in the same period. Report the following figures together for each reviewer:
- submissions completed per reviewer per period, by submission type
- agreement with a second reviewer on the sampled subset
- median and tail of review duration per submission
- appeal and overturn rate attributable to each period
- proportion of time spent on items later found deficient
Limitations, risks and common failure modes
- Throughput targets produce shallower review long before anyone reports a quality problem.
- Reviewers clear simple items first, which raises throughput without raising capacity.
- Duration measured from case opening includes time the reviewer was elsewhere.
- Averages across mixed caseloads conceal the type consuming the capacity.