What is an order effect?
An order effect is the influence of a submission's position in the reading sequence on the score it receives. The same work scores differently depending on what preceded it, because each reviewer judges against the examples still fresh in mind. A strong submission read immediately after a weak one benefits from the contrast.
Also called: Sequence effect, Position effect
Why the reading sequence is part of the decision
A submission read eleventh in a sequence of forty is judged against the ten before it, and nobody involved would describe that as the standard. Order effects are among the easiest distortions to demonstrate, since the same submissions can be presented in a different sequence to a second reviewer. They are also among the least often tested, because testing them requires deliberate duplication that capacity planning rarely allows. Where sequence follows arrival time, the effect correlates with whatever drives an early submission.
- The reference a reviewer uses is the recent past rather than the rubric alone.
- Allocation by arrival time turns an order effect into a systematic one.
- Contrast with the previous submission moves scores in both directions.
- The effect is measurable by presenting the same work in a different sequence.
How an order effect works
- Allocation to reviewers is randomised rather than following the order of arrival.
- Scoring proceeds dimension by dimension across submissions instead of file by file.
- Anchor examples are present at the point of scoring as a fixed external reference.
- Breaks are scheduled so that a long sequence is divided into shorter passes.
- A share of submissions is deliberately scored in two different positions.
- Scores are examined against position to establish whether a trend remains.
How an order effect is evaluated
An order effect is detected by varying position while holding the work itself constant. Measure the following across a full cohort:
- correlation between position in the sequence and the score awarded
- score difference for the same submission presented in two positions
- effect of the preceding submission's score on the next one
- drift in the mean score across a single reviewer's session
- agreement between reviewers who read in different sequences
Limitations, risks and common failure modes
- Allocation follows arrival order because the queue is simplest to engineer that way.
- Duplicate scoring at a second position is the first thing dropped when volume rises.
- Reviewers correct for the effect consciously and overcorrect instead.
- A model given submissions in one batch reproduces the effect from its context.