What is structured evaluation?
Structured evaluation is the assessment of submissions against criteria defined in advance, with the reasoning behind each score recorded. It replaces the overall impression of an experienced reader with separate judgements that can be examined one at a time. The structure is what makes a decision reviewable by somebody who was absent.
Also called: Structured assessment
What an organisation gives up by scoring on impression
An experienced assessor reading on impression is often accurate and always unexaminable, and the second property is what eventually costs the organisation. Impression cannot be transferred to a new assessor, and it cannot be defended at an appeal months later. Structure converts one person's judgement into an institutional capability, at the cost of defining the criteria properly. Organisations that adopt the structure without doing that definition acquire the cost and none of the benefit.
- Separate judgements can be examined individually, where an impression cannot.
- A defined criterion can be taught to a new assessor in an afternoon.
- Recorded reasoning is what an appeal panel examines, and the only thing it can.
- Structure turns a difference of opinion into a locatable disagreement about a criterion.
How structured evaluation works
- The criteria are defined as separate dimensions with their weights agreed in advance.
- Each dimension carries anchored scale points drawn from real submissions.
- Assessors score each dimension on its own and attach the supporting passage.
- Dimension scores are combined by the agreed weights and compared against the threshold.
- Borderline and contested cases are referred to a named human decision maker.
- Scores and reasoning are retained with the rubric version that produced them.
How structured evaluation is evaluated
Structured evaluation is assessed on whether the structure is doing the work the organisation claims for it. Examine the following on a completed cohort:
- agreement between assessors per dimension rather than on the total
- proportion of scores with reasoning linked to located evidence
- correlation between dimensions, which indicates one impression recorded repeatedly
- share of outcomes reproducible from the recorded scores alone
- outcome rates by group for the cohort assessed
Limitations, risks and common failure modes
- Assessors form an impression first and then fill in dimensions to match it.
- Criteria are written in language that reproduces whatever assessors already did.
- The structure is adopted for the record while the decision happens in a meeting.
- Reasoning is captured in a free text field that no later process ever reads.