Reasoning trace

Prism

The recorded path from evidence to score, which is what makes a decision reviewable afterwards.

What is a reasoning trace?

A reasoning trace is the recorded path from the evidence in a submission to the score awarded against a criterion. It names the passage relied upon and the step from that passage to the judgement. Its purpose is reconstruction by somebody who was absent, rather than explanation at the time.

Also called: Decision trace

What separates a trace from a justification written afterwards

The failure a trace prevents is the plausible account assembled after the event. A reviewer asked months later why a submission scored three will produce something coherent and unverifiable, and a model asked the same question will do it faster. A trace captured at the moment of scoring is a different object, because it can be checked against the submission it claims to rest on. This is also the only practical way to detect a score that was correct for the wrong reason.

  • A trace captured at the time can be verified; an account given later cannot.
  • It exposes a score that happened to be right without resting on the evidence.
  • An appeal argued over a located passage resolves faster than one argued over memory.
  • A model's stated reasoning becomes testable rather than taken on trust.

How a reasoning trace works

  • Each criterion declares what evidence would satisfy it before any scoring begins.
  • The scorer returns the located passage alongside the score for that criterion.
  • The step from passage to score is recorded in the scorer's own terms.
  • Passages are validated against the stored submission to confirm they still resolve.
  • The trace is stamped with the rubric version and the scorer version in force.
  • It is retained with the decision record for the full appeal period.

How a reasoning trace is evaluated

A trace is assessed on whether a stranger can follow it to the same conclusion. Measure the following on decisions already communicated:

  • proportion of scores with a passage that resolves against the stored submission
  • share of traces a second reviewer judges sufficient to reach the same score
  • relevance of the cited passage to the criterion, judged independently
  • survival of the trace through export and archival
  • elapsed time to produce a full trace on request

Limitations, risks and common failure modes

  • Reasoning generated after the score is indistinguishable from reasoning that produced it.
  • Traces cite a passage on the right topic that does not support the score.
  • Offsets shift when a submission is reprocessed, so the citation no longer resolves.
  • Export formats keep the score and discard the trace behind it.

Related terms

DefensibilityWhether a decision can be explained and substantiated when it is challenged, which is the only standard that matters at the point of appeal.Evidence linkingAttaching each score to the passage in the submission that justifies it.AI audit trailThe complete record of what a system did, on what evidence, and who authorised it, retained for the period the decision can be questioned.Append-only logA record that can be added to and never amended, which is what makes a history worth presenting.Cut-off thresholdThe score at which a decision changes, and the justification for placing it where it was placed.Large language model as judgeUsing a model to score against a rubric, which requires its own calibration and its own evidence.

All 315 terms