What is deterministic replay?
Deterministic replay is the reconstruction of exactly what a system did on a given occasion. Recorded inputs are presented again to a pinned version of the model with its original parameters, and the outputs are compared against what was recorded. Where the two match, the record is confirmed as an account of what happened.
Also called: Replay
Why replay settles an argument that testimony cannot
Two accounts of the same event rarely converge by discussion. A customer states that the system told them one thing while the operator's notes say another, and the model has been updated twice since. Replay removes the disagreement by producing the original behaviour on demand, so long as the inputs were captured at the time along with the version. Without that capture the organisation is left arguing about memory.
- Replay converts a claim about past behaviour into something observable.
- It separates a model error from an input that was never what anyone assumed.
- A pinned version makes an old decision reproducible after the system has moved on.
- Divergence on replay is itself a finding worth investigating.
How deterministic replay works
- Every input to the step is recorded with the retrieved passages and tool returns.
- The model version and its sampling parameters are stored with the record.
- Any source of variation is either fixed by a seed or captured as an input.
- Replay presents the stored inputs to the pinned version in the original order.
- The replayed output is compared against the output recorded for that occasion.
- Divergence is reported with the step at which the two accounts separated.
How deterministic replay is evaluated
Replay is judged on how often it reproduces the recorded behaviour and on how far back it reaches. Measure the following across a sample of closed cases:
- proportion of cases reproducing the recorded output exactly
- age of the oldest case that can still be replayed
- availability of pinned versions across the retention period
- completeness of captured inputs per step
- time to complete a replay from a case identifier
Limitations, risks and common failure modes
- Hosted model versions are withdrawn, which ends replay for every earlier decision.
- Sampling temperature is left unrecorded, so outputs differ on every attempt.
- Retrieved passages go unstored, leaving replay dependent on a changed index.
- Tool responses from external systems cannot be reproduced without a recorded stub.
- Replay is implemented and then never attempted until the first serious dispute.