What is data drift?
Data drift is the input distribution moving away from the distribution a system was evaluated on. The documents arriving or the questions being asked change, while the system remains exactly as it was accepted. Drift in the inputs usually precedes any visible change in the quality of the output.
Also called: Input drift, Covariate shift
Why the inputs move before the answers do
The first sign is usually nothing at all. A system accepted in March meets questions in October that nobody was asking in March, and the evaluation set holds nothing resembling them. Output quality has not yet moved, which is why monitoring the outputs alone finds this late. Measuring the inputs gives an organisation warning while it still has a choice about what to do.
- Input drift is observable without waiting for an outcome to be judged.
- An evaluation set fixed at acceptance becomes less representative every month.
- A seasonal shift in questions can resemble a regression in model quality.
- Drift detected early converts an incident into a scheduled piece of work.
How data drift works
- A reference distribution is retained from the data the system was evaluated on.
- Incoming inputs are summarised on the features that matter to the task.
- Each feature's current distribution is compared against the reference on a cadence.
- A divergence beyond a stated threshold raises an alert to a named owner.
- The affected slice is sampled and examined by a person before any change.
- The evaluation set is extended with the new material once a shift is confirmed.
How data drift is evaluated
Drift detection is itself assessed on whether it would notice a shift that mattered. Examine the following over a period containing a known change:
- divergence between current inputs and the reference distribution
- lead time between an input shift and a measured quality change
- false alert rate across normal seasonal variation
- coverage of the features the task actually depends on
- age of the reference distribution in use
Limitations, risks and common failure modes
- Monitoring is configured on volume rather than on the content of the inputs.
- Aggregate statistics stay stable while one important segment has shifted entirely.
- Alert thresholds are widened after a noisy first month and then forgotten.
- The reference distribution is never refreshed, so everything eventually reads as drift.
- Drift is detected and recorded with no owner able to act on it.