Failure Analysis for AI Agents: Turning Production Errors Into Eval Cases

If you operate an agent, your production failures are the sharpest signal you have about where the system actually breaks, far more valuable than any failure you could imagine synthetically. A synthetic test case reflects a failure you thought of. A production failure is one that actually happened, on a real input, in a way you did not anticipate, which is precisely the kind of failure your evaluation was missing. Capturing these systematically, rather than fixing and forgetting, is what turns a system that repeats its mistakes into one that only makes each mistake once.

Why fix-and-forget lets failures recur

The instinct when an agent fails is to fix it: identify what went wrong, adjust the prompt or the logic, confirm the symptom is gone, and move on. This resolves the immediate problem and leaves a gap, because fixing a failure without capturing it as a permanent test case means nothing prevents that failure, or a close variant, from returning later. A future change to the system can reintroduce the exact problem you fixed, and without a test case guarding against it, the regression slips through unnoticed until it fails in production a second time. Fix-and-forget treats each failure as a one-time event, when failures are recurring hazards that need permanent guards.

The deeper loss is informational. Each production failure carries information about a weakness in the system, and a pattern of failures carries information about a systematic weakness, but only if you capture and examine them. A team that fixes each failure in isolation and discards it never sees the pattern, never learns that a whole class of inputs causes trouble, and keeps addressing symptoms one at a time while the underlying weakness persists. Capturing failures is what makes the pattern visible, and the pattern is where the real improvement lives, because fixing a class of failures is worth far more than fixing its individual instances. This is the same logic as building the golden dataset from real cases, applied specifically to the failures the system produces.

Turning a failure into an eval case

Converting a production failure into an evaluation case is a specific practice, and doing it well is what makes the failure permanently useful. When the agent fails, you capture the case: the input that caused the failure, from the trajectory trace, and the correct outcome it should have produced, established with whatever expertise that requires. That input-and-correct-answer pair becomes a permanent case in your evaluation set, which means every future version of the system is tested against it, and the failure can never silently recur, because any change that reintroduces it fails the evaluation before reaching production.

The value of this compounds over time. Each captured failure adds a permanent guard, so the evaluation set grows precisely in the areas where the system has actually struggled, becoming sharper exactly where sharpness matters. A mature agent’s evaluation set, built this way, is a record of every way the system has broken and a guarantee that none of those ways can return unnoticed, which is a far stronger asset than a synthetic test set covering the failures someone imagined. The practice turns the painful experience of a production failure into a lasting improvement in the system’s reliability, which is the best return available on a failure that already happened.

Analysis beyond the individual case, and its cost

Capturing individual failures as cases is the foundation. The higher-value work is analysing the failures in aggregate to find the systematic weaknesses, because a collection of captured failures is data about how the system breaks, and examining that data reveals patterns no individual failure shows. Several failures that look unrelated at the surface may share a root cause, a particular kind of input, a specific reasoning step, a tool that behaves poorly under certain conditions, and finding that shared cause lets you fix the class rather than the instances. This aggregate analysis is where failure data pays off most, and it depends on having captured the failures systematically in the first place.

The trade-off is that this discipline costs ongoing effort that a fix-and-forget approach avoids. Capturing each failure properly, establishing its correct answer, adding it to the evaluation set, and periodically analysing the accumulated failures for patterns is real work layered on top of simply fixing the immediate problem. The payoff is a system that improves systematically from its own failures rather than repeating them, and an evaluation set that grows stronger exactly where the system is weak. An organisation under pressure will be tempted to fix and move on, and that temptation is a choice to keep relearning the same failures, because the failure not captured is the failure free to recur. Naming this trade-off is what justifies the investment in treating failures as assets rather than nuisances.

Where it almost broke

We operated an agent and handled its failures the ordinary way for a period: each time it did the wrong thing, we diagnosed and fixed it, confirmed the symptom was gone, and moved on. Over time a frustrating pattern emerged, where failures we had fixed reappeared after later changes, because nothing guarded against their return, and we were spending effort re-fixing problems we had already solved. Worse, we were addressing failures one at a time without seeing that many of them shared an underlying cause, so we kept treating symptoms of a systematic weakness we had not identified.

The correction changed how we treated failures. Every production failure became a permanent evaluation case, capturing the input and the correct outcome, so a fixed failure could never silently return, and any change that reintroduced it failed evaluation before deployment. We began analysing the accumulated failures in aggregate, which revealed that a cluster of seemingly unrelated failures shared a common root, and fixing that root resolved the whole class at once. The system started improving systematically from its failures instead of repeating them, and the evaluation set grew sharpest exactly where the system had struggled. The lesson was that production failures are the most valuable test material a system produces, and discarding them is discarding the system’s best path to reliability.

Make each failure the last of its kind

The reframe that turns failures into an advantage is treating every production error as material to be captured and learned from rather than a nuisance to be fixed and forgotten. Failure analysis for AI agents converts the system’s failures into permanent evaluation cases and, in aggregate, into a map of its systematic weaknesses, which is how a system stops repeating its mistakes and starts improving from them. The failure that already happened is a sunk cost. Capturing it is how you get a return on it.

The teams whose agents grow more reliable over time are the ones that stopped discarding their failures and started treating them as the sharpest signal they have, capturing each one as a guard against its recurrence and analysing the accumulation for the patterns that reveal deeper problems. An agent that fixes and forgets relearns its failures indefinitely. An agent that captures and analyses them makes each failure the last of its kind, which is what turns the painful reality of production errors into the engine of a more dependable system.