Testing AI Agents: Why Unit Tests Don’t Cover Non-Deterministic Systems

If you test AI agents, the mismatch between deterministic testing and non-deterministic systems is the reason your tests can pass while the agent fails, because the tests assert deterministic outcomes and the agent behaves probabilistically, which means the tests either fail spuriously on the non-determinism or are written so loosely they catch nothing. Testing an agent well requires methods built for non-determinism, evaluating the probabilistic behaviour rather than asserting deterministic outputs, and understanding why unit tests do not cover agents is what leads to the testing that does. The deterministic testing that serves conventional software is necessary and insufficient for an agent, and knowing where it falls short is what produces adequate agent testing.

Why unit tests do not fit agents

A unit test asserts a deterministic relationship: this input produces this expected output, and the test passes if the actual output matches the expected one, which works because the system under test is deterministic, so the same input reliably produces the same output and the assertion holds or fails cleanly. An agent is non-deterministic, which means the same input can produce different outputs on different passes, and the deterministic assertion breaks: the test that asserts a specific expected output either fails when the agent produces a different valid output, giving a spurious failure, or is written to accept any output, giving a test that asserts nothing. The deterministic assertion that unit testing depends on does not fit a system whose output for a given input is a distribution rather than a fixed value.

This mismatch has a subtler consequence than spurious failures, which is that unit tests, even where they can be made to pass, do not measure the thing that matters for an agent. An agent’s correctness is not whether it produces a specific output but whether it accomplishes the task, and an agent can accomplish the task through different valid outputs, which a unit test asserting a specific output cannot assess. The task success that matters for an agent is not a deterministic output match, and unit tests built to check deterministic output matches measure output equality rather than task success, which means they miss the agent’s actual correctness even when they can be made to operate. Unit tests do not fit agents because they assert deterministic outputs that agents do not reliably produce and measure output equality rather than the task success that is the agent’s real correctness, which is why an agent tested with unit tests alone is tested with a method that neither holds nor measures what matters.

Testing built for non-determinism

Testing an agent requires methods built for non-determinism, which means evaluating the agent’s probabilistic behaviour rather than asserting deterministic outputs. The evaluation harness is the core of agent testing, scoring the agent’s behaviour against defined success criteria on a set of cases, which assesses whether the agent accomplishes the tasks rather than whether it produces specific outputs, measuring the task success that is the agent’s actual correctness. This is the testing that fits the agent, because it evaluates the probabilistic behaviour against what the agent should accomplish rather than asserting the deterministic outputs the agent does not reliably produce.

The evaluation accounts for the non-determinism in ways unit tests cannot, assessing whether the agent’s varied outputs accomplish the task rather than whether they match a fixed output, and scoring the trajectory rather than just the endpoint, because an agent can reach a correct outcome through different paths and the quality of the path matters. Where parts of the agent’s behaviour are deterministic, the deterministic checks that unit tests provide still apply, asserting the deterministic outcomes where they exist, which means agent testing combines deterministic assertions for the deterministic parts with evaluation for the non-deterministic behaviour, using each where it fits. The non-determinism also means testing an agent once is not enough, because passing once does not mean reliably passing, which pushes agent testing toward assessing the behaviour across operates and toward the continuous evaluation that keeps testing the agent as it operates. Testing built for non-determinism evaluates the probabilistic behaviour against task success, scores the trajectory, applies deterministic checks where behaviour is deterministic, and continues into production, which is the testing that fits an agent where unit tests alone do not.

The trade-off, and what unit tests are still good for

Testing an agent properly costs more than unit testing, and naming that cost keeps the testing realistic. The evaluation that assesses the agent’s probabilistic behaviour requires the golden dataset of cases and the evaluation infrastructure, and often the LLM-as-judge with its calibration requirement, which is more effort than writing unit tests with deterministic assertions. This cost is the cost of testing a non-deterministic system properly, and it is justified because unit testing alone does not adequately test an agent, passing while the agent fails or asserting nothing, so the evaluation effort buys the testing that unit tests do not provide.

Unit tests are not useless for agents, which is the nuance, because agents have deterministic parts, the deterministic logic around the model, the validation and controls, the deterministic components of the system, and these deterministic parts are well served by unit tests that assert their deterministic behaviour. The mistake is testing the agent’s non-deterministic behaviour with unit tests, and the reasonable use is unit-testing the deterministic parts of the system while evaluating the non-deterministic behaviour with the evaluation built for it, using each testing method where it fits. This mirrors the build of the agent itself, where the deterministic scaffolding around the model is the well-tested deterministic part and the model’s behaviour is the non-deterministic part that needs evaluation, and the testing matches the build: deterministic tests for the deterministic scaffolding, evaluation for the non-deterministic behaviour. The trade-off is between the effort of evaluation and the inadequacy of unit tests alone, and using unit tests for the deterministic parts and evaluation for the non-deterministic behaviour tests the agent properly without over- or under-applying either method.

Where it almost broke

We saw a team test an agent with the unit testing that served its conventional software well, writing tests that asserted specific expected outputs for given inputs, and the tests were failing spuriously on the agent’s non-determinism, because the agent produced valid outputs that differed from the asserted expected ones, so the tests flagged failures that were not failures. The team’s response had been to loosen the assertions until the tests passed, which produced tests that asserted almost nothing and caught almost nothing, so the agent went to production with tests that passed and did not actually test its behaviour, and it failed in production in ways the emptied-out tests never flagged. The deterministic unit testing had not fit the non-deterministic agent, and forcing it had produced tests that were either spurious or empty.

The correction built testing that fit the agent. We evaluated the agent’s non-deterministic behaviour with an evaluation harness that scored task success against defined criteria on a golden dataset, assessing whether the agent accomplished the tasks rather than whether it produced specific outputs, and scored the trajectory rather than just the endpoint. We kept unit tests for the deterministic parts of the system, the validation and control logic, where deterministic assertions fit. And we extended the testing into continuous evaluation in production, because passing once did not mean reliably passing. The agent’s testing started catching the failures the emptied-out unit tests had missed, because it evaluated the probabilistic behaviour against task success rather than asserting deterministic outputs. The lesson was that unit tests do not cover a non-deterministic agent, and testing built for non-determinism, with unit tests reserved for the deterministic parts, is what tests an agent properly.

Test the behaviour the system produces

The reframe that makes agent testing adequate is evaluating the agent’s probabilistic behaviour against task success rather than asserting deterministic outputs the agent does not reliably produce. Testing AI agents requires methods built for non-determinism, the evaluation harness that scores task success and trajectory, reserving unit tests for the deterministic parts of the system, because the deterministic assertion that unit tests depend on does not hold for a system whose output is a distribution. Unit tests on an agent’s non-deterministic behaviour either fail spuriously or assert nothing.

The teams that test agents properly are the ones that recognised where unit tests do not fit, evaluated the non-deterministic behaviour with the evaluation built for it, kept unit tests for the deterministic parts of the system, and extended the testing into production. Unit tests assert deterministic outputs and measure output equality, and an agent behaves probabilistically and is correct when it accomplishes the task, which unit tests neither hold for nor measure. Testing the behaviour against task success, with unit tests reserved for the deterministic scaffolding, is what tests an agent for what actually matters, which is the coverage that unit tests alone, built for deterministic systems, cannot provide.

_______________________________________________________________________