LLM-as-a-Judge: Where Automated Evaluation Works and Where It Lies

If you evaluate agents with open-ended outputs, LLM-as-a-judge is often the only feasible way to assess them at scale, and its feasibility is exactly what makes it dangerous, because a judge that produces a quality score for every output looks like a solution and can be quietly wrong. The judge’s convenience, a scalable quality signal, comes with the judge’s own unreliability, and treating the judge’s scores as truth without validating them is trusting an unvalidated non-deterministic evaluator. Understanding where LLM-as-a-judge works and where it lies is what lets you use it for the scale it provides without being deceived by the errors it carries.

Why LLM-as-a-judge is necessary

Evaluating open-ended outputs is hard because there is often no mechanical check for whether an output is good. A deterministic outcome can be checked with an assertion, but an open-ended output, a generated response, a summary, a piece of reasoning, has no single correct answer to match against, and its quality is a semantic property that a mechanical check cannot assess. This leaves two options for evaluating open-ended outputs: a human judges them, which is accurate and does not scale to the volume of evaluation an agent needs, or a model judges them, which scales and carries the model’s error. The human does not scale, and the volume of evaluation, on every change, on production samples, across many cases, requires scale, which is why LLM-as-a-judge becomes necessary.

The judge fills the gap between the need to evaluate open-ended outputs and the infeasibility of human evaluation at scale, providing a quality signal for outputs that mechanical checks cannot assess and human evaluation cannot assess at the required volume. This is what makes it a core tool of agent evaluation, because much of an agent’s output is open-ended and the evaluation has to operate at scale, and the judge is what makes evaluating that output at that scale feasible. Without the judge, evaluating open-ended agent outputs at scale is not feasible, which would leave the agent’s open-ended behaviour unmeasured, so the judge is necessary for the evaluation that keeps the agent’s open-ended behaviour assessed. The judge is necessary because it is the only feasible way to evaluate open-ended outputs at the scale agent evaluation requires, which is what makes it worth using despite the errors it carries.

Where the judge lies

The judge is a non-deterministic model, which means it has its own error rate, and the ways it errs are the ways it lies about the quality it is supposed to measure. The judge can be inconsistent, giving different scores to the same output on different passes, because it is non-deterministic like any model, which means its scores carry noise. It can be biased, systematically scoring certain kinds of output higher or lower in ways that do not reflect actual quality, favoring outputs that share characteristics with what it judges well regardless of their real quality. It can be wrong, simply misjudging an output’s quality, scoring a bad output well or a good output poorly, because judging quality is itself a task the model performs imperfectly.

These errors matter because they are invisible in the judge’s output, which is a confident score that looks like a measurement whether or not it is accurate. An unvalidated judge produces quality numbers that appear to measure the agent’s quality and may be measuring the judge’s biases and errors instead, which is worse than no measurement because it is a confident number that is trusted. A judge that systematically scores a certain kind of output too high gives false confidence in that output’s quality, and a judge that is simply noisy gives quality numbers with error bars that are not shown, and in both cases the judge lies by presenting an error as a measurement. This is the same false-confidence risk that any unvalidated evaluator carries, acute for the judge because the judge is used precisely where mechanical validation is not available, which is where its errors are hardest to catch. The judge lies where its errors, its inconsistency, bias, and misjudgment, are presented as confident quality scores that are trusted as measurements, which is the danger of using it without validation.

Validating the judge, and the trade-off

Using LLM-as-a-judge well requires validating the judge against human judgment, which is the discipline that separates a judge you can trust from one that lies. Validation means measuring the judge’s agreement with human evaluation on a set of outputs, calibrating the judge’s scores against human-labelled examples to establish how well the judge’s judgment matches human judgment, and treating the judge’s scores as reliable only to the degree that validation shows they agree with humans. A judge validated to agree well with human judgment can be trusted for the outputs it was validated on, and a judge not validated is a judge whose scores are unverified, which means the validation is what turns the judge from a source of confident-possibly-wrong numbers into a measured evaluator with known reliability.

The validation carries a trade-off, because it requires the human judgment the judge was meant to replace, and naming that keeps the validation realistic. Validating the judge means having humans evaluate a set of outputs to compare the judge against, which is human evaluation effort, seemingly defeating the purpose of the judge that was meant to avoid human evaluation. The resolution is that validation requires human evaluation on a sample, not on everything, which is feasible, and the validated judge then evaluates at scale, so the human effort goes into validating the judge rather than evaluating everything, which is the leverage the judge provides once validated. The validation also has to be ongoing, because the judge’s reliability can drift as the outputs it judges change, which means the judge’s agreement with humans has to be periodically re-measured, tying the judge to the continuous validation that keeps its reliability known over time. The trade-off is between the human effort of validating the judge and the deception of an unvalidated judge, and validating on a sample and monitoring the judge’s reliability over time is what makes the judge trustworthy for the scale it provides.

Where it almost broke

We evaluated an agent’s open-ended outputs with an LLM-as-a-judge, which gave us the scale we needed, quality scores for every output where mechanical checks were not possible, and we trusted the judge’s scores as measurements of the agent’s quality. The scores were lying in a specific way we discovered later: the judge was systematically scoring a certain kind of output higher than its actual quality warranted, a bias in the judge that inflated the quality numbers for those outputs, so the agent looked better on those outputs than it was, and we were making decisions on quality numbers that reflected the judge’s bias rather than the agent’s actual quality. The judge’s convenience had led us to trust its scores without validating them, and the scores had deceived us.

The correction validated the judge. We had humans evaluate a sample of outputs and measured the judge’s agreement with the human judgment, which revealed the bias, the judge’s scores diverging from human judgment on the kind of output it was inflating, and we recalibrated the judge and re-measured its agreement until its scores were validated to match human judgment adequately. We made the validation ongoing, periodically re-measuring the judge’s agreement as the outputs changed. The judge’s scores became trustworthy for the outputs they were validated on, because we now knew how well the judge agreed with humans and had corrected the bias that had been lying. The lesson was that LLM-as-a-judge provides the scale to evaluate open-ended outputs and lies when trusted without validation, and validating the judge against human judgment is what makes its scores measurements rather than confident errors.

Trust the judge only as far as you have validated it

The reframe that makes LLM-as-a-judge safe is trusting the judge’s scores only to the degree you have validated them against human judgment. LLM-as-a-judge provides the scale to evaluate open-ended outputs that mechanical checks and human evaluation cannot assess at volume, and it lies when trusted without validation, presenting its inconsistency, bias, and misjudgment as confident quality scores. The judge that lets you evaluate at scale is a non-deterministic evaluator with its own errors, and validation is what tells you which of its scores to trust.

The teams that use LLM-as-a-judge well are the ones that validated the judge against human judgment, measured its agreement on a sample, calibrated it until its scores matched human judgment, and monitored its reliability over time. The judge is necessary for evaluating open-ended outputs at scale and dangerous when its scores are trusted as measurements without validation, because its errors are invisible in the confident numbers it produces. Trusting the judge only as far as you have validated it, and validating it on a sample while letting it evaluate at scale, is what captures the judge’s leverage without being deceived by the errors it carries, which is the difference between automated evaluation that works and automated evaluation that lies.