AI Agent Evals: The Judge Is The Part Nobody Measures

An LLM judge scoring the same agent output three times will often hand back three different answers. On MT-Bench, Rating Roulette measured intra-rater reliability across repeated runs at a Krippendorff's Alpha of 0.265 for Llama 3.1, 0.507 for DeepSeek-R1 and 0.563 for Qwen-3, against a conventional floor of 0.8 for acceptable reliability. The paper, presented at EMNLP 2025 Findings, puts it plainly: LLM judges have low intra-rater reliability in their assigned scores across different runs, which makes their ratings "inconsistent, almost arbitrary in the worst case."

Almost every guide to agent evaluation published this year covers rubric design, scoring techniques and metric taxonomies. Very few of them ask whether the judge doing the scoring produces the same number twice, and none of them work out what it costs to run that judge on everything rather than on a sample. Both gaps matter more than the metric taxonomy does, because a suite built on an unstable judge measures its own noise, and a suite that only ever sees 2% of traffic is blind to the other 98% by construction.

Both properties turn out to be fixable, and fixing them changes what an eval suite can be. It starts with dropping the assumption that the judge has to be a language model at all.

Why an unstable judge is worse than no judge

An eval suite exists to answer one question: is today worse than yesterday. Everything else is in service of that.

Run a judge twice over the same fifty traces and get a pass rate of 82% and then 76%, and the six-point gap tells you nothing. Maybe the agent regressed. Maybe the judge felt differently. You cannot separate the two without re-running enough times to average the judge out, which multiplies your bill by however many runs you need.

What happens next is predictable, and I have watched teams walk through every step of it. Somebody raises the failure threshold so the suite stops going red on noise. The threshold goes up again a month later. Eventually the suite passes on everything except catastrophic breakage, at which point it has become a very expensive smoke test that nobody reads. The suite did not fail loudly; it degraded into decoration.

A second failure mode is subtler and worse. A judge with high run-to-run variance on a small number of questions will produce an eval set where a handful of traces flip verdict every run. Those flipping traces get investigated, found to be fine, and marked as flaky. Real regressions that land on the same borderline traces then get dismissed under the same label.

"Set temperature to zero" does not fix this

Every engineer who has run an eval suite reaches for this within about four seconds of reading the paragraph above, and it deserves a serious answer rather than a dismissal.

Temperature zero is real and it does real work. It removes sampling randomness from token selection, and for an extraction task with one defensible answer it will give you the same output nearly every time. Several of the better guides recommend it correctly; TestMu AI{target="_blank"} specifies temperature 0.0 in its calibration workflow for exactly this reason, and that is sound advice as far as it goes.

What it does not touch is the harder half. Temperature constrains how a token is picked from a distribution; it does not make a subjective judgement less borderline. When a trace genuinely sits between "grounded" and "partially grounded", the underlying distribution is close to even, and greedy decoding at temperature zero will still tip one way or the other on tiny differences in context, ordering, or the phrasing of the preceding question in a batch. Rating Roulette ran its three judge models across three benchmarks and found reliability well below threshold; that variance is a property of judgement on ambiguous inputs, not of the sampling parameter.

There is a stronger version of the objection worth handling too. Patronus AI{target="_blank"} recommends running the same input through the judge three times as a self-consistency check, which is genuinely good practice and the only place in the mainstream guidance where this problem is acknowledged as an operational concern. It is also a three-times cost multiplier on a judge that was already the expensive part of the pipeline, which brings us to the second unmeasured thing.

Nobody models what grading actually costs

Of every major guide published on this topic in the past year, exactly one publishes a real number. The AWS tutorial on dev.to{target="_blank"} gives roughly $0.001 to $0.003 per evaluation on a small model and $0.015 to $0.045 on a mid-sized one. Useful, and it stops there. No page I could find takes that per-call figure and multiplies it by production volume, which is the only form in which the number means anything.

So here is the multiplication, using a mid-sized judge at the middle of that published range and a 14-question rubric run once per trace.

At roughly $0.03 per trace, grading one million agent traces costs about $30,000. A reasoning-model judge, which is what most teams reach for when rubric quality matters, is worse: TypeSafe's own comparison study{target="_blank"} measured Claude Opus 4.8 in reasoning mode at $0.034275 per query on a 14-question rubric, which lands at about $34,000 for the same million traces. Serially, at 13.9 seconds per call, that run takes roughly 160 days of wall clock.

Nobody spends $34,000 and five months to grade last month's traffic. So nobody does. Instead the guidance converges on sampling, and the sampling rates are stated without any justification at all: the AWS tutorial suggests 1 to 5% of live traffic, Adaline{target="_blank"} suggests 5 to 10% for judge-based checks. Neither explains where the number came from, because it did not come from anywhere. It came from the bill.

Sampling at 2% means a failure mode occurring in one conversation per thousand shows up in your eval data roughly once per fifty thousand conversations. You will find it eventually, from a customer, in a support ticket.

What changes when the judge is not a language model

A grading task has a peculiar shape. The judge never needs to write anything. It needs to answer a fixed list of closed questions: was this grounded, was the right tool called, did the agent refuse when it should have, how directly did it answer. Each of those is a yes-or-no, a pick-one, or a rating on a defined ladder.

Which means text generation, the expensive and unstable part, is pure overhead in a judge. You pay for output tokens you throw away, you wait for them, and you inherit their variance.

TypeSafe{target="_blank"} released a model in September 2026 built on that observation. Jev returns typed decisions and probabilities instead of text, across three question types: a yes-or-no probability, a pick-one from a named set with a full probability distribution, and a rating against two to ten described levels. It cannot write a sentence. That is the design.

Operating numbers, taken from the vendor's own published measurements with their method attached rather than from the launch-week coverage that rounded them:

Method, because a benchmark without one is a marketing claim: 15 repeated samples of a 14-question yes-or-no rubric against a single deliberately borderline insurance claim, sampled 11 September 2026, with the TypeSafe model resolving to jev-1.13.0. Published pricing at the time was $0.042 per million input tokens with output tokens free.

Grading a million traces on those numbers costs roughly $43.

That is the whole argument, and it is not really an argument about money. At $34,000 you sample and you argue about the rate. At $43 the rate stops being a question, and an eval suite becomes something you can point at all of production, continuously, without anyone having to approve it.

Reproducibility matters as much as price does here. Across those same 15 repeats, mean per-question probability standard deviation came in at 0.0102 for yes-or-no questions and 0.0098 for pick-one questions. On a hard content-moderation case scored with eight pick-one questions, raw label agreement across repeats was 90.8%.

Reliability is not validity, and this is where most of the excitement should stop

A model that agrees with itself is not thereby correct, and the strongest evidence on this arrived in June 2026 from a UC Berkeley team. Reliability without Validity{target="_blank"} ran 21 judges from 9 providers across three benchmarks, producing around 541,000 individual judgments over 118 evaluation runs, and found that high test-retest reliability of 0.95 or better coexisted with severe position bias above 0.10 in more than one model. A judge can be perfectly repeatable and repeatably wrong.

TypeSafe's own documentation says the same thing about its own model, which is more than most vendors manage. Their study of repeated pick-one decisions carries the sentence "This experiment does not measure accuracy," and describes its own uncertainty threshold as "an illustrative application policy, not a calibrated guarantee or a threshold chosen to maximize this run's agreement." The same study concedes that a probability near the threshold can still move between a concrete label and uncertain, and that the policy does not make the model deterministic.

So the honest claim is narrower than the launch coverage suggested. A typed-decision judge removes one specific failure mode, the one where your measurement instrument moves under you. It does not tell you the instrument is pointed at the right thing. You still have to check, and there are two halves to that check: which biases carry over to a typed judge, and whether its confidence number means anything on your data.

The biases survive the change of judge, mostly

Four biases get named repeatedly in the existing guidance, and they are worth separating by whether a typed judge inherits them.

Position bias, where a judge favours whichever option it saw first, is inherited. The Berkeley study found it ranging from 0.002 for Gemini 2.5 Pro to 0.192 for Qwen 3 8B, nearly two orders of magnitude apart, so it is a per-model property rather than a universal constant. TypeSafe's own documentation concedes the equivalent: sibling order inside a pick-one question is part of the question rather than presentation. Shuffle your option order across runs and measure whether the answer moves.

Verbosity bias, where longer answers score higher, turns out to be weaker than folklore suggests. All 21 judges in the Berkeley cohort registered a Pearson correlation below 0.011 with answer length, substantially lower than earlier estimates. Worth testing on your own rubric rather than designing around.

Self-preference, where a judge rates its own family of models more kindly, does not apply to a judge that generates nothing and therefore has no outputs of its own in the comparison set. The Confident AI guide{target="_blank"} documents the effect for language-model judges and is the best summary of it I have found.

Rubric sensitivity is the one that gets worse, not better. A typed judge takes your level descriptions extremely literally, which is a feature until two levels overlap, at which point confidence collapses and you get an honest 0.4 instead of a confident guess. Treat persistently low confidence on one question as a defect in the question.

Hand-label 200 traces yourself, or with whoever owns quality. Run the judge over them. Plot confidence against accuracy and see whether the confidence number carries information. TypeSafe's own SEC filing classification exercise is a good worked example of what a passing result looks like: over 60 filings, answers above 0.9 confidence were correct 27 times out of 30, and answers below that line were correct 12 times out of 30. The confidence score was doing real work, and they only knew that because they measured it.

Berkeley's team gives the sharper warning: every judge in their study overstated chance-corrected agreement, with exact match inflating Cohen's kappa by between 33.8 and 41.3 percentage points on MT-Bench. If you validate your judge by counting how often it matches a human, you are measuring the wrong thing, and you are measuring it generously.

Three verdicts, not two

Every guide surveyed for this piece scores on a binary pass or fail, a 0 to 1 scale, or a 1 to 5 integer. Not one proposes an explicit third outcome, and that omission causes a measurable amount of the noise people attribute to the judge.

Some traces are genuinely ambiguous. Forcing a label onto them manufactures disagreement that has nothing to do with the agent and everything to do with the trace sitting on a boundary.

def verdict(probability, low=0.30, high=0.70):
    if probability < low:
        return "fail"
    if probability > high:
        return "pass"
    return "needs_human"

Applied to that eight-question moderation rubric, the uncertainty band took raw label agreement from 90.8% to 99.2%, with 74.2% of answers still decided automatically and 25.8% routed for review. Three quarters of the work happens without a person, near-perfectly reproducibly, and the quarter that a machine should not be deciding alone gets flagged as such instead of being guessed at.

Two things make this work in practice. Pick-one questions always return something, because the probabilities have to sum to one, so on a trace where no category applies the model still nominates a winner; pairing the pick-one with a separate yes-or-no asking whether any option applies at all gives you a gate that can go to zero. And when several checks feed one decision, aggregate on the maximum rather than the mean, since one confident red flag is the whole signal and averaging is how you hide it.

Band edges are not universal. Set them from labelled examples and from what a wrong automatic decision costs you relative to the price of a human looking.

Grading before the response ships

Once a grade costs a tenth of a second, it can run before the user sees anything rather than after.

Not one page in the incumbent set mentions this, and it is the most useful structural change available. The shape is a cascade: generate with your normal model, grade the draft response inline, and escalate only when a check fires. TypeSafe documents the pattern for structured extraction, where a cheap model extracts, a per-field check asks whether each value looks wrong, and an expensive reasoning model is invoked only when any field crosses 0.7. They report quality comparable to the top-tier model at roughly a tenth of the cost.

Pointed at agent output rather than extraction, the same shape looks like this:

draft = cheap_agent.respond(request)
checks = judge.system_one(
state={"request": request, "draft": draft, "retrieved": context},
questions=INLINE_RUBRIC,      # grounding, scope, leakage, refusal correctness
)
if max(checks.flags.values()) > 0.70:
return strong_model.respond(request)   # pay only when the check fires
return draft

Your quality floor stops being whatever the cheap model does on a bad day, and your bill stops being whatever the expensive model costs on a good one. The gate has to be fast enough to hide inside the response budget, which is the entire reason a 111 ms judge makes this possible and a 13.9 second one does not.

What still has to be right: trajectories, tools and calibration

None of the above removes the work the existing guides describe well, and a piece that skipped it would be selling you a judge without a rubric.

Agent evaluation is trajectory evaluation. A correct final answer reached by calling four unnecessary tools, failing twice and recovering by accident is not a pass, and the Confident AI agent guide{target="_blank"} is right that tool correctness, argument correctness and step efficiency need scoring separately from task completion. Feed the whole trajectory as a named object and ask about the path:

state = {
"user_request": "Book a table for four on Friday",
"tool_calls": [{"name": "search_restaurants", "args": {...}}, ...],
"retrieved_context": [...],
"agent_final_reply": "...",
}

TRAJECTORY_RUBRIC = {
"correct_first_tool": Noul(
instructions="Was the first entry in tool_calls the right tool for user_request?"),
"unnecessary_calls": Score(
instructions="How many entries in tool_calls were not needed for user_request?",
criteria=["Every call was needed",
"One or two calls were not needed",
"Most calls were not needed"]),
"recovered_from_failure": Noul(
instructions="After the first failed call in tool_calls, did the agent take a reasonable next step?"),
"grounded": Noul(
instructions="Is every factual claim in agent_final_reply supported by retrieved_context?"),
}

Every question in that request is scored independently against the same state and runs in parallel, so a forty-question rubric costs roughly what a four-question rubric costs. TypeSafe measured 13 questions against a 54,000-character document as 12.2 times cheaper and 10 times faster batched into one call than sent as 13 separate calls, with answers unchanged. The usual reason rubrics stay thin, that each question is another paid round trip, stops applying, and you can afford to check the rare failure modes on every trace instead of only the common ones.

Calibration is the other piece worth taking from the existing guidance rather than reinventing. Patronus recommends targeting a Cohen's kappa above 0.8 against expert labels; TestMu specifies a calibration set of 30 to 50 samples, measuring precision and recall, then freezing the judge. Both are right, and both apply unchanged to a typed judge. A frozen, version-pinned judge with a measured kappa is the artifact you want; everything in this piece is about making that artifact cheap enough to run everywhere.

Pin the version explicitly. In TypeSafe's own published study the alias jev-latest resolved to jev-1.13.0, and a judge that silently upgrades makes last quarter's scores incomparable with this quarter's.

Where the suite actually runs

Most eval suites run nightly because that is what they can afford. Cost decides placement, and once it stops deciding, three placements open up at once.

A regression gate in continuous integration is the first and the easiest to justify. A 500-trace suite at a tenth of a second per trace finishes in well under a minute and costs a couple of cents, which makes it a pull request check rather than a scheduled job. DeepEval{target="_blank"} and Adaline both describe the assertion pattern for wiring this into a pipeline, and it works unchanged with a typed judge behind it. An eval that runs before merge stops the regression; an eval that runs at 3am documents it.

Continuous scoring of production traffic is the second, and it produces a different artifact from a pass rate. A quality signal computed on every request can be cut by customer, by prompt version, by model and by release, and alerted on when the distribution moves rather than when a threshold trips. That is monitoring, not testing, and the distinction is worth holding onto: a test answers whether the build is good, while a monitor answers whether reality has changed.

Inline gating, described above, is the third, and it is the one that changes the product rather than the process.

The thing a typed judge cannot do

It cannot tell you why.

You get 0.31 and no sentence. For the engineer staring at a red suite at 5pm, the explanation is most of the value, and any honest account of this approach has to lead with that limitation rather than bury it.

You do not have to pick one judge. Grade everything with the cheap typed judge, then spend a reasoning model only on the traces that failed or landed in the uncertain band, asking it to write the diagnosis. On a suite where 74% pass cleanly, you are paying reasoning-model prices on roughly a quarter of traces and getting a written explanation exactly where a human was going to look anyway. Cheap where the work is wide, expensive where it is deep.

Two smaller limits are worth knowing before you design around them. State and questions share a budget of roughly 32,000 tokens, about 150,000 characters, so long trajectories with large retrieved context will not fit in one call and need grading per turn. And a pick-one question accepts at most 255 options, which matters the moment you try to classify against a large taxonomy.

What I would actually build

Start with the judge you have and measure it, because everything else is downstream of knowing whether your current numbers mean anything. Run your existing rubric five times over the same fifty traces and compute the variance per question. If it is small, your suite is fine and you can stop reading. If two questions account for most of the movement, those two questions are probably asking about more than one thing at once.

Then hand-label 200 traces and calibrate whatever judge you intend to keep, plotting confidence against accuracy rather than counting agreement.

Add the third verdict before you change anything else. It is a four-line function, it costs nothing, and on the published numbers it absorbs most of the residual disagreement into an honest label.

Move the cheap judge to 100% of production traffic once it is calibrated, and keep the reasoning model for explanation duty on the failures. The point of grading everything is not thoroughness for its own sake; it is that a quality signal computed on all of production can be broken down by customer, by prompt version, by model, and alerted on when the distribution shifts. That is a different artifact from a nightly pass rate.

We grade memory recall for Maximem Synap on this shape, because memory quality is exactly the kind of property that degrades quietly and shows up in a support ticket rather than in a test. An eval that runs on a sample once a night finds that class of regression late, every time.

The part worth sitting with

For two years the eval tooling conversation has been about frameworks and metric taxonomies, and those frameworks are good. The judge underneath them has been treated as a solved component, and two recent studies say it is not, one published at EMNLP 2025 Findings and one a June 2026 preprint from UC Berkeley: judges disagree with themselves across runs, and the agreement statistics most teams use to validate them overstate the case by 30 to 40 percentage points.

Fixing that is not a matter of a better prompt. It is a matter of asking whether the thing doing the grading should have been a text generator in the first place, given that it never needed to write anything. When the answer is no, the price falls far enough that the eval stops being a test you run and becomes a measurement you always have. That changes what you can know about your agent, which is the only reason any of this exists.