Your Agent Is Silently Degrading. Here Is How to Catch It
Published 16 May 2026. Updated 6 November 2026: added a section on testing a new model inside your own agent before you switch, based on Arize's 5 November write-up and the GVS5H paper it tested; every figure in the post was re-sourced or removed, and the framework descriptions were re-checked against each project's own documentation on 6 November 2026.
To evaluate an AI agent, you give it a fixed set of realistic tasks, run each task several times, and grade two things: the outcome it leaves behind (did the refund actually get issued, do the hidden tests pass) and the transcript of how it got there (which tools it called, in what order, at what cost and latency). Anthropic's guide to agent evals puts the starting size at "20-50 simple tasks drawn from real failures", which is small enough to build this week and large enough to catch the regressions that matter early on.
The failure these evals exist to catch is rarely a crash. It is quiet degradation: a model update, a prompt edit or a tool whose API changed shifts behaviour a little, adoption flattens over a few weeks, and nobody can say why. Teams without evals find out from support tickets. Teams with evals find out from a failed build.
The same suite answers a second question that comes up every few months, which is whether to move to a newer or cheaper model. A paper can show a cheap model closing the gap to a frontier one; only your own traces show what the switch costs in your agent, how long your users will wait, and whether your grader was right in the first place.
Agent evals vs model evals: why the difference matters
A model eval is one step: an input goes in, an output comes out, and a grader checks the output. An agent eval grades a loop. The agent plans, calls a tool, reads the result, adjusts, calls another tool, and eventually stops, and every one of those steps can go wrong in a way the final answer hides. An agent can reach the right answer through three unnecessary tool calls and a misread result, which means it got lucky, and luck does not survive the next prompt change.
Errors also compound. A wrong assumption at step two corrupts steps three through ten, and a single score at the end tells you that something failed without telling you where.
Anthropic's post settled the vocabulary most teams now use, and adopting it changes how you design the suite:
- A task is one test case with defined inputs and success criteria.
- A trial is one attempt at a task. Model outputs vary between runs, so you run several trials per task.
- A grader is logic that scores some aspect of a trial; a task can have several.
- A transcript (or trace) is the complete record of a trial: outputs, tool calls, reasoning and intermediate results.
- The outcome is the final state of the environment. A flight-booking agent can say "your flight has been booked"; the outcome is whether a reservation exists in the database.
Nondeterminism is the part that changes the arithmetic. Running the suite once is a single sample from a distribution, so a pass on Monday and a fail on Tuesday can both be true of the same agent.
What to measure: five dimensions
Correctness alone is the usual shortcut. These five dimensions fail independently of each other, which is why a single accuracy number keeps hiding problems.
Correctness of the outcome. Did the agent complete the task, judged by the end state, not by what it says it did. For coding agents this is unit or hidden tests; for a support agent it is a state check (is the ticket resolved) plus a rubric.
Trajectory and tool use. Did the agent pick the right tools, pass the right arguments, follow its own plan and avoid wasted steps. DeepEval splits this into trajectory metrics (Task Completion, Step Efficiency, Plan Adherence, Plan Quality) and component-level metrics (Tool Correctness, Argument Correctness), which is a useful way to think about it even if you use another tool. Long loops make this dimension expensive to ignore: Manus wrote that a typical task in its own agent takes around 50 tool calls, and each of those is a place for the run to drift.
Cost per completed task. Track tokens and dollars per successful completion, not per call. Architecture moves this number more than most teams expect; in Anthropic's multi-agent research post, the company reported that in its own data agents used about 4 times the tokens of chat interactions and multi-agent systems about 15 times.
Latency. Time from the user's input to a finished result, plus time to first token for interactive agents. Set a target before you test anything, because a cheaper setup that misses the latency target is not an upgrade, as the model-swap test below shows.
Safety and scope. Did the agent stay inside its authorised scope, refuse out-of-scope requests and hold up against adversarial input such as prompt injection through a tool result. These cases need their own tasks; they do not show up in a suite built only from happy paths. Red-teaming tools such as Promptfoo generate adversarial inputs for this.
How to evaluate an AI agent, step by step
1. Write 20 to 50 tasks from real failures. Start with what you already check by hand before a release, then go through the bug tracker and the support queue and turn user-reported failures into tasks. Mix happy paths, edge cases and failures you have already seen. Each task needs an input, a reference outcome and success criteria two engineers would grade the same way. "The agent should work" is not a criterion; "the agent retrieves the customer's orders from the past 30 days and returns a CSV with order ID, date and total" is.
2. Pick a grader for each check. Anthropic groups graders into three kinds. Code-based graders are deterministic: does the code pass the tests, does the record exist, how many turns did it take. Model-based graders (an LLM judge with a rubric) handle tone, helpfulness and open-ended answers. Human graders are slow and expensive, so use them to calibrate the model-based graders and to review samples, not to grade everything. Use code wherever a check can be computed; every check you move from a judge to code removes a source of noise.
3. Run several trials per task. Three to five trials per task is a reasonable floor for a working suite. Then choose the metric that matches the product. pass@k asks whether at least one of k trials succeeded, which suits tools where one good answer is enough. pass^k asks whether all k succeeded, which suits customer-facing agents where users expect the same behaviour every time. The two diverge fast: Anthropic's example is an agent with a 75% per-trial success rate, whose chance of passing three trials in a row is (0.75)³, about 42%.
4. Log every transcript and read some of them. Scores tell you what moved; transcripts tell you why. Store the full trace for every trial, and read a sample of failures each week. Anthropic closes its guide on exactly this advice: read the transcripts.
5. Baseline before you change anything. Run the suite on the current agent and save the result. Every later number is a comparison against this one.
6. Version the whole stack per run. Record the model, prompts, tool definitions, data snapshot, environment, budgets and grader versions with every run, as Unite.AI's guide recommends. Without that, a score change has no cause you can point to.
7. Gate every change in CI. Run the suite on every code change, prompt change and model change, compare against the baseline, and block the merge when a key metric regresses past a threshold you set in advance. Anthropic separates two suites here: capability evals, which start at a low pass rate and measure what the agent cannot do yet, and regression evals, which should pass at close to 100% and exist to catch backsliding. Capability tasks that the agent passes reliably graduate into the regression suite.
8. Feed production failures back. Every failure a user finds becomes a task. Over time the suite stops being a list of what you guessed would break and becomes a record of what did.
9. Watch for saturation. When the agent passes every solvable task, the suite stops telling you anything. Anthropic's example, from its January 2026 post, is SWE-Bench Verified, where scores started at 30% and frontier models now sit above 80%. When your capability suite saturates, write harder tasks.
Test the eval before you trust it
A grader is code, and code has bugs. A broken grader adds more than noise; it penalises whichever model or setup happens to trip the bug, which can flip a decision.
Three cases from this year show the size of the effect:
- A harness bug worth up to 12.6 points. The GVS5H paper (Gao, Khosrowshahi and colleagues at Persis Capital) found that the upstream LiveCodeBench grader mocked
sys.stdin.buffer.readlineso that it returned the first line of input every time, failing correct programs that read input that way. Because only some models write in that style, the authors report the correction moved one arm by +12.6 points and left Claude Fable 5, which never uses the idiom, unchanged. - An exact-match checker on a problem with many valid answers. In Arize's model-swap test, problem arc191_c accepted any valid answer, but the grader compared output character for character. The agent worked out that its correct answers were being marked wrong and started hard-coding the sample outputs; an LLM judge flagged that as gaming the tests. Fixing the checker made all four setups pass the problem and removed one of the nine "production failures" the test dataset had been built around.
- Rigid grading on a public benchmark. Anthropic reports that Claude Opus 4.5 first scored 42% on CORE-Bench, until a researcher found grading that penalised "96.12" when it expected "96.124991…", along with ambiguous and irreproducible tasks; after fixing those and using a less constrained scaffold, the score was 95%.
The checks that would have caught each of these are cheap. Run known-good solutions through the grader and confirm they pass; Arize ran all 97 known-good solutions shipped with GVS5H and checked that 43 of the 47 cases the paper says flip between graders flipped in its own setup too. Keep two versions of a grader when you fix one, so you can see what the fix moved. When there is no hidden test to grade against, as with a support or research agent, an LLM judge carries the correctness check, and that judge needs its own agreement check against human labels; we cover how to measure a judge's consistency and cost in AI agent evals: the judge is the part nobody measures.
How to test a new model in your agent before you switch
The short answer: run the candidate and your current model on the same dataset drawn from your own production traces, score both with the same evals you run in production, and switch only if the candidate meets the goals you wrote down before the test on correctness, cost per solved task and latency, with the gap on correctness larger than run-to-run noise.
Arize published a worked example of exactly this on 5 November 2026. The author tested a claim from the GVS5H paper: that wrapping a small, cheap model in a manager/worker loop closes the gap to frontier models on hard coding problems. In the paper, which reports five passes over the 100 newest hard LiveCodeBench problems with a 128k output cap and reasoning on, Qwen3.8-Flash-Next scored 84.2% as a single call and 93.0% inside the loop, against 90.4% for Claude Fable 5 as a single call. The authors put the cost of one pass over the 100 problems at $5.76 for the Qwen loop and $61.11 for Fable 5, using list prices with no caching or batch discount. These are the paper authors' own figures. The same abstract notes that "gains are not universal as some models are unchanged or worse."
The method Arize used has seven steps, each feeding the next:
- Define what good looks like. Write the bar before the test. Arize's was: cheaper per solved problem, no noticeable loss in speed (each task under 15 seconds), and the same or better share of problems solved.
- Trace the agent. Every production request becomes a trace with its input, output, model calls, tokens, cost and timing. Put everything an eval needs into the root span's output so evaluators and judges do not have to read every child span.
- Run evals on those traces. The rule that makes the comparison valid is that the evals you run on production traces are the same evals you run on experiments. Otherwise you cannot tell a model change from an eval change.
- Build a dataset from the traces. Mix failures, because they show where a new model might help, with passes, because they guard against regressions. Arize used the nine production runs its current model failed plus 21 random passes, 30 problems in all. Hidden tests stayed inside the grader's sandbox and never entered the dataset, so the agent could not pass by matching outputs.
- Run each candidate as an experiment. Same agent code, only the model and the loop changed, with the current model re-run fresh as the control rather than reused from production numbers.
- Compare pass rate, cost per solve and latency against the goals from step 1.
- Read the traces behind every difference, because the numbers alone can mislead.
Arize reported the results below. These are vendor-reported numbers from one pass over 30 problems in the author's own LiveCodeBench-based agent, with the current model, Claude Opus 5.5, called directly and Qwen3.8 Flash called through OpenRouter, both at high reasoning effort with a 128k output cap and loops capped at 10 rounds:
| Setup | Solved (of 30) | Cost per solve | Median latency | Runs under 15 s | Model calls per problem |
|---|---|---|---|---|---|
| Opus 5.5, single call (control) | 24 | $0.12 | 21 s | 9 | 1 |
| Qwen3.8 Flash, single call | 22 | $0.04 | 20 min | 0 | 1 |
| Qwen3.8 Flash + Ralph loop | 22 | $0.07 | 12 min | 0 | 1.5 |
| Qwen3.8 Flash + GVS5H manager loop | 23 | $0.10 | 37 min | 0 | 5.7 |
The table carries lessons that hold well beyond coding agents.
Price per token is not cost per answer. As of 6 November 2026, Qwen3.8 Flash lists at $0.15 per million input tokens and $0.47 per million output tokens on OpenRouter, and Claude Opus 5.5 at $4 and $20 on Anthropic's pricing page, so per token Qwen is roughly 30 to 40 times cheaper, as Arize notes. Per solved problem, the manager loop came out about 15% cheaper, $0.10 against $0.12, because it made 5.7 model calls per problem and reasoned at length on each one. Only an experiment tells you what an answer costs.
One pass on 30 problems cannot separate one solve. Arize's own reading is that the loop got Qwen "close to Opus", not level with it, because a one-solve gap on one pass is noise. The paper ran five passes per setup and reports confidence intervals; for a real switch decision, do the same.
The traces changed the conclusion twice. An LLM judge reading the round-by-round record found that 25 of 30 manager-loop runs had a single coding round, because the first code already passed the public tests, so the extra spend went on planning rather than fixing. And five Qwen single-call runs that returned no code looked identical in the pass-rate column, but the traces showed three runaway generations that hit the 128k cap, one rate-limit error from OpenRouter and one dropped connection. Without the traces, the network's failures would have been charged to the model.
Arize's verdict was not to switch: the loop saved about two cents per solved problem while the median wait went from 21 seconds to 37 minutes, a bad trade for anything a person waits on and a possible one for unattended batch work, provided the experiment is rerun on that workload. The paper's numbers were not wrong; they answered a different question, on a different harness, at a different latency budget.
The decision rule we would use, built on Arize's method:
- Write the goals down first: a correctness bar (same or better pass rate, or pass^k for agents that must be consistent), a cost-per-solved-task bar, a latency bar such as a median or p95 target, and any safety checks that must not regress.
- Run the control and every candidate fresh on the same trace-derived dataset with the production evals, for several passes.
- Switch only when the candidate clears every bar, and when the correctness difference is bigger than the spread you see between passes of the same setup. A cheaper model that is within noise on correctness and inside the latency target is a switch; one that is within noise but misses latency is not.
- Before acting, open the traces behind every row that moved, and rule out grader bugs and infrastructure failures.
Two related points matter when the swap involves more than the model. When the candidate comes with a new loop around it, you are changing the harness as well as the model, and the harness sets much of the bill; our post on what an agent harness is covers why. And the same method applies when the model you are choosing is the one that writes an agent's memory rather than the one that answers; we walk through that case in which model should write agent memory.
Which eval framework fits, as of 6 November 2026
Pick the tool after you know what you need to grade, not before. Each description below was checked against the project's own documentation on 6 November 2026.
| Tool | What it is today | Fits best when |
|---|---|---|
| DeepEval | Open-source eval package that plugs into Pytest (deepeval test run), with 50+ ready-made metrics, including agent trajectory and tool-call metrics | You want evals that look like unit tests and run in your existing CI |
| Ragas | Library built around experiments and custom metrics, with dataset management and integrations with LangChain and LlamaIndex | Retrieval quality is the bottleneck, or you want to write your own metrics with decorators |
| Promptfoo | Open-source CLI and library for evals and red teaming; its docs say evals run on your machine and call the model directly. OpenAI announced it would acquire Promptfoo in March 2026 (Techzine) | You want declarative prompt and model regression tests plus adversarial testing, run locally |
| Braintrust | Hosted platform that now describes itself as "active observability" for agents: traces, experiments, an AI assistant (Loop) that analyses data and builds evals, an MCP server and a bt CLI | You want production traces and offline experiments in one place |
| Opik | Open-source, self-hostable platform from Comet for traces, evals and experiments, with integrations including LangChain, LangGraph, CrewAI, AutoGen, OpenAI Agents, Pydantic AI and Mastra | You want tracing plus evals and need to self-host |
Two things changed between May and November: DeepEval's metric library has grown from the 20-plus we cited then to more than 50, and Ragas now presents itself as an experiments-first library for LLM applications in general. Whichever you pick, the hard parts stay the same: good tasks, graders you have tested, and enough trials to see past noise.
Agents that remember
An agent whose answers depend on what it learned in earlier sessions adds a failure the steps above do not cover: a run can fail because the memory it retrieved was stale, contradicted or belonged to another user, while the model and tools behaved perfectly. Those tasks need multi-session histories with a known answer key, and graders that check the store, the retrieval and the final outcome separately so you can tell which layer failed. Our one-week test plan for agent memory sets out the cases and a pass bar for each.
Mistakes that kill reliability
Grading only the final answer. You lose the root cause, and you reward lucky runs. Grade the outcome, log the transcript, and read samples.
Running evals once at launch. Models update, tool APIs change and prompts drift. A suite that does not run on every change is a snapshot of an agent that no longer exists.
Trusting a grader nobody tested. Run known-good answers through every grader, check judges against human labels, and keep the old grader beside the fixed one so you can see what changed. The GVS5H and CORE-Bench cases above are what happens otherwise.
Measuring everything and acting on nothing. A dashboard of 50 metrics gets read by nobody. Pick the few that map to the product (usually outcome correctness, cost per solved task, a latency percentile and a safety gate), set thresholds, and make a breach block a deploy or page someone.
Frequently asked questions
How many test cases do I need to evaluate an AI agent?
Twenty to fifty tasks drawn from real failures is enough to start, per Anthropic's guidance; early on each change has a large effect, so a small set detects it. Mature agents need larger and harder sets to detect smaller effects, and any comparison that comes down to one or two tasks needs several passes before you act on it.
What is the difference between pass@k and pass^k?
pass@k is the probability that at least one of k trials succeeds, and it rises as k grows. pass^k is the probability that all k trials succeed, and it falls as k grows. Use pass@k when one good answer is enough and pass^k when users expect the same result every time.
Should I use an LLM judge or code-based graders?
Use code wherever the check can be computed, such as tests passing, a record existing or a turn count, and use an LLM judge for what code cannot check, such as tone or whether a loop made real progress. Calibrate every judge against human labels before it decides anything.
How do I know whether a cheaper model is good enough for my agent?
Build a dataset from your own production traces that mixes failures and passes, run the cheaper model and your current model on it with the same evals you use in production, and compare pass rate, cost per solved task and latency against goals you set in advance. Read the traces behind any difference before switching.
Can I trust published benchmark results when choosing a model?
Use them to decide what to test, not what to ship. The GVS5H paper's loop beat a frontier model on accuracy at a fraction of the cost per pass, yet Arize's test of the same idea in a single agent found a two-cent saving per solved problem and a median wait of 37 minutes instead of 21 seconds.
How often should agent evals run?
On every change to code, prompts, tools or model, as a CI gate, plus continuously on a sample of production traces with the same evaluators, so that drift you did not cause, such as a provider-side model update or a changed tool API, shows up too.
Build this tomorrow
Write 20 tasks from the failures you already know about, grade the outcome with code wherever you can, run each task three times, and save the result as your baseline. Put it in CI with a threshold that blocks the merge. Then run your known-good answers through your graders, because the first bug you find may well be in the eval rather than the agent.
The payoff shows up the next time a cheaper model or a new loop looks good in a paper: instead of weeks of manual checking, you run one experiment on your own traffic and know by the end of the day what the switch would cost in money, in seconds and in answers.
