The final answer can hide a failed run
An agent can return the right sentence after taking the wrong route. It may call the same tool three times, ignore a cheaper source, exceed a latency budget, or create a duplicate side effect before producing a plausible answer. A review that looks only at the final text marks that run as successful.
Google's 15 October 2026 guide to evaluating AI agents makes this failure mode explicit. A polite answer can conceal loops, repeated tool calls, exhausted quotas and unnecessary token use. Its practical recommendation is to test realistic scenarios, record complete execution traces and replace ad hoc spot checks with a repeatable pipeline.
That is a useful starting point, but a test suite is not yet a release decision. A team needs a contract that says which scenarios matter, what must be true at the end, which routes are forbidden, how much work is acceptable and which regression stops deployment.
Define the scenario before the grader
Start with a scenario, not a prompt. A prompt is only one input. The scenario should define the initial state, the user's goal, the tools and data available, the external systems the agent may change, and the expected end state.
For a support agent issuing a refund, the expected answer is not enough. The contract might require a verified order, an eligible refund reason, one refund operation, an audit record and a confirmation message. It might also forbid refunding an already refunded order or changing a different customer's account.
Write those conditions before choosing a grader. Otherwise the easiest-to-measure signal becomes the definition of success. Fluent wording is easy to score. Correct state transitions are what the business depends on.
Keep a small set of normal cases, boundary cases and adversarial cases. Include tool failures and incomplete data. When production produces a new failure, reduce it to the smallest scenario that still reproduces the mechanism and add it to the suite.
Grade the outcome and the route
An agent run has at least two test surfaces: what changed and how it got there.
Outcome checks should be deterministic wherever possible. Read the database, inspect the generated file, validate a schema, confirm that a payment identifier exists exactly once, or run the tests produced by a coding agent. These checks answer whether the requested state exists without asking another model to judge it.
The route needs a trace. OpenAI's agent evaluation guide describes a trace as the end-to-end record of model calls, tool calls, guardrails and handoffs. Its trace grading documentation recommends scoring those records to find workflow-level regressions.
A route grader can ask whether the agent selected the right tool, passed valid arguments, respected a required approval, avoided a forbidden source and stopped after the task was complete. This matters when two runs reach the same visible result but one exposes data or depends on luck.
Do not require one exact sequence when several are valid. Grade invariants instead: no write before approval, no customer lookup without an identifier, no second refund call, and no handoff after the terminal state.
Put hard limits beside quality
Quality scores do not catch every operational failure. Add hard budgets for the resources and side effects that can hurt you.
Useful limits include maximum tool calls, retry count, elapsed time, input and output tokens, external API cost and the number of state-changing operations. A run that gives a good answer after 40 unnecessary calls is not equivalent to one that finishes after four. It consumes more money, waits longer and has ten times as many opportunities to fail.
Treat these as separate gates, not ingredients in one average. A high quality score should not cancel a duplicate payment. A cheap run should not cancel an incorrect answer. Release only when the run satisfies the outcome, route and budget contracts together.
Calibrate model graders with people
Some requirements cannot be reduced to code. A research answer may need to distinguish evidence from inference. A support response may need to be clear without making a promise the policy does not support.
Anthropic's guide to agent evaluations separates graders into three groups: code-based, model-based and human. Code graders are fast and reproducible but brittle. Model graders handle nuance but are nondeterministic and need calibration. Human reviewers remain the reference for ambiguous or high-impact judgments.
Build a labelled calibration set from real examples. Have people score it using a written rubric, then compare the model grader against those decisions. Investigate disagreements rather than tuning until the headline number looks good. Recheck the calibration when the grader model, rubric or domain changes.
Do not let the same agent approve its own work. Self-critique can help the run improve before submission, but release evidence should come from independent checks and, where needed, a separate grader.
Turn failures into the next suite
Synthetic scenarios give you coverage before launch. Production traces tell you where your assumptions were wrong.
Sample failed runs, expensive successes, user corrections and sessions that required a human handoff. Remove personal data, preserve the relevant state and turn recurring mechanisms into regression cases. OpenAI's agent improvement loop uses the same sequence: traces show what happened, feedback explains what mattered, and evals make the lesson reusable.
Keep the production sample separate from the fixed release set. If every observed case moves directly into the benchmark, teams can overfit prompts and tools to yesterday's failures. A stable holdout set tells you whether the system generalises beyond the examples used during iteration.
Make the gate versioned and comparative
Every result should identify the model, prompt, tool definitions, policy, grader, dataset and code revision used for the run. Without those versions, a score cannot explain what changed.
Compare the candidate against the currently deployed system on the same scenarios. Show pass rates by scenario group, not only one aggregate. Report hard-budget failures separately. A release can improve the average while making a small but critical workflow worse.
Set the decision before running the comparison: no new safety violations, no regression in critical scenarios, and a defined tolerance for cost or latency. This prevents the team from moving the threshold after seeing a disappointing result.
What this does not change
An evaluation suite proves behavior only across the scenarios, environments and graders it contains. It does not prove that an open-ended agent is safe in every situation.
Tracing can also capture sensitive inputs, tool outputs and internal reasoning artifacts. Collect the minimum needed for diagnosis, restrict access, apply retention limits and redact data before turning production sessions into tests.
Finally, evals do not replace runtime controls. Least-privilege tools, approval gates, idempotency, rate limits and rollback paths still reduce the damage of an unseen failure.
The practical standard is narrower and stronger: define success as an observable state, inspect the path that produced it, enforce independent resource limits, calibrate subjective graders and block releases on named regressions. The score is evidence. The contract makes it operational.
Written by Dandelion Labs