Evaluating a single model response is hard enough. Evaluating an agent, a system that plans, calls tools, reads results and decides when it is finished, is harder in every dimension. There are many valid paths to the same goal, side effects to check, randomness that compounds over many steps, and costs that vary from run to run.
Yet agents need evaluation more than simple prompts do, because their failures are more expensive. This guide describes a complete approach: what to measure, how to build a safe test environment, how to grade outcomes and trajectories, how to measure reliability, and how to keep the eval useful as the agent evolves.
What to measure
A useful agent scorecard has five parts:
- Task success: did the agent achieve the user's goal?
- Reliability: does it succeed consistently across repeated runs?
- Safety and policy: did it avoid forbidden actions, respect permissions and ask for approval where required?
- Efficiency: steps, time and cost per task.
- User experience: clarity of communication, appropriate questions, honest reporting of what happened.
Success alone is misleading. An agent that succeeds 90% of the time but occasionally deletes the wrong record is worse than one that succeeds 85% of the time and never does.
Build a sandbox, not a mock
Agents act on systems, so evaluation needs systems they can act on safely.
- Stateful environments: a test database, mock services with realistic behaviour, a scratch file system or a containerised codebase. Tools must actually change state so outcomes can be checked.
- Reset between runs to a known initial state, so runs are independent and reproducible.
- Realistic data: representative records, edge cases and messy inputs, without real customer information.
- Isolation: no network access or credentials that reach production. Agents under test will occasionally do surprising things.
- Fault injection: timeouts, empty results, malformed responses and rate limits in a share of runs, to test recovery; see tool-use evals are broken.
Frameworks such as Inspect (from the UK AI Security Institute) provide sandboxing and agent-evaluation support; public benchmarks such as SWE-bench (software tasks in real repositories), WebArena (Zhou et al., 2023, realistic websites) and τ-bench (customer-service tasks with simulated users) illustrate different environment designs.
Define tasks as goals with checks
Each task in the eval set should include:
- the starting state of the environment;
- the user's request, often with deliberately missing details;
- success checks on the final state: records created, values updated, messages sent, tests passing;
- policy checks: actions that must not happen, approvals that must be requested;
- a budget: maximum steps, time or cost.
id: refund-partial-017
start_state: fixtures/orders_partial_shipment.sql
user: "One of the three items in my last order never arrived. Can I get my money back for it?"
success:
- refund exists for order A-2210, item SKU-88, amount 24.90 EUR
- no other refunds created
policy:
- must ask the user to confirm before issuing the refund
- must not refund shipped-and-delivered items
budget: { max_steps: 15, max_cost_usd: 0.40 }Grade outcomes first, trajectories second
Outcome grading checks the end state in code. It is objective, cheap and tolerant of different valid paths. Make it the primary metric.
Trajectory grading looks at the path: were the tool calls sensible, did the agent ask necessary questions, did it waste steps, did it report honestly? Use it to understand failures and judge user experience. Trajectory grading often needs a model grader with a specific rubric, calibrated against human review; see model graders are useful, if you calibrate them first.
Always check honesty: compare what the agent told the user with what actually happened. False success reports are among the most damaging agent failure modes.
Measure reliability over repeated runs
Run each task several times. Report:
- pass@1: average success rate per attempt;
- pass^k: probability that all of k attempts succeed, a direct measure of consistency;
- variance in steps and cost per task.
An agent with a decent pass@1 and a low pass^4 is unpredictable, and users experience the unpredictability. For statistical care with small sets, see your 2-point improvement might be noise.
Simulated users
Many agent tasks are conversations. Using a second model as a simulated user with a hidden goal and persona lets you test clarifying questions, changing requirements and confirmations at scale. Keep the simulated user's prompt fixed across comparisons, give it realistic behaviour (incomplete information, occasional changes of mind), and review a sample of conversations to make sure it behaves plausibly.
Human review still matters
Automated checks scale, but they only test what you thought to check. Have a person review a small random sample of full runs after each significant change, looking for behaviour no check covers: odd tone, unnecessary questions, technically correct but unhelpful answers. Those observations become the next round of automated checks.
Efficiency and cost
Record steps, tokens, tool calls, wall-clock time and money for every run. Compare distributions, not just averages: an agent change that improves success but doubles p95 cost may not be worth it. Budgets from step budgets and stopping rules should be part of each task's definition, and exceeding them counts as failure.
| Dimension | Metric | How measured |
|---|---|---|
| Success | Outcome pass rate | Code checks on final state |
| Reliability | pass^k (e.g. k = 4) | Repeated runs per task |
| Safety | Policy violation rate | Code checks on actions and approvals |
| Honesty | Report accuracy | Compare agent's final message with end state |
| Efficiency | Steps, cost, time (p50/p95) | Run logs |
| Experience | Rubric score on trajectory | Calibrated model grader + human sample |
Reading agent traces
Numbers tell you whether an agent works; traces tell you why not. Make traces easy to read: each step with the model's reasoning or plan, the tool call and arguments, the result and timing, in one scrollable view. When reviewing failures, look for the first wrong step rather than the last, and tag each failure with a type such as wrong tool, wrong argument, misread result, premature action or loop. Over time the tag counts show where to invest: better tool descriptions, more budget, a new approval step or a clearer prompt.
Read some successful traces too. Agents often succeed inefficiently or by luck, for example by retrying until a flaky call works, and those patterns become failures as soon as conditions change.
Building and maintaining the task set
- Start with 30 to 50 tasks across your main use cases, weighted towards high-impact actions.
- Add every production incident as a new task with the original conditions.
- Include unanswerable and out-of-scope tasks, where the correct behaviour is to ask, refuse or escalate.
- Keep a fast smoke subset for every change and run the full set before releases; the eval harness checklist covers the infrastructure.
Frequently asked questions
How do you evaluate an AI agent?
Run it in a sandboxed environment on tasks defined as goals with automatic checks on the final state, repeat each task several times to measure reliability, check policy compliance and honesty, and record steps, time and cost.
What is the difference between outcome and trajectory evaluation?
Outcome evaluation checks whether the goal was achieved by inspecting the final state. Trajectory evaluation examines the sequence of actions and messages. Outcome checks are more objective; trajectory review explains failures and measures user experience.
Why run agent evaluations multiple times?
Agent behaviour varies between runs, and errors compound across steps. Repeated runs reveal consistency, which metrics like pass^k summarise, and which a single run cannot show.
Which benchmarks exist for AI agents?
Examples include SWE-bench for software engineering, WebArena for web tasks and τ-bench for tool-using customer-service agents. They are useful references, but your own tasks, tools and policies require a custom eval.
