Tool-use evals are broken. Here's how to fix them.

FFunction calling, also called tool use, is the bridge between a language model and the real world. The model decides which tool to call and with which arguments; your code executes it and returns the result. Nearly every agent depends on this loop, so it is natural to want a number that says how good a model is at it.

Most tool-use evaluations, including many internal ones, produce numbers that look much better than production reality. This article explains the four most common design flaws, the evidence that reliability across repeated runs is lower than single-run scores suggest, and how to build a tool-use eval that predicts how your agent will actually behave.

Flaw 1: grading the call instead of the outcome

The simplest tool-use eval compares the model's function call to an expected call: same function name, same arguments. This is easy to automate and almost always wrong as a measure of success.

• There are often several valid paths: search then book, or check availability then book; filter by date in the query or after the results.
• Arguments can be equivalent but not identical: "2026-10-11" vs. "Oct 11, 2026," or a different but valid ordering of list items.
• A call can match exactly and still fail the task because a later step went wrong.

Fix: grade the final state of the world. After the episode, inspect the database, the mock calendar or the file system, and check whether the user's goal was achieved without forbidden side effects. τ-bench (Yao et al., 2024) popularised this approach for customer-service style agents by comparing the end database state with the expected one.

Flaw 2: a single run per task

Model outputs vary between runs. An agent that completes a task once out of three attempts looks identical to one that completes it every time if you only run it once. In production, users experience the variance.

Fix: run each task several times and report reliability, not just average success. τ-bench introduced pass^k: the probability that the agent succeeds on *all* of k independent attempts at the same task. The authors reported that pass^k dropped sharply as k increased for the models they tested, which means agents that look decent on one try are far less consistent than the headline suggests.

from math import comb

def pass_hat_k(successes: int, trials: int, k: int) -> float:
    """Unbiased estimate of P(all k attempts succeed), given `successes` out of `trials` runs."""
    if trials < k:
        raise ValueError("need at least k trials")
    return comb(successes, k) / comb(trials, k)

# a task solved in 6 of 8 runs looks like 75%, but:
print(pass_hat_k(6, 8, 4))   # ≈ 0.21 chance that 4 attempts in a row all succeed

Average per-task pass^k across the set. For anything customer-facing, k of 4 or more is a more honest reliability signal than pass@1.

Flaw 3: perfect tools

Mock tools in evals usually behave perfectly: they never time out, never return partial pages, never change their output format, and always return results in the expected shape. Production tools do all of these things.

Fix: inject realistic failures into a portion of runs:

• timeouts and rate-limit errors (does the agent retry sensibly or give up?);
• empty results (does it report "nothing found" or invent an answer?);
• partial or paginated results (does it fetch the rest?);
• ambiguous results, such as two customers with the same name (does it ask?);
• malformed responses (does it notice?).

Score how the agent handles each. Graceful failure, meaning a clear message to the user and no damaging action, should count as a pass for an injected fault. Tool design matters here too; see designing tools that agents can actually use.

Flaw 4: users who state everything up front

Many evals start each task with a complete, precise instruction: "Book the 9:40 flight from Lisbon to Berlin on 12 November for customer 4471." Real users say "I need to get to Berlin next week, the usual." Gathering missing information, confirming before acting and handling users who change their mind are central to real agent work.

Fix: use a simulated user: a separate model prompted with a hidden goal and persona, who reveals information only when asked and sometimes changes requirements mid-conversation. Simulated users are imperfect, so review a sample of transcripts to make sure they behave plausibly, and keep their instructions fixed across model comparisons.

A better tool-use harness, step by step

1. Define tasks as goals plus constraints, not expected calls: "customer ends with a refund for order X; no other orders are modified."
2. Build a stateful sandbox: a small database or set of mock services that the tools actually read and modify.
3. Add a simulated user with a hidden goal for conversational tasks.
4. Inject faults into a fixed share of runs, for example 20%.
5. Run each task several times and compute pass^1 and pass^k.
6. Grade end state and policy compliance in code, using a model grader only for parts that cannot be checked mechanically.
7. Keep transcripts and read failures every time; new failure types become new tasks.

How big should a tool-use eval be?

Agent evals are expensive to run because each task involves many model calls, so teams are tempted to keep them tiny. Some guidance:

• Start with 30 to 50 tasks spread across your main task categories, each run at least four times. That is enough to see large differences and to compute a meaningful pass^k.
• Weight the set towards the tasks that matter: the ones with irreversible actions, money or customer communication, rather than the most frequent ones.
• Add every production incident as a task. Within a few months, real failures will make up a significant share of the set, and those are the most predictive tasks you can have.
• Keep a small smoke set of 10 tasks that runs on every prompt or tool change, and run the full set before releases and model upgrades.

Record the cost of each eval run. Knowing that the full suite costs a known amount makes it much easier to justify running it regularly.

Interpreting results

Report results by task category (lookups, single actions, multi-step changes, fault handling), with confidence intervals. Small eval sets make differences between models noisy; our guide on why a 2-point improvement might be noise explains how to check. And remember that tool-use evals cover one slice of agent quality; the full picture, including cost and safety, is in evaluating agents end to end.

Public leaderboards such as the Berkeley Function Calling Leaderboard are useful for shortlisting models on raw call accuracy, but they cannot replace a harness built around your own tools, users and failure modes.

Frequently asked questions

What is a tool-use or function-calling eval?

It is a test of how well a language model chooses tools, fills in arguments and uses results to complete tasks. Good evals check whether the user's goal was achieved, not only whether the model produced a particular function call.

What does pass^k mean?

pass^k is the probability that an agent succeeds on every one of k independent attempts at the same task. It measures consistency, and it falls quickly for agents that only sometimes succeed.

Why should I inject errors into tool-use evals?

Production tools time out, return empty or partial results and change formats. Injecting faults shows whether the agent retries sensibly, reports problems honestly and avoids damaging actions when things go wrong.

Can I use a model to simulate users in agent evals?

Yes. A separate model with a hidden goal and persona can reveal information gradually like a real user. Review a sample of conversations to confirm the simulated user behaves realistically, and keep its prompt fixed across comparisons.