Evaluating AI Agents: Accuracy Is the Wrong Metric

Evaluating a classifier is straightforward: you have labels, you compute accuracy, you compare. Evaluating an agent is not, and teams that try to force it into the same shape end up with a number that looks rigorous and predicts nothing.

Why accuracy does not transfer

Three properties break it.

There is often no single correct answer. Asked to investigate a failing service, an agent might read logs first or check recent deployments first. Both are reasonable. Grading against one expected trajectory penalises a competent alternative.

Partial success is the normal outcome. An agent that completes four of five steps and stalls has not “failed” in a way a binary metric captures usefully. Whether that is a good result depends on whether the four steps were the hard ones.

The same input gives different runs. A single evaluation tells you almost nothing. You are sampling a distribution of behaviours, which means you need repeated runs and you should care about variance as much as mean.

What to measure instead

Task completion, defined by outcome

Define success by the end state rather than the path. Was the ticket actually resolved? Does the code pass its tests? Did the record get updated correctly? This admits multiple valid routes and is the metric that corresponds to what you want.

Cost per completion

An agent that succeeds after forty tool calls and one that succeeds after six are not equivalent, even though both succeeded. Track tokens, tool invocations and wall-clock time per successful outcome. Cost frequently degrades long before success rate does, and it is the earlier warning.

Failure mode distribution

More useful than the failure rate is the shape of the failures. Give up early? Loop indefinitely? Confidently produce a wrong result? Call the wrong tool?

These have very different consequences. An agent that stops and asks for help is operationally fine. One that proceeds confidently while wrong is dangerous. A single accuracy figure treats them identically.

Recovery behaviour

Things go wrong in production — a tool times out, an API returns something unexpected. What the agent does next matters enormously. Deliberately inject failures and observe whether it retries sensibly, adapts, or falls over.

Variance across runs

Run the same task twenty times. An agent that succeeds eighteen times out of twenty is deployable. One that averages the same rate by succeeding brilliantly and failing catastrophically in alternation is not, and the mean hides that entirely.

Build the evaluation set from production

Synthetic test cases are convenient and systematically unrepresentative. They are cleaner, better specified, and less strange than real requests.

The evaluation set worth having is drawn from actual traffic, including the malformed, ambiguous and adversarial cases. That means logging real interactions from the start with the specific intention of building a test set later.

Keep a human in the loop on quality

For anything involving judgement, automated scoring is a proxy. Model-based grading is useful for scale and inherits its own biases — it tends to reward fluency.

Periodic human review of a random sample is the control that catches drift the automated metrics miss. It is expensive and there is no substitute.

The framing that helps

Stop asking “how accurate is this agent?” Ask “how often does it achieve the outcome, at what cost, and when it fails, what does failure look like?”

Three numbers instead of one, and all three are actionable.