A benchmark score is a summary of one attempt. Production is a distribution.
The distinction sounds pedantic until the two disagree. An agent that answers a task correctly on the run you watched will answer it a thousand times without you watching, and the interesting question is not whether it can, but how often it does, and what it does on the occasions it does not. Those are different measurements. Most teams only take the first one.
What a single run hides
Three properties do not appear in a single-run score.
Variance. Ask the same question repeatedly and the trajectory differs each time — a different tool order, a different intermediate summary, a different decision about when there is enough evidence to answer. Some of the runs that scored as correct were correct by accident: the model took a wrong path and recovered, or skipped a step whose absence did not matter for that particular input. A score cannot distinguish a correct answer from a lucky one. A trajectory can.
Fault behaviour. In an eval harness, tools answer. In production they time out, rate-limit, return a partial page, and occasionally return something structurally valid and semantically wrong. The failure mode we care about most is the quiet one: a model that receives a broken tool response rarely stops. It improvises around the gap and continues with a corrupted context, and the answer it produces is as fluent and as confident as the correct one. Nothing in the output signals what happened upstream.
Input drift. A benchmark holds its inputs still. The application does not. Documents get re-formatted, a field goes null, an upstream service starts truncating, a user pastes something twice as long as anything in the eval set. The system does not have to be robust to inputs it has seen; it has to be robust to the ones it has not.
Published work on agent evaluation has been converging on the same conclusion from the outside: evaluations that look only at the final answer report substantially more passing cases than evaluations that inspect the whole trajectory, and single-run benchmark figures overstate what the same system does under repeated execution with real tool faults. Traces from our own systems have the same shape. The direction of the gap is never favourable.
Scoring the trajectory
We stopped asking was the answer right as the primary question, and started asking a set of questions about the path:
- Did it call the tool it should have called, or reach the answer without evidence it was entitled to use?
- Were the arguments to that call derivable from the context, or invented in a plausible format?
- Did it recognise a failed call as failed, rather than treating an error payload as data?
- Did it stop when the evidence was insufficient, or produce an answer anyway?
- On repeated runs of the same input, how far apart are the trajectories?
The last one is the cheapest signal and the one most often skipped. Trajectory spread on a fixed input is a direct measure of how much of the behaviour is under the system’s control rather than the sampler’s. When the spread is wide, no amount of prompt revision fixes it; the task is underspecified, or the tool surface allows too many routes to the same place.
What we run now
The rules that came out of this, in the order they matter:
- Every eval case runs many times, and we read the worst decile, not the mean. The mean describes a system nobody experiences. The tail is what generates support tickets.
- Fault injection is part of the eval suite, not of chaos testing. Timeouts, malformed payloads, empty result sets and stale indexes are ordinary conditions, so they belong in the ordinary evaluation, with expected behaviour defined for each.
- A refusal to answer is a passing outcome. If the evidence was insufficient and the system said so, it did the right thing. Scoring that as a failure teaches exactly the wrong behaviour, and it is the most common scoring mistake we see.
- Trajectories are stored, not just scores. A score tells you the number moved. Only the trajectory tells you why, and only stored trajectories let you ask a question you did not think to ask at the time.
What it cost
Slower shipping, and at least one agent that never shipped. It scored well, it demonstrated well, and under repeated execution with injected faults its behaviour was not stable enough for the decision it was making. We could have shipped it on the strength of the offline number. The offline number was real; it was answering a question we did not need answered.
The purpose of an eval is not to certify a system. It is to find out what the system does on the day everything is slightly wrong, which — over a long enough production window — is most days.