Evaluating Agents
Outcome versus trajectory, task success rate, step efficiency, cost distributions, and why a single passing run proves almost nothing
Evaluating Agents
TL;DR
Agents are harder to evaluate than a single model call for one reason: they are not deterministic. The same input can take a different path every run. So a passing test tells you one path worked once, not that the system works. You must measure outcome (did it get the right result?) and trajectory (did it get there sensibly?), across many runs.
| Property | Value |
|---|---|
| Level | Intermediate → Advanced |
| Reading time | ~22 minutes |
| Prerequisites | Multi-Agent Systems |
| You will understand | What to measure, how to build agent test cases, and how to read the results |
Why This Is Different
| Single model call | Agent |
|---|---|
| One input, one output | A path of many steps |
| Same input → same output at temperature 0 | Same input → different path, run to run |
| Cost is predictable | Cost varies wildly between runs of the same task |
| One thing to grade | Outcome and every step along the way |
Run every evaluation case several times. Five to ten runs per case is a reasonable minimum. A case that passes once and fails four times is a failing case, and you cannot see that from a single run. Report the pass rate, never a pass/fail.
Measure Two Things
Two questions about every run
Outcome
Is the final result correct and complete? This is what the user cares about.
Trajectory
Did it get there sensibly — right tools, no wasted steps, no lucky guesses?
Why trajectory matters even when the outcome is right:
| Trajectory problem | Why it matters despite a correct answer |
|---|---|
| Guessed instead of checking | It was right this time. It will not be next time. |
| Took 14 steps for a 3-step task | Costs 4× and will hit your cap on a harder case |
| Called a write tool it did not need | A side effect you did not want |
| Ignored a failed tool and continued | The answer is unsupported; it just happened to be right |
A correct answer reached by a bad route is a latent failure. It passes your test today and breaks in production on a slightly different input. Grading trajectory is how you catch it before it ships.
The Metrics
| Metric | What it tells you | Watch for |
|---|---|---|
| Task success rate | Fraction of runs producing a correct result. The headline number. | Report per-case pass rate, not overall pass/fail |
| Steps per task | Efficiency. | A rising average means the agent is wandering |
| Tool error rate | How often calls fail | Often the real cause behind bad outcomes |
| Tool choice accuracy | Did it pick the right tool? | Points straight at description problems |
| Cost per task | The money question | Track p95, not the mean — the tail is long |
| Latency per task | The wait | Same: p95 |
| Loop rate | How often it hits the iteration cap | Should be near zero |
| Handoff rate | How often it gives up to a human | Maps directly to business value |
| Recovery rate | How often it fixes itself after a tool error | A good measure of error-handling quality |
Averages hide agents. Agent cost and latency have long tails — most tasks are cheap and a few are enormous. A mean cost of $0.04 with a p95 of $0.60 is a very different system from one where both are $0.05, and only the second is safe to expose to unpredictable traffic.
Building Test Cases
| Field | Purpose |
|---|---|
| Task | Phrased as a real user would phrase it |
| Expected outcome | What a correct result contains |
| Expected tools | Which tools should be used — this is what makes trajectory gradable |
| Forbidden actions | What must never happen (e.g. no write calls on a read-only task) |
| Max reasonable steps | Above this, it is wandering even if the answer is right |
| Category and difficulty | So you can see which kinds of task fail |
What to cover
| Category | Why |
|---|---|
| Easy, single-tool tasks | Baseline. Should be near 100%. |
| Multi-step tasks | The real target |
| Tasks needing a tool that will fail | Tests recovery, which nothing else does |
| Impossible tasks | Does it give up honestly, or fabricate? |
| Ambiguous requests | Does it ask, or guess? |
| Tasks where the obvious tool is wrong | Tests tool selection properly |
| Long tasks | Tests context management and drift |
Include tasks the agent cannot complete. This is the agent version of the unanswerable-question rule. If every test is solvable, an agent that always produces something scores well, and you have optimised toward confident fabrication on exactly the cases where you needed honesty.
Grading
| Method | Good for | Limits |
|---|---|---|
| Exact match | Structured outputs, numbers | Only works when there is one right form |
| Programmatic check | "Did the file get created?", "does the code run?" | Best when available — cheap, exact, no judgement |
| Tool-use assertions | "Called get_revenue with region=Europe" | Grades trajectory directly |
| LLM-as-judge | Open-ended answers | Must be validated against human grading first |
| Human review | The ground truth | Expensive; use it to calibrate everything else |
Prefer programmatic checks wherever the task allows. If the agent was supposed to produce a working script, run the script. A real verifier beats any amount of model judgement, and it never drifts.
Reading the Results
Some diagnostic patterns worth recognising:
| Pattern | What it usually means |
|---|---|
| High success, high step count | It works but wastes money — check for repeated calls |
| High success, high variance | Fragile. It is getting lucky on some runs. |
| Low success, low step count | Giving up early — check termination and tool errors |
| Low success, hits the cap | Looping — no clear finish condition |
| Good on easy, bad on multi-step | Context or planning problem |
| Fails only on long tasks | Context management — start compacting |
| Fails after a tool error | Error handling: the tool is returning something unusable |
Evaluation in CI
Evaluation-driven development
Change something
Prompt, tool description, model, or limits
Run the suite, N times per case
Non-determinism means one run proves nothing
Compare against the stored baseline
Per-metric deltas — success rate, steps, cost, p95
Ship and watch production
Offline gains must show up in real traffic
Pin your model version in evaluation. Agent behaviour changes between model versions, sometimes substantially. An unpinned suite will show you drift you did not cause and hide regressions you did.
Common Mistakes
What goes wrong in agent evaluation
One run per case
The defining mistake. Agents are non-deterministic; a single pass measures luck as much as quality.
Grading only the final answer
Misses guessing, wasted steps, unwanted side effects, and ignored tool failures — all of which will bite later.
Reporting means for cost and latency
Hides the long tail that decides whether you can afford real traffic. Report p95.
No impossible tasks
You never measure whether the agent can admit failure, so you select for fabrication.
No tasks where a tool fails
Recovery is a major part of real behaviour and goes completely unmeasured.
Unpinned model version
Your baseline moves under you and regressions become invisible.
No trace stored with the result
A failing case you cannot inspect is a failing case you cannot fix.
Concept Checks
Check yourself
Why is a single passing run weak evidence for an agent?
Because the agent chooses its own path, so the same input can produce a different sequence of steps each time. One pass shows that one path succeeded once — it says nothing about the four other paths the same input might take. A case that passes once and fails four times looks identical to a solid case if you only run it once. Pass rate across many runs is the only meaningful measurement.
The final answer is correct but the agent guessed instead of calling the tool. Is that a pass?
No. It is a latent failure that happens to have landed well. The agent produced an unsupported claim, and the mechanism that produced it — guessing rather than checking — will produce a wrong claim on the next input. Grading trajectory catches this while it is still cheap; grading outcome alone ships it to production, where it fails on data you did not test.
Why report p95 cost rather than mean cost?
Because agent cost distributions have long tails. Most tasks terminate quickly and cheaply, while a minority take many more iterations, and the mean hides those entirely. A system averaging four cents per task but reaching sixty at p95 will behave very differently under unpredictable real traffic than one where both figures are close. The tail is what determines whether you can afford to expose it.
Why deliberately include tasks the agent cannot complete?
Because otherwise you never measure honesty. If every test case is solvable, an agent that always produces an answer scores perfectly, while one that correctly says "I could not verify this" is penalised for it. Every tuning decision then pushes toward producing something regardless of evidence. Impossible tasks are the only way to check that the give-up path works and is rewarded.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Non-determinism | The defining difficulty — run every case many times |
| Outcome and trajectory | Right answer, right route. Both. |
| Latent failure | Correct result reached badly; it will break later |
| Task success rate | The headline, reported as a rate not a verdict |
| p95 over mean | Agent cost and latency have long tails |
| Programmatic checks first | Run the code; do not ask if it looks right |
| Impossible tasks | The only way to measure honest failure |
| Failing-tool tasks | The only way to measure recovery |
| Pin the model version | Or your baseline drifts |
| Store the trace | An uninspectable failure is unfixable |
Next
Measuring is not enough when the agent can act on the world: Safety & Security.
Multi-Agent Systems
Orchestrator-workers, handoffs, debate, and parallel specialists — the coordination patterns, what they cost, and the honest case for using one agent
Safety & Security
Prompt injection, tool permissions, sandboxing, human approval gates, and cost limits — the controls an agent needs before it touches anything real