Failure Modes & Debugging
Every way an agent goes wrong — from silent tool failures and vague descriptions to loops, drift, and non-determinism — and how to find the real cause from the trace
Failure Modes & Debugging
TL;DR
From outside, every agent bug looks the same: it did the wrong thing. The cause is almost never the model being insufficiently clever. It is usually a tool that failed quietly, a description that did not say when to use it, a goal with no finish line, or a context that filled up. You find out which by reading the trace and locating the first step that went wrong — not the step where the damage showed.
| Property | Value |
|---|---|
| Level | Advanced |
| Reading time | ~25 minutes |
| Prerequisites | Production & Operations |
| You will understand | How to go from "it did something stupid" to a specific, fixable cause |
The One Question That Splits Every Agent Bug
Open the trace. Find the first step where the agent's state stopped matching reality.
Everything after that point is the agent behaving reasonably given bad information. Debugging the visible symptom means debugging a consequence.
An agent's final output is the end of a path you did not choose. Reading it tells you almost nothing about the cause, and the cause is nearly always several steps upstream of where you noticed.
Bisect the run
| Question | If no | If yes |
|---|---|---|
| Did it call any tool at all? | The tool was never sent, the description is vague, or the list is too long | Continue |
| Did it call the right tool? | Two tools sound alike, or the list is too long to choose from | Continue |
| Were the arguments sensible? | The schema is too loose, or the agent lacked the information to fill it | Continue |
| Did the tool return what you expected? | The bug is in the tool, not the agent | Continue |
| Did it use the result correctly? | Context, ordering, or the result format is misleading | The bug is in the goal or the stopping rule |
Why did the agent do that?
Did the agent call any tool at all?
Open the trace and look at the first few steps. An agent that never acts and an agent that acts wrongly are completely different bugs.
Notice that “the model reasoned badly” is the last leaf, not the first. Most agent failures are mechanical — a tool that never reached the model, a vague description, an error string the agent read as data. Reach for a bigger model only after the boring causes are ruled out.
Do this before forming a theory. It is remarkably common to spend an afternoon rewriting a system prompt for a problem that turns out to be a tool returning [] and a 200 OK when its upstream is down.
Part 1 — Tool-Side Failures
The largest category by a wide margin, and the one most often misattributed to the model.
| Symptom | Real cause | Fix |
|---|---|---|
| The tool is never used | It was registered in your code but never included in the request | Print the exact tool list you sent, not the one you think you sent |
| The tool is never used, and it was in the request | The description does not say when it applies | Describe the trigger, not just the function |
| The wrong tool is chosen | Two descriptions overlap | Make the boundary explicit in both: what each is for, and what it is not for |
| Choice degrades as you add tools | More than roughly fifteen options | Group them, or route to a subset first |
| Bad arguments | Free-text parameters where a fixed set of values was possible | Use enums; validate and return a specific error |
| Repeated identical calls | The result did not visibly answer the question | Return clearer output, and cap calls per tool per task |
The silent failure
A tool that returns an error string returns data as far as the model is concerned.
Tool result: "Error: connection refused"The agent reads this as a fact about the world, and will happily continue — sometimes reporting it as a finding, sometimes reasoning around it. Return errors as a distinct, unambiguous result the agent can recognise, and make an empty result look different from a failed one.
| Bad return | Why it hurts | Better |
|---|---|---|
[] on failure | Indistinguishable from "nothing matched" | An explicit error the agent can react to |
null | The agent invents an interpretation | State what was searched and what was not found |
| Raw stack trace | Wastes context and teaches nothing | One line saying what failed and whether retrying helps |
| A 200 with an error body | The harness never notices | Check the body; treat it as a failure |
Part 2 — Loop and Control Failures
| Symptom | Real cause | Fix |
|---|---|---|
| Runs forever | No definition of "done" | Say explicitly what a finished task looks like |
| Runs forever, and "done" is defined | The goal is unreachable with the tools it has | Detect the impasse and hand off rather than grinding |
| Repeats the same two steps | Each result appears to make progress but does not | Detect repeated state; break the cycle |
| Stops too early | Ambiguity resolved as "good enough" | Enumerate completion criteria; check them before returning |
| Stops after the first tool error | The prompt never said errors are recoverable | Say so, and show what a recovery looks like |
| Hits the iteration cap regularly | Usually a broken tool underneath, not a hard task | Look at tool error rates before raising the cap |
Raising the iteration cap is almost never the fix. An agent that needed twenty-five steps and now gets forty usually spends the extra fifteen doing the same unproductive thing. The cap did not cause the problem; it revealed it.
Part 3 — Context and Reasoning Failures
| Symptom | Real cause | Fix |
|---|---|---|
| Forgets the original goal midway | The goal is buried in the middle of a long transcript | Restate the goal near the end of the context on every step |
| Ignores a result that was right there | Lost in the middle — long context, weak attention to the centre | Summarise, and keep the working state compact |
| Contradicts an earlier finding | Two tool results disagree and nothing reconciled them | Keep an explicit running state, not just an append-only log |
| Confidently wrong after a failure | It read an error as data | See Part 1 |
| Drifts into a different task | Instructions arrived inside content it fetched | Treat all fetched content as data — see Safety & Security |
| Degrades on long runs only | Context growth, not reasoning quality | Compact the transcript — see Context Engineering |
A useful test: take the failing run's context at the step where it went wrong, and read it as if you were the model. Very often the answer is obvious — the goal was fifteen thousand tokens back, the relevant fact is buried in the middle of a raw API dump, and the most recent thing in front of it is an unrelated error. The model is not being stupid; it is being given a bad prompt that you assembled one step at a time.
Part 4 — Failures That Hide
These do not show up as errors. They show up as a system that quietly is not as good as you think.
Failures that pass inspection
Right answer, wrong route
The agent guessed, or got lucky with a poor search. The outcome metric is green. The next similar task fails. This is why trajectory is measured, not just outcome.
It works every time you test it
Agents are non-deterministic. A single passing run is one sample from a distribution you have not looked at. Run each case several times before believing it.
Truncated at the cap, answered anyway
The run stopped early and the agent produced something plausible from partial work. Nothing errored. Track the loop rate or you will never see it.
Slowly rising cost per task
Often the first visible sign of a tool failing intermittently — the agent retries and works around it long before the error rate moves.
Fine in testing, wrong for one customer
A permission filter or a tenant scope is silently returning less than the agent expects, and it reasons confidently from the gap.
Regression after a model upgrade
Tool-choice behaviour changed. Nothing in your repository changed. Pin versions and re-run the evaluation set on every upgrade.
Great demo, poor production
Demo questions were written by people who knew the tools. Real questions are vaguer, longer, and refer to things the agent cannot see.
The Debugging Checklist
Work down this list. It is ordered by how often each item turns out to be the answer.
| # | Check |
|---|---|
| 1 | Read the full trace — reasoning, tool names, arguments, and raw results |
| 2 | Find the first wrong step, not the visible symptom |
| 3 | Print the exact tool definitions sent in that request |
| 4 | Call the tool yourself with the arguments the agent used |
| 5 | Check whether errors are distinguishable from empty results |
| 6 | Read the context as the model saw it at the failing step |
| 7 | Check the stop reason — finished, capped, error, or handed off |
| 8 | Re-run the same input five times and see whether the failure is consistent |
| 9 | Diff the prompt, tools, and model version against the last known-good run |
| 10 | Only then consider the model — it is the least likely cause |
Steps 1 to 5 resolve the large majority of agent bugs, and none of them involve the model at all. The instinct to reach for a bigger model or a cleverer prompt is strong and usually wrong — those changes are expensive, hard to verify, and tend to paper over a tool problem that will resurface.
Concept Checks
Check yourself
Why debug from the first wrong step rather than the visible failure?
Because everything after the first bad observation is the agent behaving sensibly on false information. If a lookup silently returned an empty list at step three, the confident wrong answer at step nine is not a reasoning failure — it is correct reasoning from a wrong premise. Fixing the symptom means adding instructions to work around a broken tool, which leaves the tool broken and makes the prompt worse.
Why is an error string a worse tool return than an explicit failure signal?
Because the model has no channel separation between data and status. Text arriving in a tool result slot is treated as the answer to the call, so "Error: connection refused" becomes a fact about the world that the agent reasons from and sometimes reports. An unambiguous failure signal lets the agent do the sensible thing — retry, try another route, or hand off — instead of building on a sentence that was never data.
Why does raising the iteration cap rarely fix an agent that keeps hitting it?
Because the cap is a symptom, not a cause. An agent burning twenty-five steps is usually cycling — retrying a failing tool, re-searching with near-identical queries, or working without a clear finish line. Given forty steps it does the same unproductive thing for longer, at higher cost and latency. The useful response is to look at tool error rates and at whether "done" is actually defined.
Why can a passing test run be actively misleading?
Because the same input can take a different path on every run. One green result is a single sample from a distribution whose shape you have not seen, and a case that passes seventy percent of the time looks identical to one that passes always. Running each case several times and reporting the pass rate is the only way to distinguish a reliable behaviour from a lucky one.
Why does a rising cost per task often precede a visible error?
Because the agent absorbs failures rather than surfacing them. When a tool starts failing intermittently, the agent retries it, tries a different approach, and often still reaches the right answer — so the success rate holds while step count and token usage climb. Cost per task and steps at p95 therefore move first, which makes them better early-warning signals than the error rate itself.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Read the trace first | The final answer tells you nothing about the cause |
| Find the first wrong step | Everything after it is reasonable behaviour on bad data |
| Tools are the usual culprit | Missing, vague, overlapping, or silently failing |
| Errors must not look like data | And empty must not look like failed |
| Loops mean no finish line | Or a broken tool underneath |
| Long-run decay is context, not intelligence | Compact the transcript |
| Watch trajectory, not only outcome | A lucky right answer will not stay right |
| Run cases many times | One pass is one sample |
| Loop rate and rising cost are early alarms | They move before the error rate does |
| The model is the last suspect | Not the first |
Next
You now know how agents work, how they are measured, and how they break. Time to put it together: Designing an Agent System.
Production & Operations
Hard limits, tracing, timeouts, retries, idempotency, latency, cost, and the change management an agent needs once real users depend on it
Designing an Agent System
A complete worked example — gathering requirements, deciding whether you need an agent at all, sizing the run, choosing every component with reasons, and planning the rollout