Production & Operations
Hard limits, tracing, timeouts, retries, idempotency, latency, cost, and the change management an agent needs once real users depend on it
Production & Operations
TL;DR
A prototype agent and a production agent differ in almost nothing except the boring parts: limits, traces, timeouts, retries, and a way to change things without breaking them. The one number that governs everything else is steps per task — it drives latency, and because the whole transcript is re-sent on every step, it drives cost faster than linearly. Reduce steps and most of your operational problems shrink at once.
| Property | Value |
|---|---|
| Level | Advanced |
| Reading time | ~25 minutes |
| Prerequisites | Safety & Security |
| You will understand | What has to exist before an agent is allowed near real users, and why |
Part 1 — Limits That Must Exist
An agent decides its own path, which means it decides its own bill. Every agent needs hard stops that live in your code, not in the prompt.
| Limit | Typical value | What it prevents |
|---|---|---|
| Iterations per task | 10–25 | The endless think-act-think loop |
| Tokens per task | From your budget | A context that grows until the model refuses |
| Money per task | From your budget | The limit that actually matters — the other two are proxies |
| Wall-clock per task | 30–120 seconds | A user staring at a spinner forever |
| Calls to any one tool | 3–5 | Retrying the same failing search twenty times |
| Tasks per user per hour | Fits your traffic | One user consuming the whole budget |
The prompt is not a limit. "Do not use more than ten steps" is a request the model will sometimes honour. The loop counter in your code is a limit. Anything you would be unhappy to see exceeded belongs in the harness, where the model has no vote.
What to do when a limit is hit
Hitting a cap is normal, not exceptional. Decide the behaviour deliberately.
| Behaviour | When it fits |
|---|---|
| Return partial results with a clear caveat | Research and summarisation, where half an answer has value |
| Hand off to a human with the trace attached | Support and anything customer-facing |
| Fail loudly and retry the whole task once | Batch jobs where nobody is waiting |
| Return the last good state | Long multi-stage pipelines with checkpoints |
Whatever you choose, the cap must be visible in your metrics. An agent that silently truncates at twenty steps and returns something plausible looks healthy right up until someone reads the output carefully. This is the loop rate from Evaluating Agents; it should be close to zero, and a rise in it is one of the earliest signals that something upstream has broken.
Part 2 — Tracing
You cannot debug an agent from its final answer. The answer is one line at the end of a path you did not choose and cannot reconstruct.
What to log, per step
| Field | Why you will want it |
|---|---|
| Task ID and step number | Ties the whole run together |
| The model's reasoning text | Shows why it did the wrong thing, not just that it did |
| Tool name and full arguments | Wrong arguments are more common than wrong tools |
| Tool result, or the error | The input to the next decision |
| Tool duration | Finds the one slow call in a slow task |
| Tokens in and out | The only way to attribute cost |
| Model and prompt version | So a regression can be traced to a change |
| Stop reason | Finished, capped, error, or handed off |
Log the arguments, and scrub them. Traces are stored, exported, and read by more people than you expect. Redact credentials and personal data at the point of writing, not in a later cleanup job that will not happen.
Metrics worth alerting on
| Metric | What a change means |
|---|---|
| Task success rate | The headline. Everything else explains it. |
| Steps per task (p50 and p95) | A rising p95 means the agent is wandering on hard cases |
| Tool error rate, per tool | Usually the real cause of a drop in success rate |
| Cost per task (p95) | Agent cost has a long tail; the mean hides it |
| Loop rate | Runs ending at the iteration cap. Should be near zero |
| Handoff rate | Maps directly to human workload and to business value |
Alert on rate of change, not only on thresholds. A tool that has started failing intermittently shows up first as a slow rise in average steps per task, because the agent retries and works around it, long before it shows up as a visible error.
Part 3 — Reliability
An agent depends on tools, and tools fail. The question is not whether one goes down but what the agent does for the minutes it is down.
Timeouts
Every tool call needs one. An agent waiting on a hung API is an agent holding a user's session open for as long as that API feels like it.
| Timeout | Set it |
|---|---|
| Per tool call | Just above the tool's real p99 |
| Per model call | Generous, but finite |
| Per task | The user-facing promise |
Retries
| Rule | Reason |
|---|---|
| Retry transient failures with backoff | Timeouts, 429s, 5xx — the call may simply work next time |
| Never retry a bad argument | It will fail identically. Return the error to the agent so it can fix the argument. |
| Cap retries per tool per task | Otherwise the loop limit gets consumed by one broken dependency |
| Make the error message useful | "No customer with that ID" teaches the agent something; "Error" does not |
Retries and idempotency are the same subject. An agent retries on its own initiative, in addition to whatever your infrastructure does. A send or a payment that timed out after succeeding looks like a failure to the agent, which will reasonably try again. Idempotency keys on every write are the only reliable protection, because the caller genuinely cannot tell a lost response from a failed action.
Degrading rather than dying
| Failure | Graceful behaviour |
|---|---|
| One tool is down | Remove it from the tool list for this run and tell the agent it is unavailable |
| The model provider is degraded | Fall back to a second model with the same tool definitions |
| Everything is slow | Shed load: queue, or refuse new tasks, rather than time out mid-task |
| A tool is failing for everyone | Circuit-break it — stop calling it until it recovers |
Part 4 — Latency
Agents are slow in a way that single model calls are not, and the reason points straight at the fix.
Where the time goes
For a typical multi-step agent, the arithmetic is simple and unforgiving:
Task time ≈ steps × (model call + tool call)
6 steps × (2.5s model + 0.4s tool) ≈ 17 seconds
12 steps × (2.5s model + 0.4s tool) ≈ 35 secondsStep count dominates everything. Shaving 200 ms off a tool call saves you about a second across a whole task. Removing four unnecessary steps saves twelve. Before optimising any individual call, look at why the agent needed that many steps — usually a vague goal, a missing tool, or a tool that returns too little per call.
What actually helps
| Change | Effect |
|---|---|
| Fewer, higher-yield tools | One tool returning the whole customer record beats four returning fragments |
| Run independent tool calls in parallel | Most models can request several at once; run them concurrently |
| Stream the final answer | Does not reduce total time, but drastically changes how long it feels |
| Show progress during the run | "Searching orders…" turns dead air into visible work |
| A smaller model for simple tasks | Route by difficulty; most traffic is easy |
| Prompt caching on the stable prefix | System prompt and tool definitions are identical on every step of every task |
Independent tool calls
get_order
180 ms
get_customer
140 ms
get_shipping_status
220 ms
Part 5 — Cost
Agent bills are not large because the model is expensive. They are large because of how the loop uses it.
Where the money goes
Agents spend most of their budget on input tokens, and the reason surprises people.
Every step re-sends the entire transcript so far.
Step 1: system + tools + question → 2,000 tokens in
Step 2: ... + step 1 thought + result → 3,400 tokens in
Step 3: ... + step 2 thought + result → 5,100 tokens in
...
Step 10: ... → 21,000 tokens in
Total input for a 10-step run ≈ 110,000 tokens
Total output for the same run ≈ 4,000 tokensCost grows faster than step count. Each extra step costs more than the one before it, because it carries everything that came before. A run that takes twice as many steps typically costs three to four times as much. This is the single strongest argument for keeping runs short.
Levers, ordered by return
| Lever | Typical saving | Notes |
|---|---|---|
| Reduce steps | Very large | Clearer goals, better tools, a plan for long tasks |
| Prompt caching | Large | The system prompt and tool definitions repeat on every single step |
| Summarise old context | Large on long runs | Replace raw tool output with a compact state |
| Route by difficulty | Large | A small model handles the easy majority |
| Trim tool output | Moderate | Return the five fields the agent needs, not the whole record |
| Shorten the system prompt | Small | Real, but it is the last place to look |
Track cost per completed task, not per model call. Per-call cost looks fine while an agent takes fifteen steps to do a two-step job. Per-task cost against a success rate is the number that tells you whether the system is worth running.
Part 6 — Changing Things Safely
Agents are non-deterministic, which makes every change harder to verify than in ordinary software.
| Practice | Why |
|---|---|
| Pin the model version | Behaviour shifts between versions, including tool-choice behaviour |
| Version prompts and tool definitions | A tool description edit is a behaviour change; it belongs in review |
| Re-run the evaluation set on every change | Several times per case — one pass proves nothing |
| Roll out to a fraction of traffic first | Compare success, steps, and cost against the current version |
| Keep replayable traces | A stored trace lets you re-run a real failure against a candidate change |
Shipping a change
1. Change one thing
A prompt, a tool description, a model version — not three at once
2. Run the evaluation set
Multiple runs per case; compare distributions, not single results
3. Replay real failed traces
The cases that motivated the change should now pass
4. Ship to a slice of traffic
Watch success rate, steps, cost, and handoff rate side by side
5. Roll out or roll back
Decide on the numbers, not on how the change felt in testing
Change one thing at a time. With a non-deterministic system and a noisy metric, two simultaneous changes cannot be untangled afterwards. You will know the result moved and never know which edit moved it.
Part 7 — The Handoff Path
Every production agent gives up sometimes. Design that path deliberately, because it will be used every day.
Giving up well
A handoff that arrives with a clear summary and the full trace is worth far more than one that says "the assistant could not help". The agent has usually already done half the investigation — losing it is pure waste, and it is the difference between an agent that reduces human workload and one that adds a step to it.
Production Readiness Checklist
Before you call it production
Hard caps on iterations, tokens, cost, and wall-clock
In code, not in the prompt. With a defined behaviour when each one is hit.
Full trace for every task
Reasoning, tool names, arguments, results, durations, tokens, stop reason — with secrets scrubbed.
Timeout on every tool call
Plus retries with backoff for transient failures only, and idempotency keys on every write.
An evaluation set that runs on every change
Multiple runs per case, including the failures that motivated past fixes.
Model version pinned, prompts and tools versioned
Upgrades are a deliberate, tested event.
A designed handoff path
With a summary and the trace attached, routed somewhere a person will see it.
Dashboards for success rate, steps, tool errors, cost p95, loop rate, handoffs
Alerting on rate of change, not only on thresholds.
A kill switch
One setting that stops the agent taking actions, without a deploy.
Concept Checks
Check yourself
Why does agent cost grow faster than the number of steps?
Because the model is stateless and every step re-sends the entire transcript. Step ten carries the system prompt, the tool definitions, the question, and nine rounds of thoughts and tool results, so it costs several times what step one cost. Summed across a run the input tokens grow roughly with the square of the step count, which is why a task that takes twice as many steps typically costs three to four times as much rather than twice.
Why is optimising an individual tool call usually the wrong place to start?
Because task time is step count multiplied by the cost of a step, and the model call dominates the tool call in most agents. Removing 200 ms from a tool that runs six times saves about a second; removing four unnecessary steps saves ten or more. A high step count also points at a real design problem — a vague goal, a missing tool, or tools that return too little per call — so investigating it fixes cost and reliability at the same time.
Why must a bad argument not be retried, when a timeout should be?
Because they fail for different reasons. A timeout is a property of the moment — the same call may well succeed a second later — so retrying with backoff is the correct response. A malformed or invalid argument will fail identically forever, so retrying only burns iterations and money. The right move is to return a descriptive error into the conversation so the agent can correct the argument itself, which it usually does on the next step.
Why does a near-zero loop rate matter so much as a metric?
Because hitting a cap means the agent did not finish, and what it returns in that case is often plausible enough to pass casual inspection. If the rate is normally near zero, any rise is an early and specific alarm — it usually means a tool has started failing and the agent is looping around it. If the rate is routinely high, the signal is gone, and you are shipping truncated work as a matter of course.
Why pin the model version rather than always taking the newest one?
Because an agent's behaviour depends on the model's judgement about when to call which tool, and that judgement changes between versions in ways that are not announced and not always improvements for your particular tool set. A pinned version makes the upgrade a deliberate event you can evaluate against your own test set and roll back. Floating means an upstream change can alter your success rate on a random Tuesday with no corresponding commit.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Limits live in code | Iterations, tokens, money, wall-clock, per-tool calls |
| Define the cap behaviour | Partial result, handoff, retry, or last good state — and measure it |
| Trace every step | Reasoning, arguments, results, durations, tokens, stop reason |
| Steps dominate latency | Optimise the count before the call |
| Cost grows super-linearly | Every step re-sends the whole transcript |
| Prompt caching is nearly free money | The stable prefix repeats on every step of every task |
| Retry transient, never invalid | And return useful errors to the agent |
| Idempotency keys on every write | Agents retry on their own initiative |
| Change one thing, then evaluate | Non-determinism makes bundled changes untraceable |
| Design the handoff | With a summary and the trace attached |
Next
Everything above assumes you can find out what went wrong. That is its own skill: Failure Modes & Debugging.
Safety & Security
Prompt injection, tool permissions, sandboxing, human approval gates, and cost limits — the controls an agent needs before it touches anything real
Failure Modes & Debugging
Every way an agent goes wrong — from silent tool failures and vague descriptions to loops, drift, and non-determinism — and how to find the real cause from the trace