Designing an Agent System
A complete worked example — gathering requirements, deciding whether you need an agent at all, sizing the run, choosing every component with reasons, and planning the rollout
Designing an Agent System
TL;DR
This page puts everything together. We take one realistic brief, ask the questions that actually shape the design, do the arithmetic, and choose every component with a reason attached. The first decision is the one most teams skip: does this need an agent at all? Then we change one requirement and watch half the design flip — because there is no universally correct agent design, only one that fits a set of constraints.
| Property | Value |
|---|---|
| Level | Advanced — brings together every previous page |
| Reading time | ~30 minutes |
| Prerequisites | Failure Modes & Debugging |
| You will understand | How to go from a vague brief to a sized, justified agent design |
Part 1 — Ask Before You Build
Most bad agent designs come from starting with the framework. Start with these questions instead.
| Question | Why it changes the design |
|---|---|
| Can you write down the steps in advance? | If yes, you want a workflow, and you are done deciding |
| What can it actually do to the world? | Read-only is a different system from one that can send, pay, or delete |
| What does a wrong action cost? | Sets how much verification, approval, and human oversight you can justify |
| Who is waiting for the answer? | A customer waits seconds; a nightly batch waits minutes |
| How many tasks per day? | Sets cost, capacity, and whether per-task price matters |
| How varied are the requests? | Narrow and repetitive favours routing; open-ended favours an agent |
| Does it act for a specific person? | Per-user permissions change the tool layer completely |
| What does success look like? | If nobody can define it, you cannot evaluate it |
| What happens when it gives up? | The handoff path is part of the design, not an afterthought |
The two questions that change a design most are "can you write down the steps?" and "what does a wrong action cost?". Everything else adjusts budgets and limits. Those two decide whether you build an agent at all, and how much machinery has to surround it.
Part 2 — The Brief
We sell project-management software to businesses. We want an assistant that handles customer support requests: billing questions, seat changes, refund requests, and "why is this feature not working" questions. It should resolve what it can and pass the rest to our support team. Customers use it in-app. We get around 3,000 requests a day. It should feel responsive. Getting a refund wrong costs us money and trust.
Turning that into numbers:
| Requirement | Value |
|---|---|
| Users | Paying customers, in-app, identified and authenticated |
| Volume | ~3,000 requests/day |
| Request mix | Roughly 60% simple lookups, 30% multi-step, 10% genuinely hard |
| Latency target | Something visible in 2 seconds; resolution within ~20 |
| Actions available | Read account data; change seats; issue refunds; open a ticket |
| Cost of a wrong action | Moderate to high — a wrong refund is real money |
| Permissions | Strictly scoped to the requesting customer's own account |
| Success measure | Fraction of requests resolved without a human, with no reversal afterwards |
That last row is the valuable one. "Resolved without a human, and not reversed later" is countable, tied to money, and it cannot be gamed by an agent that answers confidently and wrongly — a reversal cancels the win.
Part 3 — Do the Arithmetic
Five minutes here removes several arguments later.
TRAFFIC
3,000 requests/day ÷ 10 busy hours = 300/hour ≈ 0.08/second
Assume a 5× peak ≈ 0.4/second peak
STEPS (from a prototype run over 100 real tickets)
Simple lookup 1–2 tool calls → 3 steps
Multi-step resolution 3–5 tool calls → 7 steps
Hard / escalated caps out → 12 steps
Weighted average ≈ 5 steps
TOKENS PER TASK
Stable prefix (system prompt + 8 tools) ≈ 1,800 tokens, repeated every step
Growing transcript over 5 steps ≈ 9,000 tokens cumulative
Total input per average task ≈ 18,000 tokens
Total output per average task ≈ 1,200 tokens
DAILY VOLUME
3,000 × 18,000 input ≈ 54M input tokens/day
Of which the cacheable prefix is ≈ 27M (5 steps × 1,800 × 3,000)Two conclusions fall straight out of this. First, 0.4 requests per second is a trivial load — your constraints are cost and correctness, not scale, and anyone proposing a distributed architecture is overbuilding. Second, half your input tokens are the same 1,800 tokens sent over and over. Prompt caching on that prefix is the single largest cost lever available, and it requires no change to the agent's behaviour at all.
Part 4 — The Decisions
Each choice names the requirement that drove it.
Shape: agent, workflow, or both?
What each coordination pattern costs
No agent at all. One prompt, one answer. Always try this first.
Read the task and answer it
Wall-clock time so far
0 ms
One call was 900 ms
Cost so far
0.00 units
One call was 1.00
Run them all. Two results are worth sitting with: the router can beat a single call on cost, because it sends easy work somewhere cheap. And parallel workers cost three times more but finish almost as fast as one, because you wait for the slowest, not the sum.
| Decision | Choice | Driven by |
|---|---|---|
| Overall shape | Router in front, single agent behind | 60% of traffic is a simple lookup that needs no agent at all |
| The router | Small model classifying into lookup / resolve / escalate | Cheap, fast, and it keeps the easy majority off the expensive path |
| The agent | One ReAct agent with a plan step for multi-step work | Steps depend on what earlier lookups return, so the path cannot be fixed |
| Multi-agent | No | Nothing here needs separate expertise; it would add cost and failure modes for nothing |
Notice that the majority of this system is not an agent. "Check my invoice" is a lookup with a template answer. Routing it away is the difference between a five-step agent run and a single cheap call, on 1,800 requests a day.
Tools
| Decision | Choice | Driven by |
|---|---|---|
| Tool count | 8, grouped by area | Choice quality drops past roughly fifteen |
get_account | Returns plan, seats, billing status, and recent invoices in one call | Fewer, higher-yield tools mean fewer steps |
search_help_articles | Retrieval over product documentation | Most "not working" questions are documented |
change_seats | Bounded: within the plan's limits only | Keeps a routine action safe enough to automate |
issue_refund | Capped amount, human approval above it | A wrong refund is real money |
create_ticket | The handoff path, always available | Giving up must be a first-class action |
| Permissions | Every tool scoped server-side to the caller's account | Customers must never reach another account, whatever the prompt says |
| Errors | Distinct failure results, never empty lists | An error read as data produces confident wrong answers |
The loop
| Decision | Choice | Driven by |
|---|---|---|
| Iteration cap | 12 | 90% of real tasks finish within 7 steps; 12 covers the hard tail without funding a runaway |
| Wall-clock cap | 25 seconds | The responsiveness requirement |
| Cost cap | Per task, enforced in the harness | Bounded worst case per request |
| Calls per tool | 3 | Stops the retry-the-same-search cycle |
| On cap | Create a ticket with the trace attached | Never truncate and answer anyway |
| Definition of done | Explicit criteria per request type | The most common cause of loops is an undefined finish line |
Context and memory
| Decision | Choice | Driven by |
|---|---|---|
| Working memory | Full transcript up to 6 steps, then summarise | Runs are short; compaction only matters on the long tail |
| Restate the goal | Appended near the end of context every step | Prevents drift on longer runs |
| Long-term memory | Account history and past tickets, retrieved at the start | Context the agent needs but should not have to search for |
| Tool output | Trimmed to the fields the agent uses | Raw API dumps waste context and bury the useful line |
Model and cost
| Decision | Choice | Driven by |
|---|---|---|
| Router model | Small | Classification is easy and high-volume |
| Agent model | Mid-tier, pinned version | Tool choice quality matters; pinning makes upgrades deliberate |
| Prompt caching | On the system prompt and tool definitions | Half of all input tokens, at no behavioural cost |
| Streaming | Final answer only | Intermediate reasoning is not something customers should read |
| Progress display | "Checking your account…" per tool call | Makes 15 seconds feel like work rather than a hang |
Safety and operations
| Decision | Choice | Driven by |
|---|---|---|
| Refunds above the cap | Human approval, showing the exact amount and reason | Irreversible and expensive |
| Seat changes | Automatic within plan limits, ticket above them | Routine and reversible |
| Untrusted content | Help articles and ticket text marked as data, never instructions | Customers can write anything into a ticket |
| Tracing | Every step, with arguments, scrubbed of card and personal data | Nothing is debuggable without it |
| Metrics | Resolution rate, reversal rate, steps p95, cost p95, loop rate, handoffs | These map directly to the success measure |
| Kill switch | One setting disables all write tools | Read-only degradation beats an outage |
The resulting design
Final design
Part 5 — Build It in Stages
Do not build the diagram above on day one. Build the smallest version, measure it, and add only what the measurements justify.
Staged rollout
Stage 1 — One model call, no tools
Answer from help articles only. Build the evaluation set from 100 real tickets at the same time.
Stage 2 — Measure
Resolution rate by request type. Now you know which requests actually need tools.
Stage 3 — Add read-only tools
Account lookups. Still cannot change anything, so the blast radius is zero.
Stage 4 — Add the router
Keep the easy majority off the agent path once you know what the majority is
Stage 5 — Add write tools, gated
Seat changes first, refunds last, with approval above a low cap that you raise slowly
Stage 6 — Ship to a slice of traffic
Compare resolution and reversal rates against the human baseline before widening
Stage 1 exists to produce measurements, not a product. Teams that skip it end up with eight tools, a planner, and no idea which part is carrying the resolution rate — so they cannot remove anything, and every subsequent change is a guess. It also forces the evaluation set into existence while there is still time for it to shape the design.
Part 6 — Change One Requirement
Now swap a single line of the brief. The assistant is for hospital staff, answering clinical questions from protocols and patient records. A wrong action can harm a patient.
Volume, traffic, and step counts are unchanged. Look at how much of the design flips anyway:
| Decision | Support version | Clinical version | Why it changed |
|---|---|---|---|
| Latency target | Under 2 seconds to first output | Seconds are fine | Clinicians will wait for a careful answer |
| Write tools | Seat changes and capped refunds | None without approval | No action should reach a patient unreviewed |
| Human in the loop | Above a refund threshold | Required for anything actionable | Clinical governance |
| Model | Mid-tier | Strongest available | Do not economise where the downside is harm |
| Router to a small model | Yes | No | Misrouting a clinical question is not a cost problem |
| Iteration cap behaviour | Create a ticket | Refuse explicitly and loudly | Silence is far safer than a partial clinical answer |
| Untrusted content | Marked as data | Marked as data, and no write tools in the same agent | Split reading from acting entirely |
| Audit log | Traces with scrubbing | Full, retained, per-user, per-record | Regulatory requirement |
| Evaluation | Resolution and reversal rates | Plus clinician review of trajectories | Right answer by a bad route is unacceptable here |
| Tool scoping | Per customer account | Per role and per patient relationship | Access is far more granular |
| Tracing | Yes | Yes | Unchanged |
| Caps on iterations, cost, wall-clock | Yes | Yes | Unchanged |
| Distinct tool errors | Yes | Yes | Unchanged |
Notice which rows did not change. Hard caps, full tracing, scoped permissions, and unambiguous tool errors are the same in both designs — they are engineering fundamentals that serve every agent. What changed is everything downstream of "what does a wrong action cost?": autonomy, approval gates, model choice, refusal behaviour, and oversight.
Concept Checks
Check yourself
Why put a router in front of the agent rather than letting the agent handle everything?
Because 60% of the traffic is a lookup with a templated answer, and running a five-step agent loop over it costs several times more, takes several times longer, and introduces failure modes a direct lookup does not have. The router is a cheap classification call that keeps the expensive, unpredictable path for the requests that genuinely need it. It also makes the agent's own evaluation cleaner, because the easy cases no longer inflate its success rate.
Why does 'resolved without a human, and not reversed' beat 'resolution rate' as a success measure?
Because resolution rate alone rewards confidence. An agent that answers everything and gets a fifth of them wrong scores brilliantly on resolution and terribly in reality. Adding the reversal condition means a wrong resolution actively cancels the credit for it, so the metric can only be improved by being right — and both halves are already recorded by the ticketing system, with no extra instrumentation and no survey response bias.
Why size the token arithmetic before choosing anything else?
Because it changes which optimisations are worth doing. The calculation shows that half the daily input tokens are the same 1,800-token prefix repeated on every step of every task, which makes prompt caching the largest single cost lever and one that requires no behavioural change. It also shows that 0.4 requests per second is a trivial load, which removes an entire category of infrastructure discussion before anyone starts it.
Why add write tools last, and refunds after seat changes?
Because blast radius should grow only as fast as your evidence. Read-only stages let you measure the agent's judgement while it cannot damage anything. Seat changes come next because they are bounded and reversible. Refunds come last, behind a low approval cap, because they move real money irreversibly — and by then you have a trajectory record showing how the agent behaves when it is uncertain, which is exactly what you need to decide where the cap belongs.
Which parts of an agent design stay constant across use cases?
Hard caps on iterations, cost and wall-clock; full step-level tracing; permissions enforced server-side per tool; tool errors that cannot be mistaken for data; and a designed handoff path. These address problems every agent has, regardless of domain. What varies is the layer above — how much autonomy the agent gets, what needs approval, which model, how it refuses, and how closely humans review it — all driven by what a wrong action costs.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| First decide whether you need an agent | If you can write the steps down, write them down |
| The two decisive questions | Can you write down the steps, and what does a wrong action cost |
| Most of a good agent system is not the agent | Route the easy majority away from the expensive path |
| Do the token arithmetic early | It names your biggest cost lever in five minutes |
| Fewer, higher-yield tools | Fewer steps, better choices, lower cost |
| Bound every write tool | Limits in the tool, approval above them |
| Attach a reason to every choice | A design you cannot justify is a design you cannot revise |
| Success must be countable and un-gameable | Resolution alone rewards confidence; add reversals |
| Grow the blast radius with the evidence | Read-only, then bounded writes, then money |
| Constant across designs | Caps, traces, scoped permissions, distinct errors, handoff |
Next
That completes the track. Keep the Glossary open as a reference, then go and build: Tool Calling Agent.
Failure Modes & Debugging
Every way an agent goes wrong — from silent tool failures and vague descriptions to loops, drift, and non-determinism — and how to find the real cause from the trace
Glossary
Every agent term and abbreviation, expanded and explained in one place — from ACI and blast radius to ReAct, MCP, trajectory, and idempotency