Memory
Working, short-term, and long-term memory — what to keep, what to forget, and how agents remember anything at all between calls
Memory
TL;DR
The model remembers nothing between calls. It is stateless. Every request must carry everything it needs to know. So "agent memory" is not a feature of the model — it is a system you build that decides what to put into each request. That decision is the whole subject.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~20 minutes |
| Prerequisites | Planning & Reasoning |
| You will understand | The three kinds of memory, and what to keep, summarise, or drop |
The Model Has No Memory
Start here, because everything else follows from it:
Call 1: "My name is Sarah." → "Nice to meet you, Sarah."
Call 2: "What is my name?" → "I don't know your name."Nothing was remembered. If call 2 had included the text of call 1, the model would have answered correctly — not because it recalled anything, but because the information was in front of it.
Every apparent memory in every AI product is this: text put back into the prompt. There is no hidden store inside the model. When people say an agent "learned" something, they mean a database was written to and later read from.
The Three Kinds
Agent memory
Working memory
Short-term memory
Long-term memory
| Type | Holds | Lifetime | Where it lives |
|---|---|---|---|
| Working | The current run — every thought, action, result | This task | The context window |
| Short-term | The current conversation | This session | Context, plus a store |
| Long-term | Facts worth keeping about a user or domain | Indefinite | A real database |
Working Memory — the run itself
This is the trace of the current task, and it has one defining property: it only grows.
Iteration 1: system + tools + task ~1,200 tokens
Iteration 2: + thought 1 + action 1 + observation 1 ~1,900
Iteration 3: + thought 2 + action 2 + observation 2 ~2,700
Iteration 8: ... ~9,400| Problem | Effect |
|---|---|
| Cost climbs every step | You re-send everything, every iteration |
| The window eventually fills | Long runs simply stop working |
| The middle gets skimmed | Models read the start and end most reliably |
| Old detail stops being useful | Iteration 2's raw API dump rarely matters at iteration 9 |
Managing it
| Technique | What it does | Trade-off |
|---|---|---|
| Summarise observations | Store "3 suppliers found, all EU-based" instead of a 4,000-token payload | Loses detail you might want later |
| Keep a running state | A compact structured block — goal, findings, remaining steps — updated each iteration | Extra bookkeeping in your code |
| Drop old observations | Remove results already used and no longer needed | Cannot revisit them |
| Compact on a threshold | At 70% full, summarise everything older than the last few steps | One extra model call |
| Restate the goal late | Put the goal and remaining plan near the end of the request | A few tokens per call |
Never append raw tool output. This is the single highest-value habit in agent engineering. One verbose API response can consume most of your budget and push the reasoning that mattered out of reliable attention. Summarise as it arrives, and keep the full payload outside the context in case you need it.
Short-Term Memory — the conversation
Across turns of a chat, you need the user's earlier messages without carrying all of them forever.
| Strategy | How it works | Trade-off |
|---|---|---|
| Sliding window | Keep the last N turns | Simple. Forgets early details. |
| Summary | A running summary of the conversation | Remembers the shape, loses specifics |
| Summary + window | Summary of old turns, last few in full | The usual choice |
| Everything | Send it all | Works briefly, then gets slow and expensive |
Watch tokens per turn, not per request. A conversation that is fine at turn 3 can be slow and costly by turn 25 while nothing appears to be wrong. Log context size per turn and alert on it.
Long-Term Memory — across sessions
Things worth remembering after the session ends.
| Kind | Example |
|---|---|
| Facts about the user | "Prefers metric units", "works in the Berlin office" |
| Past decisions | "Approved supplier X in March" |
| Learned preferences | "Wants short answers, no preamble" |
| Task history | "Already researched this company last week" |
How it works
Long-term memory
Step 4 should look familiar — searching a store and putting the results in the prompt is exactly RAG. Long-term agent memory is retrieval applied to things the agent learned rather than documents you loaded. If you know RAG, you already know how this works.
What makes it hard
| Problem | Why |
|---|---|
| Deciding what to store | Store everything and retrieval gets noisy; store too little and it forgets what mattered |
| Facts going stale | "Works in Berlin" was true last year |
| Contradictions | Two memories disagree — which wins? |
| Wrong memories retrieved | An irrelevant memory in the prompt actively misleads |
| Privacy | You are now storing personal data, with everything that implies |
A wrong long-term memory is worse than none. It arrives in the prompt looking authoritative and the agent uses it as fact. Store timestamps, let newer memories supersede older ones, and make memories inspectable and deletable — users must be able to see and correct what you believe about them.
What To Keep, Summarise, and Drop
A practical default:
| Content | Do this |
|---|---|
| The original goal | Keep verbatim, restate near the end |
| The current plan and progress | Keep, updated each step |
| Recent 2–3 observations | Keep in full — they drive the next decision |
| Older observations | Summarise to their conclusion |
| Large tool payloads | Summarise immediately; keep the raw data outside the context |
| Failed attempts | Keep a one-line note so it does not retry the same thing |
| Reasoning from many steps ago | Drop, once its conclusion is in the state block |
Common Mistakes
What goes wrong with memory
Appending raw tool output
The most common and most expensive mistake. One large response can eat the budget and push out the reasoning that mattered.
Assuming the model remembers
It does not, ever. If it is not in the request, it does not exist.
Storing everything long-term
Retrieval gets noisy and starts surfacing irrelevant memories that mislead the agent.
Memories with no timestamps
You cannot resolve contradictions or expire stale facts, so old beliefs win at random.
Memory the user cannot see or delete
A correctness problem and a privacy problem at once. People must be able to correct what you store about them.
No context-size monitoring
You find out at the failure, not before it.
Concept Checks
Check yourself
Why is 'the agent remembered my preference' technically wrong?
Because nothing inside the model changed. The weights are fixed and identical between calls. What happened is that your system wrote the preference to a store earlier, searched that store when the new request arrived, and placed the result into the prompt. The model then read it like any other text. Every memory feature in every product is that loop — a database you built, not recall by the model.
Why is appending raw tool output the most expensive habit in agent engineering?
Because the cost compounds. A 4,000-token payload added at iteration 2 is re-sent on iterations 3 through 10, so you pay for it repeatedly rather than once. It also crowds the window, pushing earlier reasoning toward the middle where attention is least reliable, and it eventually triggers the context limit. Summarising to the conclusion — "3 suppliers, all EU-based" — preserves what drives the next decision at a fraction of the price.
How is long-term agent memory the same problem as RAG?
Because both are retrieval into a prompt. You have a store larger than the context window, so you search it for what is relevant to the current request and insert only that. The chunking, embedding, ranking, and staleness problems are identical — the only difference is that the corpus is facts the agent recorded rather than documents you loaded. The failure modes carry over too, including retrieving something irrelevant that then misleads.
Why is a wrong long-term memory worse than having no memory?
Because it enters the prompt with the same authority as a correct one, and the agent has no way to tell them apart. Without memory the agent asks or reasons from what it can verify; with a wrong memory it proceeds confidently from a false premise, and the error propagates into everything downstream. Timestamps, supersession rules, and user-visible, user-deletable records are the defences.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Stateless model | It remembers nothing; memory is a system you build |
| Working memory | The current run — grows every step |
| Short-term | The conversation — summary plus a window |
| Long-term | A database, retrieved into the prompt. This is RAG. |
| Never append raw output | Summarise as it arrives |
| Running state block | Compact, structured, updated each step |
| Restate the goal late | Where the model reads reliably |
| Timestamp everything | So stale facts can be superseded |
| Wrong memory beats no memory — badly | It misleads with authority |
| Monitor context size | Per turn, before it fails |
Next
Memory decides what goes in the prompt. Next, how to arrange it: Context Engineering.