What Is Agent Memory? – Definition and How It Works
Learn what agent memory is, how working, episodic, and semantic memory differ, where each is stored, and why retrieval beats accumulation.
Agent memory is everything an agent knows beyond its model weights: what happened earlier in this run, what happened in previous runs, and what it has learned about the user or domain.
1. Why Memory Is External
A model has no persistence. Between calls it retains nothing, so every "memory" is something the application stored and chose to include in the next prompt.
That makes memory a retrieval problem: what to store, and what to bring back at the right moment.
2. Three Kinds
| Kind | Holds | Lifetime | Typical store |
|---|---|---|---|
| Working | The current run's state and observations | One run | Context window |
| Episodic | What happened in past runs | Across runs | Database or log, retrieved by relevance |
| Semantic | Durable facts about the user or domain | Long-term | Key-value store or vector index |
Confusing them is the usual design error: dumping every past conversation into the prompt is treating episodic memory as working memory.
3. Working Memory Is a Budget
Everything in the context window competes for space: the goal, tool schemas, observations, and the answer being generated.
Practical management means summarising older steps, storing large results externally and passing references, and keeping observations trimmed to what the next decision needs.
4. Episodic Memory Needs Retrieval, Not Accumulation
Past runs grow without bound, so they cannot all be included. They must be searchable — by identifier, by recency, or by embedding similarity — and only the relevant few brought back.
The retrieval criterion matters more than the storage format. "The last five runs" is rarely the right selection; "runs about this entity" usually is.
5. Semantic Memory Needs Curation
Durable facts — a user's preferences, a project's constraints, a domain's terminology — are valuable precisely because they are few and stable.
Extract them deliberately rather than storing everything, timestamp them, and allow correction. An agent that remembers a preference the user has since changed is worse than one that remembers nothing.
6. Failure Modes
- Unbounded growth. Memory that only accumulates eventually returns noise.
- Stale facts. Nothing expires unless expiry is designed in.
- Contradiction. Two remembered facts disagree, and the model picks one arbitrarily.
- Privacy leakage. Memory captured from one context surfacing in another.
- Context crowding. Retrieved memory displacing the current task's information.
7. Designing It
- Decide what is worth remembering before deciding how to store it.
- Keep the three kinds separate, with different retention rules.
- Retrieve selectively; never inject the whole store.
- Timestamp everything and prefer recent facts on conflict.
- Make memory inspectable and correctable by the user.
- Apply the same privacy rules as to any other stored personal data.
8. Summary
Agent memory is external state the application maintains and selectively re-injects: working memory for the current run, episodic for past runs, semantic for durable facts.
The discipline that makes it work is retrieval rather than accumulation — deciding what comes back, and when, rather than how much can be stored.