Why Memory Is the Hard Problem in AI Agents
LLMs are stateless by default — every request arrives without any memory of previous interactions unless the relevant history is explicitly included in the context. For simple chatbots, this is manageable: include the last N conversation turns and the user’s question feels contextually grounded. For AI agents that operate over extended tasks, build persistent relationships with users, or maintain working knowledge across multiple sessions, the stateless default is a fundamental limitation. Building effective memory into agent systems requires deliberate architectural design — choosing the right types of memory for each use case, managing the storage and retrieval of that memory efficiently, and deciding what to remember, what to forget, and when.
The Four Memory Types
In-context (working) memory is everything included in the current prompt’s context window. It is the most reliable memory — the model attends directly to in-context information — but it is limited by context window size, costs tokens on every request, and is lost when the session ends. Working memory is appropriate for the information that must be immediately available to the model in its current task: the current conversation turns, retrieved relevant documents, the agent’s current task description and progress, and the results of recent tool calls.
External (episodic) memory is a persistent store of past interactions, events, and experiences that the agent can retrieve when relevant. Implemented as a vector database of past conversation summaries, user interactions, or task outcomes, episodic memory allows agents to recall relevant past experience when it is pertinent to a current task. “The last time this user asked about X, they found approach Y most helpful” — that context can be retrieved and included in the prompt for the current interaction, making the agent feel like it remembers the user without including the entire interaction history in every prompt.
Semantic (knowledge) memory is a store of general facts, domain knowledge, and learned information that the agent can access regardless of when it was acquired. In practice, this is often implemented as a RAG system over a curated knowledge base — documents, policies, product information, domain expertise — that the agent retrieves from when it needs factual grounding. Semantic memory is the component most directly improved by knowledge base quality investments.
Procedural memory encodes how to perform tasks — the agent’s learned skills and strategies. In LLM systems, procedural memory is typically encoded in the system prompt (explicit instructions about how to approach tasks) or in fine-tuning (behavioural patterns trained into the model’s weights). Unlike the other memory types, procedural memory is not retrieved per-query — it shapes the model’s behaviour by default. Prompt engineering and fine-tuning are, in effect, the mechanisms for writing to the agent’s procedural memory.
Short-Term Memory: Managing Conversation History
Conversation history management is the most common memory challenge in deployed LLM applications. The naive approach — include all previous turns in every prompt — fails at scale as conversations grow long. Three effective patterns manage this. Fixed sliding window: include the last K turns, discarding older ones. Simple, predictable cost, but loses context from earlier in the conversation. Suitable for simple Q&A where recent turns are all that matter. Hierarchical summarisation: maintain a running summary of earlier conversation turns that updates as the conversation progresses. The summary provides semantic continuity at lower token cost than the full transcript. The quality of the summary is critical — a bad summary loses important context more thoroughly than a sliding window, which at least preserves the full text of recent turns. Selective retention: identify and retain specific turns that established important context — user preferences stated, decisions made, constraints introduced — while discarding lower-information turns (acknowledgements, simple confirmations, off-topic diversions). The hardest to implement but the most token-efficient and semantically accurate.
Figure 1 — The Four Memory Types in LLM Agents
Long-Term User Memory: Personalisation Across Sessions
Building agents that remember users across sessions requires an explicit user memory store — a structured record of relevant facts about each user that is retrieved and included in the context at the start of each new session. The key design decisions are: what to store, how to update it, and how much to include per session. Store facts that are stable (user preferences, professional context, stated goals, relevant background) rather than ephemeral (what the user asked about today, which might not matter tomorrow). Update the user memory at the end of each session by summarising any new relevant facts that emerged during the conversation. Include a compressed user memory summary at the start of each new session — typically 200–400 tokens — that gives the agent enough context to feel continuous without consuming excessive prompt space.
Privacy is a non-trivial concern with user memory stores. Users should know what you are storing, should be able to request deletion, and the storage should comply with applicable data protection law. GDPR and CCPA both have implications for persistent AI memory about individuals — treat user memory data with the same data governance discipline you apply to other personal data, because that is exactly what it is.
Tool Outputs as Working Memory
For agents that call tools — search engines, databases, APIs, code execution environments — the outputs of those tool calls constitute a form of working memory that must be managed carefully. As an agent accumulates tool results across many steps of a long task, the context fills with intermediate outputs that may or may not remain relevant to the current step. Two patterns manage this: selective inclusion (only include tool results that are still relevant to the remaining task, discarding superseded results) and tool result summarisation (compress verbose tool outputs into concise summaries that preserve the information needed for downstream steps). An agent that accumulates full tool results without management will exhaust its context window on long tasks or spend excessive tokens on intermediate outputs that are no longer needed.
Memory Retrieval: When and What to Include
Having stored memory is not sufficient — retrieving the right memory at the right time is the hard part. For episodic and semantic memory, semantic retrieval (embedding the current query and finding the nearest relevant memory items) is the standard approach. But semantic similarity is an imperfect proxy for relevance — the memory item that is semantically most similar to the current query is not always the one that would be most useful. Recency weighting (preferring more recent memories over older ones for the same semantic similarity score), frequency weighting (preferring memories that have been relevant in multiple past sessions), and importance weighting (memories explicitly marked as high-importance at storage time) all improve retrieval quality over pure semantic similarity. The right weighting depends on your agent’s specific use case — test different retrieval strategies on representative sessions and measure whether the retrieved memory was actually used by the agent in its response.
Forgetting: The Underappreciated Half of Memory
Memory systems that only add, never prune, eventually become noise — too much stored memory degrades retrieval quality by making it harder to find the actually relevant items among the accumulated irrelevant ones. Designing a principled forgetting mechanism is as important as designing storage. Time-based decay (automatically expire memories older than N days unless explicitly marked as permanent), relevance decay (memories not retrieved in any of the last K sessions are considered stale and expired), and explicit deletion (users can request specific memories be deleted, and the system complies) are the standard approaches. Combine them: time-based decay for routine interactions, explicit deletion for user-requested privacy, and importance-flagging to protect high-value memories from automatic decay. An agent with a well-tuned forgetting policy delivers better personalisation than one with unlimited memory accumulation, because its retrieved context is more signal and less noise.
Practical Implementation: Getting Started
For most teams, the right starting point for agent memory is in-context history management with a simple sliding window, plus a RAG-based semantic memory store for domain knowledge. These two components — short-term conversation context and long-term knowledge retrieval — address the majority of memory needs for typical agent deployments without the complexity of full episodic or procedural memory systems. Add episodic user memory once you have validated that cross-session personalisation improves your key quality and retention metrics. Add procedural memory refinement (fine-tuning or more sophisticated system prompt engineering) once you have identified specific behavioural patterns that prompting alone cannot reliably produce. This sequenced approach mirrors the broader principle of starting with what is necessary and adding complexity only when it demonstrably improves outcomes. Memory architecture is one of the areas where over-engineering relative to actual need is most common — the simplest memory system that meets your quality requirements is almost always the right choice, leaving headroom to add sophistication as your application and your understanding of user needs both mature.
The teams that build effective agent memory are those who measure whether each memory component actually improves user experience before investing in the next layer of complexity — and who resist the temptation to build the full theoretical memory architecture before they have evidence that their users actually benefit from it.