# Agent Memory Is a Real Problem. Most Builders Just Have Not Hit It Yet. > Every LLM call is stateless; agent frameworks solve that within a single run, which is why the memory problem only shows up once the run ends. _Gaurav Dadhich ยท 2026-09-18_ Every LLM call is stateless. Agent frameworks solve that within a single run, which is why [the memory problem](https://www.maximem.ai/the-memory-problem) does not show up until the run ends. The first time it does show up, it does not look like a memory problem at all. It looks like an agent that asks a returning customer the same question it asked last week, a support copilot that contradicts a decision it made two sessions ago, or a bill for tokens that grew faster than usage did. By the time the pattern is legible, it has usually already cost something. This piece is about that gap: what [agent memory](https://www.maximem.ai/glossary/agentic-memory-system) actually is, why so many teams building agents have not run into the problem yet, why the obvious fixes look like solutions and are not, and how we approach the problem at Maximem. ## What is Agent Memory? Agent memory is the layer that lets an agent carry facts, preferences, episodes, and entities across turns, sessions, users, and tools, instead of treating every interaction as a cold start. A model on its own has no memory; each call is stateless, and whatever continuity an agent appears to have was engineered on top of that by somebody. Within one run, the framework holds that state. Across runs, across days, across a customer's whole relationship with your product, something else has to. Memory is what turns a capable-but-forgetful model into an agent that gets better the more it is used. It is the difference between a customer explaining their situation once and a customer explaining it every single time. ## Is Agent Memory an Imagined Problem or a Real One? It is real, and the reason it does not feel real to most people building agents today is worth taking seriously, because the same reasons explain why the teams who have hit it treat it as one of the harder problems in the stack. ### Why most agent-builders have not come across the agent-memory problem yet The honest answer is that most agents in the wild are not yet operating in the conditions that make memory bite. Six things are usually true. **Production adoption is still early, and scale is low.** A large share of what gets called an "agent" today is an experiment or a demo. Real agents in production, carrying real users across real time, are still the minority, and the ones that exist are often running at a scale where the seams do not tear yet. Memory failures are a function of returning users and elapsed time. With few returning users and little elapsed time, the failure has nothing to act on. **Many teams are building workflows, not agents.** A rigid flow that moves a request through predefined steps does not need to remember anything, because the state it needs lives in the flow itself. This is a legitimate way to build, and for many problems it is the correct one. It also means the builder never encounters the memory problem, because they never built the thing that has the problem. The distinction matters: an agent decides what to do next; a workflow was told. Only the first one has anything to remember. **Many agents are non-conversational by design.** Two common shapes here. The first is short-call voice agents built for a single advertising or qualifying interaction, where the entire job is over in ninety seconds and there is no second conversation to be continuous with. The second is the far larger category of teams converting existing SaaS into AI-enabled SaaS, bolting a natural-language surface onto software whose data model and workflows were designed for humans clicking buttons. That is AI-enabled software, and it is valuable, but it is not the same as reimagining the work itself around an agent that accumulates context. The first shape has nothing to remember. The second remembers through its existing database and does not yet feel the absence of an [agent memory layer](https://www.maximem.ai/research/agent-skills/skills-vs-agent-memory). **Token costs have not started pinching, because scale is low.** The most common way to fake memory is to stuff prior history into the prompt. At low volume, that is cheap enough to ignore. The cost of faking memory this way scales with conversation length multiplied by user count multiplied by frequency, so it stays invisible until two of those three start climbing, at which point it arrives as a line item nobody forecast. **Returning customers are sparse, and there are almost no evals for agent CSAT yet.** Memory quality is only observable when the same user comes back and the agent either does or does not carry them forward. Most teams are not yet measuring satisfaction from agent interactions across sessions, so even where memory is quietly failing, there is no instrument pointed at it. The failure is real; the detector is not installed. **Locally-run agent harnesses create the illusion that long context solves it.** A great deal of early agent building happens in a local [harness](https://www.maximem.ai/glossary/harness) against a single developer's own history, where a long [context window](https://www.maximem.ai/glossary/context-window) really does hold everything relevant. It is easy to conclude from that experience that a big enough window is memory. It is not, and the reasons that it is not become obvious the moment there is more history than a window can hold, or more than one user whose histories must not bleed into each other. None of these are signs that the problem is not real. They are the conditions under which a real problem stays hidden. Every one of them dissolves as agents move to production, run longer, serve more users, and start being measured. ### For builders who have hit the problem, the alternatives look easy. They are not. Once a team does run into it, a set of primitives and adjacent tools present themselves as the fix, and each of them looks appealing because each of them solves a piece of the shape of the problem. The trap is that they look sufficient right up until someone builds an eval and measures precision and recall on retrieved context. Without that measurement, a partial solution and a real one are indistinguishable. **[Larger context windows](https://www.maximem.ai/memory-vs-context-windows).** A bigger window is a bigger desk, not a better filing system. It postpones the problem by letting you hold more at once, but it does not decide what is worth holding, it does not persist anything past the session, and it does not keep one user's context out of another's. Past a certain history length the window is full again, and now you are paying to re-read everything on every turn. Bigger context is real capability that solves a different problem. **[Vector RAG](https://www.maximem.ai/memory-vs-vector-rag),** [**pgvector**](https://github.com/pgvector/pgvector)**, [graph RAG](https://www.maximem.ai/glossary/graph-rag).** Semantic search over stored chunks is the most common homegrown memory, and it is useful for retrieval over documents. The failure mode is that raw conversation is not a document. Chunk-and-embed does not resolve that "the client", "Acme", and "Acme Corp" are one entity, it does not know which of two contradictory statements is current, and it retrieves on similarity rather than on relevance to the moment. High recall on chunks is not the same as retrieving the right fact, and the gap between those two is exactly what an eval exposes and intuition does not. **Skills.** [Skills](https://www.anthropic.com/news/skills) give an agent reusable, packaged competence, which is a real and useful primitive. It is also a different axis from memory. A skill is knowledge the agent brings to every user; memory is what the agent has learned about this user, this account, this history. Confusing the two leads to building excellent procedures that still forget who they are talking to. **Agent framework state machines.** This is the objection every serious builder reaches for first, and it deserves a straight answer. Frameworks such as [LangGraph](https://langchain-ai.github.io/langgraph/) give you checkpointers, thread state, and cross-thread stores, and they do their job well: they carry an agent through a long, multi-step run without dropping most of what matters, and for completing a single execution correctly they are the right tool. The checkpointer exists to make one run reliable, and it does. What it was never trying to do is decide what, out of thousands of past interactions, is worth remembering next month, resolve entities across those interactions, reason about what has since changed, and keep all of it correctly scoped across many tenants. Those are not gaps in the framework. They are simply a different problem, one that begins where the run ends. ### DIY is the first real response, because it looks easy until you run benchmarks The teams who see through the primitives usually reach the same next conclusion: we will build it ourselves. It is an understandable instinct, and it is where a lot of engineering time goes to quietly die. The reason is that the naive version really is a weekend of work, so the effort estimate is anchored on the wrong version. The first cut, embed the history, store it, retrieve by similarity, works in the demo. Then the benchmarks start. Precision and recall on retrieved context, contradiction handling when a user changes their mind, entity resolution across messy references, temporal reasoning about what is current, isolation across tenants, latency inside the conversation hot path, and retrieval quality that does not degrade as the store grows. Each one is a research problem with a literature behind it, and getting the whole thing right is iterative depth work that is hard to do well without a disproportionate investment of time. The build-it-ourselves estimate is almost always the estimate for the weekend version, and the production version is a different project entirely. This is the actual shape of the thing: agent memory is a data pipeline and fast-search problem, not a storage problem. Storing conversations is trivial. Deciding what to keep, resolving it into entities, keeping it current, forgetting what has gone stale, and returning exactly the right context in single-digit milliseconds is the work. ## So, What Is Agent Memory? The Jobs To Be Done Stated as the jobs a real memory layer has to do, rather than as a definition, the shape becomes concrete. A memory layer earns its place when it does the following: - **Persist context across sessions, users, and tools.** Facts, preferences, episodes, and entities survive past the end of a run, so a returning user is carried forward rather than re-interviewed. A support agent that already knows the customer's plan, their last three tickets, and the workaround it gave them last week is a different product from one that starts every chat by asking for an account number. - **Resolve entities automatically.** "The client", "Acme", and "Acme Corp" collapse into one entity instead of fragmenting into three, so the agent reasons about a coherent world rather than a pile of near-duplicates. For a B2B sales or account agent, this is the difference between one accurate view of a company and several partial ones that disagree. - **Reason about time.** The layer knows what is current versus stale, so the agent acts on the latest state of a fact and not on something a user corrected two months ago. A scheduling or operations assistant that still believes a cancelled meeting is on the calendar is worse than no assistant. - **Scope memory across tenancy levels.** Context is visible at the right level, from an individual user up through a customer organization and beyond, and never leaks across the boundaries that separate one tenant's data from another's. For a multi-tenant B2B copilot, this is not a nicety, it is the line between a working product and a data incident. - **Manage PII and permissions.** Memory respects who is allowed to see what, retrieval never surfaces something the current user should not have, and sensitive data is retained, masked, and retired according to policy rather than sitting in a prompt forever. A healthcare or fintech agent lives or dies on getting this right. - **Forget consciously.** Stale and low-value memory is retired on purpose, so the store does not bloat into noise and retrieval stays sharp as history grows. - **Learn what to keep.** The layer improves what it retains over time instead of dumping raw history back into the prompt, so a coding agent gradually holds the conventions and decisions that matter to a codebase rather than every line anyone ever typed. - **Manage memory across multiple agents.** Context passes cleanly between agents in a multi-agent system, with shared knowledge and per-agent private memory kept distinct, rather than each agent re-deriving the world from scratch at every handoff. - **Reduce token cost.** Faking memory by appending history grows cost quadratically with conversation length; a memory layer that retrieves only what the turn needs and compacts the rest keeps cost close to linear, which is the difference between a margin that survives scale and one that does not. A high-volume support deployment feels this first. - **Return context fast enough to sit in the conversation.** Retrieval happens inside the latency budget of a live turn, not as an offline batch job. For a voice agent, where a pause of a second reads as a broken call, this constraint is unforgiving. ## Signs You Need Agent Memory The clearest signals that you have crossed from "memory is overkill" into "memory is now the bottleneck": - Your agent is conversational, multi-turn, or long-running, and users come back. The moment a second session with the same user matters, memory matters. - You are serving multiple customers from one product, and keeping each tenant's context isolated has become something you think about. - You are stuffing conversation history into the prompt to fake continuity, and the token bill is starting to track conversation length rather than value delivered. - Your agent repeats questions, contradicts its own earlier decisions, or treats a returning user as a stranger. - You are running a multi-agent system and finding that context has to be re-derived every time control passes from one agent to another. - You have started building memory in-house, and the second version of that build is turning out to be much larger than the first. If two or more of these are true, the problem is no longer hypothetical for you, and the fastest way to know how much it is costing you is to measure it rather than guess. We run that measurement for you, free. You fill in a short form, we run our diagnostic against your agent, and we show you where its memory breaks today: what it fails to recall, where precision and recall drop, and what that is costing you in tokens and in users repeating themselves. You get the findings whether or not you ever use Maximem Synap. [Start the free diagnostic here.](https://docs.google.com/forms/d/e/1FAIpQLSeyTWs93X1aD3avdiY7B9ZtCJrMDNCs-PsDHpsAdCnXSSp3TQ/viewform) ## How We Approach Agent Memory at Maximem The way we see it, agent memory is not merely a storage and retrieval problem, it is an architecture problem, and treating it as one is what separates a memory feature from a memory system. Maximem Synap ingests conversations asynchronously and extracts structure, not raw text. That structure is stored across three kinds of store at once, vector, graph, and file, and retrieval nets across all three together rather than betting the whole system on similarity search alone. Because ingestion runs in the background, the expensive extraction work never sits in the conversation hot path, and context is pre-fetched while the conversation is still going, which is how in-conversation retrieval stays under 15ms at P75. Conscious forgetting runs as background cycles over the stores, retiring what has gone stale so precision holds as history grows, and extraction, scoping, and retention are all configurable per deployment rather than fixed. On the public benchmarks this comes out as the most accurate memory system we are aware of, with 92% on LongMemEval and 93.2% on LoCoMo, against next-closest published scores in the low seventies. The full picture is laid out in our research paper, [Agentic Context Management: Agent Memory Is Not Merely a Storage & Retrieval Problem, It Is an Architecture Problem](https://www.maximem.ai/blog/agentic-context-management-paper). It treats memory as a five-stage lifecycle rather than a store: architecting a memory schema per agent, ingesting asynchronously with entity resolution, scoping retrieval narrowest-first across tenancy, anticipating the context an agent will need before it asks for it, and compacting with validation so cost stays close to linear in the length of the conversation rather than quadratic. Figure 1 of the paper draws this as a cycle around the agent. A store makes none of those decisions. Storage is one moment in the lifecycle, not the whole of it. --- Source: [https://www.maximem.ai/blog/agent-memory-is-a-real-problem-most-builders-just-have-not-hit-it-yet](https://www.maximem.ai/blog/agent-memory-is-a-real-problem-most-builders-just-have-not-hit-it-yet)