Agent Memory vs RAG: When Do You Use Which?

Published 27 October 2026 by Gaurav Dadhich. Every paper, price and product page cited here was read on 27 October 2026.

Use RAG for knowledge that belongs to everyone and changes only when someone publishes a new version, and use agent memory for facts that belong to one person or account and get corrected over time. In between sits the case that two papers from October 2026 have reopened: answering questions about a user's own past conversations, where plain retrieval over the raw history matched or beat purpose-built memory systems in six of seven benchmark settings (Bao et al., arXiv 2610.30755).

Agent memory vs RAG: when do you use which?

The decision turns on ownership and change, not on storage technology, because both approaches usually end in the same retrieval step. A refund policy is the same text for every customer and is replaced wholesale when legal signs off a new one, so an index over the policy documents is the right tool and the mechanics of RAG are well understood. A customer saying in March that they deploy on AWS and in June that they moved to Azure is a different shape of data: it belongs to one account, and the second statement retires the first. A similarity search over both chunks returns both, and the model has to guess which is current on every call.

Agent memory is the layer that does work at write time: it decides what is worth keeping, attaches it to the right person, and records that one statement replaced another. That write path is the whole difference, and it is also where the cost and the risk sit, which is why the honest answer to "which one?" is usually "RAG for the shared corpus, memory for the per-person state, and the system of record for anything transactional." The step-by-step build of that loop, from extraction to a minimal Postgres version, is a separate guide.

Two diagnostic questions settle most cases. If the agent gives a wrong answer because it never had the information, you have a retrieval problem and RAG fixes it. If it gives a wrong answer because it had two versions of the information and used the old one, or because it treated a returning customer as a stranger, you have a memory problem, and a bigger index will not help.

Memory vs RAG: which is better for AI agent memory?

On today's question-answering benchmarks, plain RAG over the raw conversation history is as good as or better than most purpose-built memory systems, and anyone choosing between the two should start from that fact. Memory systems still win on the categories those benchmarks under-weight: facts that get updated, questions about time, and deletion.

The strongest evidence is Raw-RAG, posted on 22 October 2026 by Zhengye Bao and five co-authors (arXiv 2610.30755). Their baseline does no extraction, summarising or graph building: it splits timestamped conversations into 256-token windows with a 32-token overlap, takes the top 128 by cosine similarity, reranks them with a cross-encoder down to 15, and hands those to a reader model. They compared it against twelve memory systems, including Mem0, MemOS, EverMemOS, MemMachine and Memora, on LoCoMo (1,540 questions), LongMemEval (500), MEME (1,188) and MemoryAgentBench (2,071), with GPT-4o-mini, GPT-4.1-mini and DeepSeek-V4-Flash as readers; the appendix names GPT-4o-mini as the judge for LoCoMo and LongMemEval and GPT-4o for MEME, and names no single answer judge for MemoryAgentBench. Raw-RAG ranked first in six of seven benchmark and model settings, by margins of 0.13 to 5.53 points. The one loss was MEME with DeepSeek-V4-Flash, where MemMachine scored 44.71 to Raw-RAG's 43.19.

Two details in that paper matter more than the headline. The margins are small, so "RAG wins" really means "the extra machinery bought almost nothing on these tests". And in the ablations on MemoryAgentBench, turning the raw windows into atomic facts cost 19.62 points, while adding summaries or graph expansion cost 5.81 and 5.78. Information thrown away at write time does not come back at read time, and the authors say so directly: "Preserving information during ingestion matters for downstream recovery."

MemoryTax, by Zedong Peng of Shanghai Jiao Tong University (arXiv 2610.29823, project page), points the same way from a different angle. On one LoCoMo conversation (419 turns across 19 sessions, 152 questions, one run per cell, GPT-4o-mini as judge), Mem0 beat plain full context only when GPT-4o-mini was the model, 78% to 74%, and building Mem0's store cost $0.56 against $0.43 for the whole full-context run. With Claude Opus 5.5, the Pi coding agent searching the raw history scored 99% at $6.95 per rollout, against 95% and $17.84 for full context. The study also found that the model writing the memory caps every reader: a Zep graph written by GPT-4o-mini scored 45% when Claude Opus 5.5 read it, while the graph Opus wrote itself scored 87%.

Where the result holds, I agree with it. Inside a history that a strong model can still search or read, keeping the raw record beats summarising it badly, and a July 2026 paper we summarised on our agent skills research hub found the same pattern: on PersonaMem 32k (32 questions across three conversations, gpt-5.4-mini answering at high reasoning effort, graded by exact match with no judge model), a verbatim transcript answered 78.1% correctly against 37.5% for a store the same agent curated (Zhou et al., arXiv 2607.26637). Thirty-two questions is a small sample.

The conditions are where the answer changes, and both papers state them. Raw-RAG's own category tables show memory systems ahead on LongMemEval knowledge-update questions (Memora 97.40 against 91.03) and temporal reasoning (89.50 against 81.45), and on MEME deletion questions with DeepSeek (MemMachine 48 against 29). The authors write that their "evidence is limited to offline conversation on public benchmarks" and that "raw-history approaches also raise storage and privacy concerns." MemoryTax notes that the full history fits inside every model's context window in its test, so the result may not carry over to longer histories. Neither study tests many users sharing one system, a preference corrected across months, or a requirement to erase one person.

Is a bigger context window a replacement for agent memory?

Within one session, often yes, and the October papers make that case better than any vendor page does: on MEME with DeepSeek-V4-Flash, putting the whole history in the prompt beat Raw-RAG by 19.98 points (Bao et al.). If the entire relevant history fits, a strong model reading all of it is the baseline every memory system has to beat.

Across sessions, no, for a mechanical reason: a context window is the input to one call, and an LLM keeps nothing between calls. Whatever survives to next week's conversation survives because your application stored it somewhere and chose to send it again, which makes it memory or retrieval whatever you call it. Our case for why a bigger window is not a memory holds across sessions; inside one short session, the window wins more often than that page suggests.

Quality is the other limit. Chroma's Context Rot report tested 18 models and found that "model performance varies significantly as input length changes, even on simple tasks". On 306 LongMemEval_s prompts (knowledge-update, temporal and multi-session questions, graded by GPT-4.1 as an LLM judge), models from the Claude, GPT, Gemini and Qwen families all did significantly better with focused prompts of about 300 tokens than with full prompts averaging about 113,000 tokens.

Context window vs agent memory: cost per call against cost per write

The trade is cost per call and accuracy at length against write-time cost and the risk of losing detail. Prices make the first half concrete. On Claude Sonnet 5.5, input is $2 per million tokens and a cache read is $0.10, so sending a 200,000-token history costs $0.40 per call uncached and $0.02 when the prefix is cached, against $0.004 for 2,000 tokens of retrieved context. Claude Haiku 5.5 adds a cliff: a prompt over 100,000 tokens pays $0.50 per million input tokens instead of $0.10, so the same history costs five times as much per token once it crosses the line.

Memory moves cost from read time to write time, and the October papers measured that side honestly. On BEAM-1M, about a million tokens of conversation, Raw-RAG finished ingestion in 0.33 hours, MemOS in 3.25 hours and Memora in 26.95 hours. With Claude Opus 5.5 as the writer in MemoryTax, Mem0's build alone cost $25.88. A memory layer earns its write cost only when the reads it saves, or the updates it gets right, are worth more than that.

A workable rule from these numbers: keep the full history in the window while it is short, cacheable and inside one session; switch to retrieval over the raw record when it outgrows the window or the price tier; add memory with update rules when facts about a person change and the agent must use the current one across sessions. The short-term and long-term distinction underneath this is its own explainer.

What are the alternatives to RAG for agents that need long-term memory?

There are six serious options besides a vector index over chunks, and they differ in where the work happens.

Full context with compaction. Keep the history in the prompt and summarise the oldest part as the window fills. Anthropic's memory tool documentation recommends pairing the two: "compaction keeps the active context small without client-side bookkeeping, and memory preserves the information that must survive summarization." The risk is the one the Raw-RAG ablations measured, a summary that drops what a later question needs.

An agent that searches the raw history. Give a capable agent the transcript as files and a shell, and let it grep. MemoryTax's 99% with Pi and Claude Opus 5.5 is this pattern, with the caveats the study states itself: one conversation, and the agents could read photo captions the memory systems never received. The JAZ framework (Li et al., arXiv 2609.26891) takes the same idea further by exposing the whole history to the agent as a variable; on StuLife tasks that needed information from more than 50 tasks earlier, it passed 69.9% against 61.8% for Letta (share of 207 graded tasks scored perfect, mean of three runs; the main text does not say how StuLife grades a task), at $18.3 against $42.1, with GPT-5.4 nano for every method.

File-based memory the model edits. Claude's memory tool lets the model create, read, update and delete files under a /memories path that your application stores, so the model decides what to write down and the storage, isolation and deletion stay in your hands.

Self-editing memory blocks. Letta pins labelled memory blocks to the system prompt and lets the agent rewrite them through memory tools, while older messages stay in a database the agent can still reach through retrieval tools after compaction (Letta memory docs).

Temporal knowledge graphs. Zep stores facts as entities and relationships with validity windows, so a changed fact is invalidated instead of sitting beside the new one (Zep's comparison page). This targets exactly the knowledge-update and temporal categories where Raw-RAG trailed.

An extraction-based memory layer. Services such as Mem0 and Maximem Synap read each conversation, keep discrete statements scoped to a user or tenant, and apply update rules when a new statement conflicts with an old one.

Keyword search deserves a line of its own, because it is RAG without embeddings and it often wins. In our 50,000-document retrieval study, scored by exact document-ID match with no model judge, vector search led the five-dataset average on MRR@10, 0.6320 to 0.5325, but the whole margin came from one code dataset; without it, keyword search won the remaining four, 0.5931 to 0.5614, and indexed the corpus in 2.11 seconds against 161.6 seconds for embedding.

How to choose on your own data

Run the cheapest baseline first. Index the raw history in 256-token windows with overlap, rerank, and measure answer accuracy on questions your users actually ask; Raw-RAG's ablations show the reranker alone is worth 3.92 points, so do not skip it. Then write the questions the benchmarks under-sample: a preference stated and later corrected, "what changed since last month", a request to forget something, and the same question asked for two users on one tenant. If the raw baseline passes those as well, you do not need a memory layer yet, and the October papers say so with data. If it fails them, you know which job a memory layer has to do, and you can test candidates on that job rather than on a leaderboard. Measure retrieval relevance and cross-session recall separately, so you know which half to fix.

Isolation and erasure belong in that test, not after it. A raw-history index has to be partitioned per user and tenant before ranking, and erasing a person means deleting their windows and anything derived from them. A memory layer has the same obligation for its extracted statements, and the derived copies are where deletions get missed.

Where Maximem Synap fits

Synap is an extraction-based memory layer, which is the design the Raw-RAG ablation penalises, so the honest description starts there. Synap reads each conversation as it arrives and keeps self-contained statements with a type, a confidence score, a scope and the time the thing happened, and it does not keep the transcript to search later. What we built against the loss the papers measured: every returned memory carries its source evidence, the verbatim text it came from; a newer statement that is poorer than the one it resembles never replaces it, and a replaced statement is kept, marked historical and linked to its successor (we call this Conscious & Lossless Forgetting). Memory is scoped by default to the account, the tenant, the user and the conversation, a read never sees sideways or below its scope, and per-person erasure is previewed before it runs.

Synap also holds your organisation's own context, from files or connectors for sources such as Google Drive, Notion and Zendesk, at a separate level from user memory, so shared knowledge and per-person state sit in one service without mixing. We have not published Synap results on the Raw-RAG or MemoryTax setups. Both are public and reproducible, so that comparison is the one to ask any memory vendor for, including us.

A memory layer is worth paying for when your agent serves many people over months and the facts about them change; for a single long session with a strong model, the papers say keep the raw record and search it. The business case is narrow and real: customers who stop repeating themselves, and an agent that stops acting on what stopped being true.

Frequently asked questions

Does agent memory replace a vector database?

No. Most memory layers use vector search as one retrieval step, and your shared documents still belong in a document index. Memory adds the write path: what to keep, whose it is, and what it replaced.

Is agent memory just RAG with a new name?

They share the read step and differ on the write step. RAG indexes documents someone else wrote; memory writes, updates and deletes its own records about a user. Our longer answer on memory vs vector RAG covers the vector-store side.

Should I store raw transcripts or extracted facts?

Keep the raw record wherever privacy rules allow it, because the October 2026 papers show extraction alone loses accuracy on recall questions. Extracted, scoped statements earn their place for changed facts, cross-session use and erasure, and they work best with a link back to the source text.

Does plain RAG over chat history beat memory systems?

On LoCoMo, LongMemEval, MEME and MemoryAgentBench, it ranked first in six of seven settings by 0.13 to 5.53 points (arXiv 2610.30755). Memory systems led on knowledge updates, temporal reasoning and deletion, and the authors limit their claim to offline public benchmarks.

Sources

All retrieved on 27 October 2026.

• Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought, Bao et al., arXiv 2610.30755, 22 October 2026 and code
• MemoryTax: The Bitter Lesson of Agent Memory, Zedong Peng, arXiv 2610.29823, 11 October 2026, project page and code
• Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity, Li et al., arXiv 2609.26891
• Context Rot, Hong, Troynikov and Huber, Chroma, 14 July 2025
• Filesystem-Based Memory for LLM Agents: Organization, Evolution, and Sustainability, Zhou et al., arXiv 2607.26637, 29 July 2026
• Claude API pricing and memory tool documentation
• Letta documentation and Letta memory guide
• Agent Memory vs. RAG, Zep, 31 May 2026
• Build vs Buy Agent Memory, Maximem and Agent skills research hub, Maximem