# Which LLM Should Write Your AI Agent's Memory? How to Choose and Test a Memory Extraction Model

> Which LLM to use for memory extraction: in Mem0's October 2026 test of 17 writer models on LoCoMo, the writer alone moved accuracy from 84.1% to 93.5% and cost from $0.68 to $158.55 per 1,000 writes, yet the top six sat within 0.9 points. How to test writer models on your own transcripts, the failure modes to measure (rules written as memory, over-extraction, missed updates), and where a small judge model fits with a type-aware write policy.

_Gaurav Dadhich · 2026-11-04_

\# Which LLM Should Write Your AI Agent's Memory? How to Choose and Test a Memory Extraction Model \*Published 4 November 2026 by Gaurav Dadhich. Every figure below was read at its primary source on 4 November 2026, every vendor figure is labelled as vendor-reported, and the evaluation code was run the same day.\* The model that writes your agent's memory sets the ceiling on what the agent can ever remember, because retrieval can only return what extraction wrote down. In the largest published comparison so far, \[Mem0's bring-your-own-key test\](https://mem0.ai/blog/bring-your-own-key-accuracy-cost-latency-by-model) of 17 writer models on LoCoMo, changing nothing but the extraction model moved answer accuracy from 84.1% to 93.5% and the cost of writing from $0.68 to $158.55 per 1,000 add requests (vendor-reported, 20 October 2026). The same table also shows why "use the best model" is the wrong answer: the six most accurate writers sit within 0.9 points of each other, and the most expensive of them costs about 45 times the cheapest. So the practical answer to "which LLM for memory extraction" is a short list plus a test. Start with a fast, mid-priced model that scored near the top of a public writer benchmark, then run a small evaluation on your own transcripts that measures what public benchmarks do not: whether the model writes generalised learnings or one-off rules, whether it catches updates, and how much it writes that nobody needed. Put a cheap judge step and a type-aware write policy behind it, and the writer model stops being the only thing standing between a conversation and a polluted memory. ## What the published numbers say about writer models Mem0's test held everything constant except the model that "reads each conversation and decides what to remember", ran LoCoMo's 10 long conversations and 1,540 questions twice per model, and reported the average. Cost is Mem0's estimate of provider billing per 1,000 add requests of about 8 messages each, and latency is the p95 time from an add request until its memories are searchable. The post does not name the answer model or the judge model, and it does not name the model behind Mem0's own default, which scored 92.5%. | Writer model (Mem0 BYOK, LoCoMo) | Weights | Accuracy | $ per 1,000 adds | p95 to searchable | |---|---|---|---|---| | Claude Opus 5.5 | Closed | 93.5% | $158.55 | 18.1 s | | Claude Sonnet 5.5 | Closed | 93.4% | $78.44 | 10.4 s | | GLM-5.3 Flash | Open | 93.0% | $7.25 | 86.5 s | | DeepSeek V4.1 Flash | Open | 92.9% | $5.34 | 18.5 s | | GPT-6 Luna | Closed | 92.6% | $3.55 | 15.2 s | | GPT-5 mini | Closed | 92.6% | $17.82 | 37.7 s | | Claude Haiku 5.5 | Closed | 92.3% | $4.93 | 17.7 s | | Gemini 3.5 Flash-Lite | Closed | 89.9% | $7.08 | 6.1 s | | Amazon Nova Micro | Closed | 88.0% | $0.86 | 10.2 s | | Mistral Nemo | Open | 84.1% | $0.68 | 9.8 s | \*Ten of the 17 rows, all vendor-reported by Mem0 on 20 October 2026 (updated 23 October). Mem0 notes that GLM-5.3 Flash, GPT-5 mini and Gemini 3.8 Flash are slower because they reason before writing.\* Three readings of this table matter for a build decision. The floor is real: a small, old model loses close to ten points against the top of the table, which on Mem0's own framing means the least accurate model "misses 2.4 times as many questions" as the most accurate one. The top is flat, which means that above roughly 92.5% on this benchmark you are paying for fractions of a point. And latency does not track price: Sonnet 5.5 was the fastest of all the models above Mem0's default, while GLM-5.3 Flash, the third most accurate, took 86.5 seconds at p95, which is fine for overnight memory and wrong for an agent that has to use a fact two turns after hearing it. The other data point came from Letta's Charles Packer on 3 November. He \[wrote\](https://x.com/charlespacker/status/2117705131756249520) that "GPT models are significantly worse than the open weights frontier models (Kimi, GLM) at memory generation", that "OpenAI's models have a bad tendency to create 'rules' as memory, which are the opposite of generalized learnings", and that "Opus 5.5 leads here - still nowhere near the ceiling". The post does not say which test produced the observation. On the surface that contradicts Mem0's table, where GPT-6 Luna finishes fifth of 17 and 0.9 points behind Opus 5.5. Both can be true, and the reason is the most useful thing to understand about choosing a writer. LoCoMo grades whether a question about a past conversation can be answered from what was written, which rewards capturing facts. Packer is describing agents that learn from their own experience, where the memory is supposed to be a lesson that transfers to the next situation. A model can capture every fact and still write lessons badly, and a fact-recall benchmark will never show it. ## The failure modes a writer-model test has to catch \*\*Rules written as memory.\*\* A rule is a narrow imperative tied to one case: "when the user asks about invoice INV-2231, check the March export first". A learning is the reusable version: "this team reconciles invoices from the monthly export, so billing questions start there". A memory store full of rules looks busy and helps almost never, because the exact trigger rarely recurs. The \[UMEM paper\](https://arxiv.org/abs/2602.10652) (February 2026) names the same failure from the research side: methods that treat extraction as a static process end up with agents that "accumulate instance-specific noise rather than robust memories". Its fix, training the extractor with a reward measured across clusters of related queries, is the right mental model for your test too; a good memory is one that helps several future questions, not the one it came from. \*\*Over-extraction.\*\* The writer stores greetings, acknowledgements, restatements and session trivia. Each write costs a model call, adds a row that retrieval has to rank past, and dilutes the memories that matter. Rudraj Mehta of Mem0 \[estimates\](https://hackernoon.com/persistent-ai-agent-memory-is-a-write-path-cost-problem) that extraction is roughly 60% to 75% of memory spend in an illustrative breakdown and recommends gating trivial turns such as "ok thanks" so they never reach a model. \*\*Missed and stale updates.\*\* The user says "I moved to Madrid" three weeks after saying they live in London. A weak writer either misses the change or writes "lives in Madrid" next to "lives in London" with nothing linking them. \[HaluMem\](https://arxiv.org/abs/2511.03506) found that memory systems "tend to generate and accumulate hallucinations during the extraction and updating stages" and that those errors carry into question answering, which is why update handling deserves its own number rather than being folded into overall accuracy. \*\*Unsupported inferences.\*\* The writer turns "I do not eat meat" into "vegan for ethical reasons". This is the failure that \[our memory write policy post\](https://www.maximem.ai/blog/agent-memory-write-policy) treats at length, where one workshop study found value and belief claims were supported by their source 77.9% of the time against 96.2% for every other type. \*\*Under-extraction\*\* is the quiet failure. Small models skip a preference stated in passing or a constraint buried in a long message, and nothing errors; the agent simply asks again next week. Purpose-trained extractors go after this failure directly: \[MemReader\](https://arxiv.org/abs/2604.07877) (April 2026) pairs a 0.6-billion-parameter passive extractor with a 4-billion-parameter active one that weighs a message's value, ambiguity and completeness before deciding to write, defer, look something up or discard it, and its authors report gains over extraction baselines on LoCoMo, LongMemEval and HaluMem. ## How to test a writer model on your own transcripts A public benchmark tells you which models are worth testing. Your own conversations tell you which one to ship, and the evaluation is small enough to run in a day. 1. \*\*Pick the conversations.\*\* Take a sample of real transcripts that covers your traffic, and deliberately include the hard cases: conversations where the user changes a fact, long messages with a buried constraint, sessions where the agent learned something by failing, and sessions with nothing worth remembering. A few dozen conversations is enough to separate models that are far apart; you need more to separate models within a point of each other. 2. \*\*Write the gold memories by hand.\*\* For each conversation, a person lists what a good memory system should hold afterwards, with a type for each (fact, preference, instruction, episode, learning) and a flag on any item that replaces something said earlier. Include the expected nothing: conversations where the right output is no write at all. 3. \*\*Run every candidate writer on the same input\*\* with the same extraction prompt and schema, and log cost and time to searchable per write. 4. \*\*Have a reviewer label each written memory.\*\* Record which gold item it matches, if any; whether it is supported by the source; whether it is a one-case rule or a reusable learning; and whether it states a value the conversation later replaced. 5. \*\*Score and compare.\*\* The code below turns those labels into the numbers that matter. \`\`\`python from dataclasses import dataclass from collections import defaultdict @dataclass class Gold: conv\_id: str gold\_id: str type: str # "fact", "preference", "instruction", "episode", "learning" is\_update: bool # True if it replaces something said earlier in the conversation @dataclass class Written: conv\_id: str model: str text: str matches: str | None # gold\_id a reviewer matched it to, or None supported: bool # a reviewer found it in the source is\_rule: bool # a one-case imperative ("always do X when Y") rather than a reusable learning stale: bool # it states a value the conversation later replaced cost\_usd: float seconds\_to\_searchable: float def score(gold: list\[Gold\], written: list\[Written\]) -> dict\[str, dict\]: gold\_ids = {g.gold\_id for g in gold} update\_ids = {g.gold\_id for g in gold if g.is\_update} by\_model = defaultdict(list) for w in written: by\_model\[w.model\].append(w) n\_convs = len({g.conv\_id for g in gold}) report = {} for model, ws in by\_model.items(): hit = {w.matches for w in ws if w.matches in gold\_ids} lat = sorted(w.seconds\_to\_searchable for w in ws) report\[model\] = { "recall": len(hit) / len(gold\_ids), "update\_recall": len(hit & update\_ids) / len(update\_ids) if update\_ids else None, "unsupported\_rate": sum(not w.supported for w in ws) / len(ws), "rule\_rate": sum(w.is\_rule for w in ws) / len(ws), "stale\_rate": sum(w.stale for w in ws) / len(ws), "unmatched\_rate": sum(w.matches is None for w in ws) / len(ws), "writes\_per\_conversation": len(ws) / n\_convs, "cost\_per\_1k\_conversations": 1000 \* sum(w.cost\_usd for w in ws) / n\_convs, "p95\_seconds": lat\[min(len(lat) - 1, int(0.95 \* len(lat)))\], } return report \`\`\` It uses the standard library only and runs on Python 3.10 or later; a small test with two models, four gold memories and seven written ones passed on Python 3.13 on 4 November 2026. Read the outputs together, and do not rank on any single one. \*\*Recall\*\* and \*\*update recall\*\* tell you what the writer misses. \*\*Unsupported rate\*\* is the hallucination number. \*\*Rule rate\*\* is Packer's failure, measured on your data instead of taken on trust. \*\*Unmatched rate\*\* with a high \*\*writes per conversation\*\* is over-extraction: memories that are true but that nobody listed as worth keeping. Cost and p95 time decide between the models that survive. Two practical notes. Label blind, with the model name hidden from the reviewer, because people grade the model they expect to win more kindly. And if you use an LLM to do the labelling at scale, check it against a human-labelled slice first; \[a judge you have not measured\](https://www.maximem.ai/blog/ai-agent-evals-the-judge-is-the-part-nobody-measures) moves the scores as much as the writer does. ## Cost and latency per write are now a design axis A memory write runs on every turn or every session, so the writer's price is multiplied by your whole traffic. Mem0's own monthly examples at 10,000 add requests make the spread concrete: $8.60 on Amazon Nova Micro, $53 on DeepSeek V4.1 Flash, $784 on Claude Sonnet 5.5 and $1,586 on Claude Opus 5.5 (vendor-reported). At that volume the choice is a rounding error; at a hundred times that volume it is a line item someone will ask about. Latency matters less than it looks, as long as writes are asynchronous. Mem0 points out that a slow writer does not block the app, but a new memory is not searchable until it is written. The question to ask is how soon after a user says something the agent needs to use it. If the answer is "next session", an 86-second p95 is acceptable. If the answer is "two turns from now", handle that information in the turn, which is the split between hot-path and background writes covered in the \[write policy post\](https://www.maximem.ai/blog/agent-memory-write-policy). Vendor defaults point the same way. SurrealDB's \[models-per-stage documentation\](https://surrealdb.com/docs/agent-memory/tuning/models-per-stage) recommends a "fast, inexpensive model" for extraction, defaults to \`gemini-3.5-flash-lite\` for it and keeps a larger model for background consolidation, and Mehta argues that extraction and reconciliation "rarely need a frontier model". The pattern that holds up is a mid-tier writer on the hot path, a stronger model for the less frequent jobs that merge and rewrite memories, and a gate that keeps trivial turns away from both. ## Where a small judge model fits The newest change is a second model in the write path whose job is to decide, not to write. On 3 November Vectorize launched \[Hindsight Models\](https://hindsight.vectorize.io/blog/2026/11/03/hindsight-models) on the principle "The LLM writes, the decision model judges". The decision model, \`hs-decide-1-flash\`, has 0.8 billion parameters, is built on Qwen3.5-0.8B under an Apache 2.0 licence, and returns a choice with a probability and generates no text. In consolidation it sees each new fact before any LLM call: facts that restate an existing observation are attached as evidence, chatter is skipped, and only facts that change, add to or are uncertain about an observation reach the LLM, which still writes every word of memory text. Vectorize reports, on its own evals run on Hindsight 1.2.0 and graded by \`gemini-3.1-flash-lite\`, 2.5 times lower cost, 1.9 times less model-call time and a 2.2-point higher score when \`gemini-3.8-flash\` is paired with the decision model across consolidation, reflect and knowledge pages, and a median of 167 ms per decision request against 1 to 7 seconds for the LLM call it replaces. The same post records the trade-offs: reflect is where quality dips most, and for the two cheapest LLMs the decision model's own tokens made the total cost higher. Time is model-call time, not end-to-end latency, and the judge model is one of the seven LLMs being graded. One detail matters for choosing a writer: Hindsight's retain step, where new memories are first extracted, does not use the decision model. The judge decides what happens to a memory once the writer has produced it; it does not replace the writer. That is the right way to think about a judge in your own pipeline. It is cheap enough to run on every candidate, so give it the yes-or-no questions that are expensive to ask a writer: is this new or a restatement, does it change an existing memory, is this a reusable learning or a one-case rule. It does not get the final say on anything a deterministic rule can decide better. The order I would use: 1. \*\*Writer\*\* extracts candidates with a type, an evidence span and a stated-or-inferred flag. 2. \*\*Deterministic gate\*\* rejects never-store patterns, writes from origins that may not write that type, and candidates whose evidence span is not in the source. No model call, so it runs first and cannot be argued with. 3. \*\*Judge\*\* labels what survives: duplicate, update, new, rule or learning, with a probability. 4. \*\*Type-aware policy\*\* applies a different threshold per type to the judge's probability, holds values and instructions for confirmation, and turns "update" into a supersede that keeps the old memory as history. 5. \*\*Write\*\*, logging the judge's label and probability beside the memory so you can recalibrate later. The per-type thresholds are where the judge earns its place. A probability of 0.7 that a candidate is a new fact is enough to write it; the same 0.7 on an inferred value or an instruction is not, and the \[write policy post\](https://www.maximem.ai/blog/agent-memory-write-policy) sets out the bar for each type. Rules are the one label to handle differently: do not discard them, route them to \[consolidation\](https://www.maximem.ai/blog/agent-memory-consolidation), where a stronger model can see several rules at once and write the learning they share. ## Which writer to pick, by situation | Situation | Start with | Why | |---|---|---| | Fact and preference memory for a chat or support agent, moderate volume | A mid-priced model near the top of Mem0's table, such as DeepSeek V4.1 Flash, GPT-6 Luna or Claude Haiku 5.5 | Within about a point of the top at a few dollars per 1,000 adds (vendor-reported) | | Facts must be usable within the same session | A fast writer (Sonnet 5.5 had the lowest p95 among models above Mem0's default) or an explicit hot-path write | Time to searchable matters more than the last point of accuracy | | Agents that learn procedures from their own runs | Test open-weight frontier models (Packer names Kimi and GLM) and Opus 5.5, and measure rule rate | Fact-recall benchmarks do not grade whether lessons generalise | | Very high volume, tight budget | A sub-$1 writer such as Amazon Nova Micro, plus a judge and a strict gate | Cheap writers over-extract and miss more; the gate and judge recover precision | | Regulated data or self-hosting | An open-weight writer you can run yourself, plus a self-hosted judge | Hindsight's decision model ships open weights; GLM and DeepSeek have open weights in Mem0's table | Every row is a starting shortlist for your own test, not a verdict. A model that wins on someone else's conversations can still lose on yours, and the gap between the top writers on LoCoMo is smaller than the gap you will see between a good and a bad extraction prompt. The fuller list of what goes wrong once memories are flowing is in \[what breaks when you build your own agent memory\](https://www.maximem.ai/blog/what-breaks-when-you-build-agent-memory). ## How Maximem Synap handles the write step \[Maximem Synap\](https://www.maximem.ai/synap) runs extraction in the background, so a write returns immediately and never blocks the agent; extraction takes several seconds, so a memory written now is not retrievable a second later. What gets extracted, how aggressively, and into which categories is set per agent by a memory architecture that Synap generates from a description of what the agent does. That architecture is authored by Anthropic's Claude Opus, checked by two further frontier-model judges, never applied until a person approves it, and versioned so it can be rolled back. A healthcare agent and a food delivery agent get different extraction on purpose, which is the type-aware policy above applied before the first conversation rather than tuned after the fortieth. The writer model decides the ceiling, the judge and the policy decide how close to it you stay, and getting both right is the difference between an agent that gets more useful with every conversation and one whose memory slowly fills with rules nobody will trigger again. ## Frequently asked questions ### Which LLM is best for memory extraction? On the largest public comparison, Mem0's October 2026 test of 17 writer models on LoCoMo, Claude Opus 5.5 scored highest at 93.5%, with Claude Sonnet 5.5 at 93.4% for about half the price and DeepSeek V4.1 Flash, GLM-5.3 Flash and GPT-6 Luna within a point of the top for under $10 per 1,000 add requests (vendor-reported). The best model for you is the one that wins a small evaluation on your own transcripts, because LoCoMo measures fact recall and not whether memories generalise. ### Do I need a frontier model to write agent memory? Usually not for facts and preferences. In Mem0's test, several models costing $3.55 to $7.25 per 1,000 adds scored within 0.9 points of the most expensive one. Frontier models matter more when the agent is learning procedures from its own experience, where Letta's Charles Packer reports that Opus 5.5 leads and that GPT models tend to write rules instead of learnings. ### What does "rules as memory" mean? A rule is a memory tied to one case, such as "when the user asks about invoice INV-2231, check the March export first". A learning is the reusable version, "billing questions for this team start from the monthly export". Rules rarely trigger again, so a memory store that fills with them grows without getting more useful. Measure it by having a reviewer flag rules in a sample of written memories for each candidate model. ### How do I evaluate a memory extraction model? Hand-label the memories a good system should hold for a sample of real conversations, run each candidate writer on the same input, label what each one wrote, and compare recall, update recall, unsupported rate, rule rate, writes per conversation, cost per 1,000 conversations and p95 time to searchable. Include conversations where the right answer is to write nothing. ### What does a judge model do in a memory pipeline? It makes cheap yes-or-no decisions about a candidate memory, such as whether it restates something already stored, changes an existing memory or is chatter, so the expensive writer is only called when there is something new to write. Vectorize's 0.8-billion-parameter \`hs-decide-1-flash\` is one example, released with open weights on 3 November 2026. It should sit after deterministic checks and before a type-aware write policy, not replace either. ### Does a slow extraction model slow down my agent? Not if writes are asynchronous, which is how most memory layers run them. The cost is freshness: a memory is not searchable until it is written, which in Mem0's test took anywhere from 6.1 to 86.5 seconds at p95 depending on the model. Anything the agent must use within the same session belongs in a hot-path write. ## Sources All sources were read on 4 November 2026. - \[Bring Your Own Key: Accuracy, Cost & Latency by Model, Mem0 (Deshraj Yadav), 20 October 2026, updated 23 October\](https://mem0.ai/blog/bring-your-own-key-accuracy-cost-latency-by-model) - \[Introducing Hindsight Models, Vectorize (Boschi, Neeser, C. Bartholomew, B. Bartholomew), 3 November 2026\](https://hindsight.vectorize.io/blog/2026/11/03/hindsight-models) - \[Vectorize launch post on X, 3 November 2026\](https://x.com/Vectorizeio/status/2117647994048704588) - \[Charles Packer on X, 3 November 2026\](https://x.com/charlespacker/status/2117705131756249520) - \[Persistent AI agent memory is a write-path cost problem, Rudraj Mehta, HackerNoon, 4 August 2026\](https://hackernoon.com/persistent-ai-agent-memory-is-a-write-path-cost-problem) - \[Models per stage, SurrealDB agent memory documentation\](https://surrealdb.com/docs/agent-memory/tuning/models-per-stage) - \[UMEM: Unified Memory Extraction and Management Framework for Generalizable Memory, arXiv 2602.10652\](https://arxiv.org/abs/2602.10652) - \[HaluMem: Evaluating Hallucinations in Memory Systems of Agents, arXiv 2511.03506\](https://arxiv.org/abs/2511.03506) - \[MemReader: From Passive to Active Extraction for Long-Term Agent Memory, arXiv 2604.07877\](https://arxiv.org/abs/2604.07877)

---

Source: [https://www.maximem.ai/blog/which-model-should-write-agent-memory](https://www.maximem.ai/blog/which-model-should-write-agent-memory)