Benchmarks
AI Memory Benchmarks
Published Updated
A benchmark score with no configuration attached is not a score — the same system moves ten to thirty points on the answer model and the judge alone. So every figure here carries the conditions that produced it, weakest categories first, on a harness that is open source and runs against our competitors as readily as against us.
Both accuracy figures were produced with gpt-5-mini as the answer model and as the judge — the cheap tier, the one you would actually run an agent loop on — on the official public dataset releases, with no custom subsets and no relabeling. Both come from repeated runs, and the figure published is the score the harness returns most often — not the best one we recorded. Competitors reach lower numbers on flagship models; what the answer model is worth is measured below, using their figures.
01LongMemEval
LongMemEval: 92.0% (460 / 500)
Human-curated questions over long, multi-session chat histories. Wu et al., ICLR 2025.
- Answer
- gpt-5-mini
- Judge
- gpt-5-mini, binary
- Scope
- Full 500-question set
- Dataset
- LongMemEval_S, official release
- Runs
- Multiple runs, most-reproduced result
| Category | Accuracy | Correct / n |
|---|---|---|
| Multi-session | 75.2% | 100 / 133 |
| Single-session assistant n=56 | 87.5% | 49 / 56 |
| Single-session user | 100.0% | 70 / 70 |
| Single-session preference n=30 | 100.0% | 30 / 30 |
| Knowledge update | 100.0% | 78 / 78 |
| Temporal reasoning | 100.0% | 133 / 133 |
The hairline across each bar marks the 92% overall figure.
02LoCoMo
LoCoMo: 93.2%
Very long-term conversations, roughly 300 turns each. Maharana et al., ACL 2024.
- Answer
- gpt-5-mini
- Judge
- gpt-5-mini, binary
- Scope
- Categories 1–4, 1,540 questions
- Dataset
- locomo10, official release
- Runs
- Multiple runs, most-reproduced result
| Category | Accuracy | Correct / n |
|---|---|---|
| Single-hop | 88.8% | n = 282 |
| Temporal | 90.8% | n = 96 |
| Open-domain | 93.4% | n = 841 |
| Multi-hop | 97.3% | n = 321 |
The hairline across each bar marks the 93.2% overall figure. Correct counts are published for LongMemEval and not yet for LoCoMo; n is the official category distribution.
03Reproduction
What happens when you re-run the published numbers
We ingested the same benchmark datasets into each vendor’s paid hosted product, exactly as a customer would, then ran the questions through a standardized answerer and judge that we control. That measures the memory layer rather than the vendor’s evaluation stack.
Read this as a set of gaps, not as a ranking. Each row compares one system against its own published claim. These reproduction runs used gpt-5 with five-seed averaging, which is a different configuration from the Synap results above — so the gaps are comparable across rows and the absolute scores are not.
Every row below is LongMemEval. There is no LoCoMo equivalent yet: the only LoCoMo reproduction we have started is against Mem0’s post-April-14 product, and that run is still going. When it lands it gets its own table here rather than being folded into this one, because a LoCoMo figure and a LongMemEval figure are not the same measurement.
LongMemEval · claimed vs observed
| System | Our harness | Vendor published | Gap | |
|---|---|---|---|---|
| Maximem Synap | 92.0% | 92.0% | 0.0 | |
| Mem0 | 73.8% | 93.4% | −19.6 | |
| Supermemory | 71.3% | 85.2% | −13.9 | |
| Zep | Not run | 71.2% | — | |
| Evermind | Not run | 83.0% | — | |
| Letta | Not run | Not published | — | |
| Cognee | Not run | Not published | — |
- Maximem Synap
- Our published number is our harness number. There is nothing between the memory layer and the figure, which is the only reason we are entitled to run this chart at all. Results repo.
- Mem0
- Measured against the post-April-14 hosted product. The same evaluation against the pre-April-14 product returned 57.5%, so the memory layer genuinely improved by 16.3 points across that window. The remaining gap is not the memory layer. Mem0, April 2026.
- Supermemory
- The published figure is their strongest configuration, generated with Gemini-3 Pro. The same system on the same benchmark scores 81.6% with gpt-4o, which is the clearest demonstration on this page that the answer model moves a memory score on its own. supermemory.ai/research.
- Zep
- Not re-run on the configuration behind the rows above, so there is no gap to state. The figure cited is from Zep’s own paper; a separate third-party report puts Zep at 63.8%. arXiv:2501.13956.
- Evermind
- Self-reported, with no independent verification we can cite. Not ingested on our harness.
- Letta
- No LongMemEval figure published, and not ingested on our harness.
- Cognee
- No LongMemEval figure published, and not ingested on our harness.
The part of the Mem0 gap that is real
Pre-14 April product
57.5%
Post-14 April product
73.8%
+16.3
points, earned
Mem0’s memory layer genuinely improved by 16.3 points across their April 2026 product update, measured on our harness before and after. That is engineering progress and they have earned the right to claim it. The remaining distance to their published 93.4% is not attributable to the memory system. It sits in the answer and judge prompts that ship with the published claim:
- Fourteen dataset-specific equivalence rules in the answer prompt, mapping one-to-one onto public LongMemEval question ids.
- A hidden chain-of-thought block where those rules are applied before the visible answer is emitted. The judge only ever sees the cleaned answer.
- An explicit "lean toward yes" instruction in the judge prompt, paired with a five-step gauntlet to clear before marking anything wrong — and no symmetric gauntlet before marking anything correct.
- A one-directional gold-override clause that can promote a wrong prediction to correct, but cannot demote a correct prediction to wrong.
Every one of those is pinned to a specific commit and line, with SHA-256 hashes on mirrored copies and Wayback Machine captures for independent timestamps. The full evidence chain. Mem0 has been invited to respond publicly, and the harness accepts pull requests from any vendor who wants their configuration tested differently.
04Configuration
What the answer model is actually worth
The standing objection to any memory benchmark is that the answer model does the work: put a large enough model behind a mediocre memory layer and the score comes up on its own. The objection is correct, and it is measurable — because Supermemory published the measurement against themselves. Three LongMemEval scores, one memory layer, three answer models, nothing else changed.
| System | Answer model | Tier | LongMemEval |
|---|---|---|---|
| Supermemory | gpt-4o | Frontier | 81.6% |
| Supermemory | gpt-5 | Frontier | 84.6% |
| Supermemory | Gemini-3 Pro | Frontier | 85.2% |
| Maximem Synap | gpt-5-mini | Mini-tier | 92.0% |
Rail spans 75–95%, so the differences the figure is about stay readable. The hairline marks the best competitor configuration at 85.2%.
+3.6
points — everything three answer-model generations bought Supermemory, from gpt-4o through gpt-5 to Gemini-3 Pro, with the memory layer held constant.
+6.8
points from their best configuration to ours, produced on a mini-tier answer model — below the bottom rung of their own ladder.
Read the two spans against each other. Two years of frontier progress in the answer model was worth 3.6 points to Supermemory. The distance from their best configuration to ours is 6.8 — close to twice as far — and we cover it going down a tier rather than up one. Whatever produces that gap, it is not the answer model, and the reason we can say so is that the numbers bounding it are theirs.
Which matters well past the leaderboard. The answer model is invoked on every turn of an agent loop, so its tier is a standing cost rather than a benchmark-day detail. A memory layer that holds its accuracy on the cheap tier is a different economic proposition from one that needs a flagship model to reach a lower number, and the gap compounds with every conversation your product runs.
Every published score above 85% on this page other than ours was either produced on a frontier answer model or did not survive an independent re-run. Mem0’s 93.4% is the single published figure above ours, produced on an answer model they have never named, and it is the one that returned 73.8% in the section above. That is the whole reason this page runs the reproduction table before this one: a cost-per-point argument is only worth making over scores that survive being re-run.
Each system’s own best published LongMemEval score
| System | Best published | Answer model | Tier | On our harness |
|---|---|---|---|---|
| Mem0 | 93.4% | Not stated | Not stated | 73.8% |
| Maximem Synap | 92.0% | gpt-5-mini | Mini-tier | 92.0% |
| Supermemory | 85.2% | Gemini-3 Pro | Frontier | 71.3% |
| Evermind | 83.0% | Not stated | Not stated | Not run |
| Zep | 71.2% | gpt-4o | Frontier | Not run |
| Letta | Not published | — | — | Not run |
| Cognee | Not published | — | — | Not run |
Ordered by the claim, not by our assessment of it — the highest figure in this table is the one that fell 19.6 points under reproduction, two sections above. Tier is the vendor’s own positioning of the model — the cheap workhorse tier against the flagship — not a judgement of it, and it is stated only where the vendor named the model behind the figure. Letta and Cognee have published no LongMemEval figure at all. “On our harness” is the reproduction run from the section above, carried down unchanged. Sources: mem0ai/memory-benchmarks, supermemory.ai/research, arXiv:2501.13956, and our results repository.
Why vendor scores differ
The answer model has the largest published effect and it is not the only variable. These are the rest — the settings that decide whether two memory benchmark numbers can be compared at all.
| Variable | Maximem Synap | Mem0 | Zep | Letta | Supermemory | Why it matters |
|---|---|---|---|---|---|---|
| Answer LLM | gpt-5-mini | gpt-4o-mini | gpt-4o-mini / gpt-4o | gpt-4o-mini | gpt-4o / gpt-5 / Gemini-3 Pro | High. Newer and larger models score materially higher on identical data — measured above at 3.6 points across three generations. |
| Judge LLM | gpt-5-mini | gpt-4o-mini | gpt-4o-mini / gpt-4o | gpt-4o-mini | gpt-4o | High. Judges differ in false-positive and false-negative rates. |
| Embedder | Not published | text-embedding-3-small | BGE-m3 | File search | Not disclosed | Medium. Drives retrieval quality. |
| Reranker | Not published | None / graph | BGE-m3 reranker | n/a | Not disclosed | Medium. Second-stage precision. |
| Ingestion granularity | Not published | Message by message | Full conversation via API | File based | Session by session | Medium. Changes how memories are structured and time-grounded. |
| Adversarial (LoCoMo Cat 5) | Excluded | Excluded | Excluded | n/a | n/a | High. Inclusion shifts an overall score by more than ten points. |
| Runs / variance | Multiple runs; most-reproduced result | Single run | 10 runs, mean ± std | Single run | Not reported | Medium. A single run carries real variance, and a best-of-N figure carries selection on top of it. |
| Timestamp handling | Not published | Embedded in text | Bitemporal (event + ingest time) | n/a | Dual-layer (doc + event date) | High, for temporal questions. |
Compiled from each vendor’s published papers, blogs and evaluation code. Every cell is a claim you can check against the source it came from; where a vendor has not stated a value, the cell says so rather than guessing. Four cells in our own column read “Not published”, and Zep reports ten runs with a standard deviation where we report one — both stay visible, because a page arguing that every figure needs its configuration cited does not get to quietly omit its own.
05Limits
Where we lose
Multi-session is our worst category at 75.2%, and it is the hardest category for every vendor on every published leaderboard. It asks the system to synthesize evidence from separate conversations rather than pull from one recent thread, which means linking the same entity across sessions when its surface form changes. That is part retrieval, part graph reasoning and part ontology engineering, and nobody has solved it.
Single-session assistant sits at 87.5%, seven questions short of the four categories at 100%. On LoCoMo, single-hop at 88.8% is our lowest category and sits below multi-hop at 97.3%, which is counter-intuitive and which we do not yet have a published explanation for.
The scope limits matter as much as the scores. Both figures come from repeated runs and we publish the score the harness returns most often, not the best one we recorded — but we have published neither the run count nor the spread, so until we do you are taking the most-reproduced claim on trust. Both use one answer model. Both benchmarks are synthetic, which means neither reproduces the way a production workload accumulates contradictions over months — the case memory systems actually have to survive. And a LoCoMo number from this harness is only comparable against another Category 1 to 4 number.
06Reproduce
Run this yourself
A number you cannot inspect is a claim, not proof. Everything above is produced by an open harness against the official public dataset releases, with no custom subsets and no relabeling.
- 01Clone the harness: maximem-ai/memory_and_context_eval_harness.
- 02Download the official datasets, LongMemEval_S and locomo10.
- 03Ingest into your provider, then run the answer and judge pass with gpt-5-mini.
- 04Score LongMemEval across all 500 questions; score LoCoMo across Categories 1 to 4.
git clone https://github.com/maximem-ai/memory_and_context_eval_harness cd memory_and_context_eval_harness && pip install -e . python scripts/download_datasets.py --variant s python -m runner.server
The harness ships adapters for Synap, Mem0, Zep and Supermemory, and a provider interface for anything else. Every deviation from the published benchmark protocols is recorded in DEVIATIONS.md, and the full methodology, including the cross-vendor configuration table above, is in METHODOLOGY.md. Results are published under CC BY 4.0.
Questions about benchmark numbers
LongMemEval and LoCoMo are the two that get cited. LongMemEval tests recall and reasoning across long multi-session histories; LoCoMo tests conversational memory over very long dialogue. Both are useful and both are synthetic, which is the limitation, because neither reproduces the way a real workload accumulates contradictions over months.
Usually the adversarial category. LoCoMo Category 5 questions are intentionally unanswerable, so they measure abstention rather than memory retrieval, and including or excluding them moves an overall score by more than ten points. We exclude it, matching the original paper, Mem0 and Zep. A LoCoMo number should only ever be compared against another Category 1 to 4 number.
Trust them exactly as far as the methodology is published and the run is reproducible. A score with a linked harness, a stated dataset version and a named answer model is a claim you can check. A score in a marketing table is a claim you cannot. Ask for the harness before you ask about the number — ours is open source and runs against our competitors as well as against us.
Because a larger answer model can carry part of the memory system’s job. Supermemory published the size of that effect against themselves: the same memory layer scores 81.6% on LongMemEval with gpt-4o, 84.6% with gpt-5 and 85.2% with Gemini-3 Pro, so three answer-model generations were worth 3.6 points. Maximem Synap scores 92.0% on gpt-5-mini, a mini-tier model below the bottom rung of that ladder — 6.8 points above their best configuration, which is close to twice what the answer model was worth to them. That is what isolates the contribution of the memory layer, and it also makes it a cost argument: the answer model is invoked on every turn of an agent loop.
Retrieval accuracy asks whether the system returned the correct memory. Task accuracy asks whether the agent then did the right thing. A system can score well on the first and still fail the second, which happens whenever the right memory is retrieved and then buried in a context window large enough that the model reads past it.
Take fifty real conversations from your product, split each one so that facts established early are needed late, and score whether the agent completed the task rather than whether it retrieved the note. Run the same set against your current setup and against any candidate system. The absolute numbers matter less than the delta on your own data.