Benchmarks
AI Memory Benchmarks
Published Updated
A benchmark score with no configuration attached is not a score — the same system moves ten to thirty points on the answer model and the judge alone. So every figure here carries the conditions that produced it, weakest categories first, on a harness that is open source and runs against our competitors as readily as against us.
The LongMemEval figure was produced with Gemini 3.8 Flash as the answer model and gpt-4o as the judge, single run. The LoCoMo figure is an older run with gpt-5-mini as the answer model and as the judge. Both use the official public dataset releases, with no custom subsets and no relabeling. Competitors reach lower numbers, several on flagship models; what the answer model is worth is measured below, using their figures.
01LongMemEval
LongMemEval: 95.6% (478 / 500), #1
Human-curated questions over long, multi-session chat histories. Wu et al., ICLR 2025.
All 500 questions of LongMemEval_S (cleaned), Gemini 3.8 Flash as the answer model, gpt-4o as the judge with LongMemEval’s own prompts, single run, about 7K tokens of retrieved context per question (7,391 mean). Every answer and verdict: eval_benchmark_runs_output/results/longmemeval. Full write-up: Synap is #1 on LongMemEval.
- Answer
- Gemini 3.8 Flash
- Judge
- gpt-4o, LongMemEval's own prompts
- Scope
- All 500 questions
- Dataset
- LongMemEval_S (cleaned)
- Runs
- Single run
| Category | Accuracy | Correct / n |
|---|---|---|
| Multi-session | 91.7% | 122 / 133 |
| Single-session user | 95.7% | 67 / 70 |
| Single-session assistant n=56 | 96.4% | 54 / 56 |
| Single-session preference n=30 | 96.7% | 29 / 30 |
| Knowledge update | 97.4% | 76 / 78 |
| Temporal reasoning | 97.7% | 130 / 133 |
The hairline across each bar marks the 95.6% overall figure.
Same retrieval, gpt-5.4 as the answer model: 95.2% (476 / 500).
Retrieval: the evidence needed for the answer was in Synap’s top 20 retrieved items for 461 of 470 answerable questions (98.1% Recall@20, gpt-4o relevance judge). That is a retrieval figure, not answer accuracy.
02LoCoMo
LoCoMo: 93.2% (older run, gpt-5-mini answer and judge)
Very long-term conversations, roughly 300 turns each. Maharana et al., ACL 2024.
- Answer
- gpt-5-mini
- Judge
- gpt-5-mini, binary
- Scope
- Categories 1–4, 1,540 questions
- Dataset
- locomo10, official release
- Runs
- Multiple runs, most-reproduced result
| Category | Accuracy | Correct / n |
|---|---|---|
| Single-hop | 88.8% | n = 282 |
| Temporal | 90.8% | n = 96 |
| Open-domain | 93.4% | n = 841 |
| Multi-hop | 97.3% | n = 321 |
The hairline across each bar marks the 93.2% overall figure. Correct counts are published for LongMemEval and not yet for LoCoMo; n is the official category distribution.
03Published scores
Published LongMemEval scores, and the conditions behind them
Every row except ours is the vendor's own published figure. We did not re-run them. Checked 28 Oct 2026.
| System | Accuracy | Context per question (tokens) | Results public | Eval harness open | Answer model | Judge model |
|---|---|---|---|---|---|---|
| Maximem Synap | 95.6% (478/500) · #1 | ~7.4K | Yes | Yes | Gemini 3.8 Flash | gpt-4o |
| Mastra | 94.87% | ~30K | Not found | Yes | gpt-5-mini | gpt-4o |
| Hindsight | 94.6% (473/500) | ~43.6K | Yes | Yes | Gemini 3.1 Pro | Gemini 2.5 Flash-Lite |
| Mem0 | 94.4% (472/500) | ~6.8K | No | Yes | Not stated | Not stated |
| Honcho | 90.4% (452/500) | ~11.6K | Yes | Yes | Claude Haiku 4.5 | gpt-4o |
| Zep | 90.2% (451/500) | ~4.4K | No | No | gpt-5.4 | gpt-5.4 |
| Supermemory | 85.2% | Not published | Not found | Yes | Gemini-3 Pro | gpt-4o |
| Letta | No LongMemEval score published | — | — | — | — | — |
Context = tokens the answer model reads per question, out of a ~115K-token history (Zep: median; others: mean or vendor-stated). Supermemory's headline 97% is Recall@20, a retrieval metric. Sources: mastra.ai/research/observational-memory, github.com/vectorize-io/agent-memory-benchmark, github.com/mem0ai/memory-benchmarks, honcho.dev/evals, getzep.com/research, supermemory.ai/research/longmembench.
Zep's research page reports 90.2% (451/500) with gpt-5.4 as answer model and judge.
04Configuration
What the answer model is actually worth
The standing objection to any memory benchmark is that the answer model does the work: put a large enough model behind a mediocre memory layer and the score comes up on its own. The objection is correct, and it is measurable — because Mastra published the measurement against themselves. Three LongMemEval scores, one memory layer, three answer models, nothing else changed.
| Memory system | Answer model | LongMemEval_S |
|---|---|---|
| Mastra Observational Memory | gpt-5-mini | 94.87% |
| Mastra Observational Memory | gemini-3-pro-preview | 93.27% |
| Mastra Observational Memory | gpt-4o | 84.23% |
| Maximem Synap | Gemini 3.8 Flash | 95.6% |
| Maximem Synap | gpt-5.4 | 95.2% |
Rail spans 80–100%, so the differences the figure is about stay readable. The hairline marks the best competitor configuration at 94.87%.
+10.6
points — the spread across three answer models for Mastra, from gpt-4o to gpt-5-mini, with the memory layer held constant.
+0.4
points — the spread across our two answer models, gpt-5.4 and Gemini 3.8 Flash, with retrieval held constant.
Read the two spans against each other. Changing the answer model was worth 10.6 points to Mastra. Changing ours moved Synap by 0.4. When the retrieved context already holds the evidence, the answer model has less left to carry, and the numbers bounding that comparison are theirs as well as ours.
Which matters well past the leaderboard. The answer model is invoked on every turn of an agent loop, so its tier is a standing cost rather than a benchmark-day detail. A memory layer that holds its accuracy on the cheap tier is a different economic proposition from one that needs a flagship model to reach a lower number, and the gap compounds with every conversation your product runs.
Why vendor scores differ
The answer model has the largest published effect and it is not the only variable. These are the rest — the settings that decide whether two memory benchmark numbers can be compared at all.
| Variable | Maximem Synap | Mem0 | Zep | Letta | Supermemory | Why it matters |
|---|---|---|---|---|---|---|
| Answer LLM | Gemini 3.8 Flash (LoCoMo: gpt-5-mini) | gpt-4o-mini | gpt-4o-mini / gpt-4o | gpt-4o-mini | gpt-4o / gpt-5 / Gemini-3 Pro | High. Newer and larger models score materially higher on identical data, measured above at 10.6 points across three answer models. |
| Judge LLM | gpt-4o (LoCoMo: gpt-5-mini) | gpt-4o-mini | gpt-4o-mini / gpt-4o | gpt-4o-mini | gpt-4o | High. Judges differ in false-positive and false-negative rates. |
| Embedder | Not published | text-embedding-3-small | BGE-m3 | File search | Not disclosed | Medium. Drives retrieval quality. |
| Reranker | Not published | None / graph | BGE-m3 reranker | n/a | Not disclosed | Medium. Second-stage precision. |
| Ingestion granularity | Not published | Message by message | Full conversation via API | File based | Session by session | Medium. Changes how memories are structured and time-grounded. |
| Adversarial (LoCoMo Cat 5) | Excluded | Excluded | Excluded | n/a | n/a | High. Inclusion shifts an overall score by more than ten points. |
| Runs / variance | Single run (LoCoMo: multiple runs; most-reproduced result) | Single run | 10 runs, mean ± std | Single run | Not reported | Medium. A single run carries real variance, and a best-of-N figure carries selection on top of it. |
| Timestamp handling | Not published | Embedded in text | Bitemporal (event + ingest time) | n/a | Dual-layer (doc + event date) | High, for temporal questions. |
Compiled from each vendor’s published papers, blogs and evaluation code. Every cell is a claim you can check against the source it came from; where a vendor has not stated a value, the cell says so rather than guessing. Four cells in our own column read “Not published”, and Zep reports ten runs with a standard deviation where we report a single run — both stay visible, because a page arguing that every figure needs its configuration cited does not get to quietly omit its own.
05Limits
Where we lose
Multi-session is our worst category at 91.7% (122 / 133), and it is the hardest category for every vendor on every published leaderboard. It asks the system to synthesize evidence from separate conversations rather than pull from one recent thread, which means linking the same entity across sessions when its surface form changes. That is part retrieval, part graph reasoning and part ontology engineering, and nobody has solved it.
Single-session user sits at 95.7%, three questions short of perfect, and no category reached 100%. On LoCoMo, single-hop at 88.8% is our lowest category and sits below multi-hop at 97.3%, which is counter-intuitive and which we do not yet have a published explanation for.
The scope limits matter as much as the scores. The LongMemEval figure is a single run, so it carries the variance any single run carries. The LoCoMo figure comes from repeated runs and we publish the score the harness returns most often, but we have published neither the run count nor the spread for it. LongMemEval was run on two answer models; LoCoMo on one. Both benchmarks are synthetic, which means neither reproduces the way a production workload accumulates contradictions over months — the case memory systems actually have to survive. And a LoCoMo number from this harness is only comparable against another Category 1 to 4 number.
06Reproduce
Run this yourself
A number you cannot inspect is a claim, not proof. Everything above is produced by an open harness against the official public dataset releases, with no custom subsets and no relabeling.
Every answer and judge verdict behind 95.6% is public. Count the CORRECT verdicts yourself: 478 of 500. Raw data.
- 01Clone the harness: maximem-ai/memory_and_context_eval_harness.
- 02Download the official datasets, LongMemEval_S and locomo10.
- 03Ingest into your provider, then run the answer pass with Gemini 3.8 Flash and the judge pass with gpt-4o (LoCoMo: gpt-5-mini for both).
- 04Score LongMemEval across all 500 questions; score LoCoMo across Categories 1 to 4.
git clone https://github.com/maximem-ai/memory_and_context_eval_harness cd memory_and_context_eval_harness && pip install -e . python scripts/download_datasets.py --variant s python -m runner.server
The harness ships adapters for Synap, Mem0, Zep and Supermemory, and a provider interface for anything else. Every deviation from the published benchmark protocols is recorded in DEVIATIONS.md, and the full methodology, including the cross-vendor configuration table above, is in METHODOLOGY.md. Results are published under CC BY 4.0.
Questions about benchmark numbers
LongMemEval and LoCoMo are the two that get cited. LongMemEval tests recall and reasoning across long multi-session histories; LoCoMo tests conversational memory over very long dialogue. Both are useful and both are synthetic, which is the limitation, because neither reproduces the way a real workload accumulates contradictions over months.
Usually the adversarial category. LoCoMo Category 5 questions are intentionally unanswerable, so they measure abstention rather than memory retrieval, and including or excluding them moves an overall score by more than ten points. We exclude it, matching the original paper, Mem0 and Zep. A LoCoMo number should only ever be compared against another Category 1 to 4 number.
Trust them exactly as far as the methodology is published and the run is reproducible. A score with a linked harness, a stated dataset version and a named answer model is a claim you can check. A score in a marketing table is a claim you cannot. Ask for the harness before you ask about the number — ours is open source and runs against our competitors as well as against us.
Because a larger answer model can carry part of the memory system’s job. Mastra published the size of that effect against themselves: the same memory layer scores 84.23% on LongMemEval_S with gpt-4o, 93.27% with gemini-3-pro-preview and 94.87% with gpt-5-mini, so the answer model was worth 10.6 points. Maximem Synap scores 95.6% with Gemini 3.8 Flash and 95.2% with gpt-5.4 on the same retrieval, a 0.4-point spread. That is what isolates the contribution of the memory layer, and it also makes it a cost argument: the answer model is invoked on every turn of an agent loop.
Retrieval accuracy asks whether the system returned the correct memory. Task accuracy asks whether the agent then did the right thing. A system can score well on the first and still fail the second, which happens whenever the right memory is retrieved and then buried in a context window large enough that the model reads past it.
Take fifty real conversations from your product, split each one so that facts established early are needed late, and score whether the agent completed the task rather than whether it retrieved the note. Run the same set against your current setup and against any candidate system. The absolute numbers matter less than the delta on your own data.