THE BENCHMARK QUESTION
The benchmarks are rigged. Everyone grades themselves.
Published Updated
This one cuts against us, because we lead with being first on LongMemEval, so let me meet it head-on instead of around it.
The criticism is fair. Much of this category grades its own homework, vendor disputes vendor on methodology, and there is little independent validation, which means every number, including ours, should be read with suspicion until you can see how it was produced.
So here is how we handle it. We show the methodology and link the evaluation, we report the cases where we lose, and we run the tests the skeptics correctly say nobody runs, memory pollution with contradictory facts, strategic forgetting, and concurrent-user stress, against ourselves. And we think the metric itself is usually the wrong one. The question is not "did it retrieve the right note." The question is "did memory improve the next action." We would rather be measured on that, even when it is harder.
The three figures we cite are the same everywhere on this site, and each one comes from a published run you can reproduce. On LongMemEval we score 92%. On LoCoMo we score 93.2%. P50 retrieval latency is under 15ms. The same system can score 30 or more points higher on a self-reported number than on a standardized harness, so reproducibility, not the headline number, is what to weight. The harness is open source. Re-run it.
A number you cannot inspect is a claim, not proof. Inspect ours.
How does memory retrieval improve AI agent performance?
Retrieval improves performance when it changes the next action, not when it returns the right note. An agent with working retrieval answers from what the user established three sessions ago instead of asking again, resolves a contradiction instead of averaging over it, and skips the re-explaining loop that burns tokens and patience. Retrieval accuracy is the input. Task completion is the thing worth measuring.
The proof, in numbers
Questions people ask before deciding
Retrieval accuracy asks whether the system returned the correct memory. Task accuracy asks whether the agent then did the right thing. A system can score well on the first and still fail the second, which happens whenever the right memory is retrieved and then buried in a context window large enough that the model reads past it.
LongMemEval and LoCoMo are the two that get cited. LongMemEval tests recall across long multi-session histories; LoCoMo tests conversational memory over extended dialogue. Both are useful and both are synthetic, which is the limitation, because neither reproduces the way a real workload accumulates contradictions over months.
A benchmark score tells you how a system behaves on the benchmark's distribution. It does not tell you what happens when two facts conflict, when a user asks for something to be forgotten, or when a thousand tenants write at once. We run memory pollution with contradictory facts, strategic forgetting, and concurrent-user stress against ourselves for that reason, and we publish the cases where we lose.
Take fifty real conversations from your product, split each one so that facts established early are needed late, and score whether the agent completed the task rather than whether it retrieved the note. Run the same set against your current setup and against any candidate system. The absolute numbers matter less than the delta on your own data.
The failure is quiet, which is what makes it dangerous. The agent produces a confident answer built on a stale or mismatched fact, and nothing in the output signals that memory was the cause. This is why a retrieval system needs to surface what it retrieved and why, and why an audit trail is a debugging requirement rather than an enterprise nicety.
Trust them exactly as far as the methodology is published and the run is reproducible. A score with a linked harness and a stated dataset version is a claim you can check. A score in a marketing table is a claim you cannot. Ask for the harness before you ask about the number.
Fewer clarifying questions on turn one, fewer repeated preferences, and a shrinking gap between what the user told you last month and what the agent behaves as though it knows. None of those are benchmark metrics, and all of them are visible in your own logs within a fortnight.
See the methodology behind the numbers.