Verging Labs ran 8 agent memory systems through 2,176 tasks across 56 simulated working sessions. We did not run it, fund it, or see the questions. Cortex finished first among products, second overall, and answered nothing incorrectly. Our own LongMemEval numbers are further down, labelled as ours.
LongMemEval (Wu et al., ICLR 2025) is the field’s standard academic test of long-term memory: 500 questions over long, timestamped chat histories. Frontier assistants lose around 30 points of accuracy on it, and that difficulty is what makes it worth measuring.
Read this section differently from section 03. We run this one, so it is self-reported. Everything below is our own harness against our own production system, published with the protocol attached so it can be argued with.
Two axes decide a production memory system: how often it is right, and how fast it answers. Every figure below is end-to-end — an LLM judge grades the final answer, never a retrieval-only proxy.
Sources: vendor publications and independent reproductions, cited in full below. Cortex accuracy is preliminary — scope stated under Method.
The Agentic Memory Index is an evaluation by Verging Labs. Eight agent memory systems, plus Claude Code’s built-in memory as a baseline, are each dropped into weeks of simulated company work and tested every time the agent touches memory: 272 tasks per system, 2,176 in total, across 56 working sessions.
Every company, person and tool in the corpus is fictional, so no model prior helps and contamination is detectable. We had no hand in the design, the questions, or the grading. The figures below are from their revision dated 3 August 2026.
The chart above covers the seven commercial products plus the Claude Code baseline. It excludes Karpathy Wiki, which scored 98%: it is a hand-maintained wiki that a person curates, not a system you can buy and run unattended. Its full figures are in the source deck linked below. On the false memory check, every system in the field scored 100% except one, so that row separates nobody.
Answered confidently, but wrong. Cortex did this zero times in 272 tasks. Mem0, the highest in the field, did it in 3.0% of runs. This is the failure that costs you the invoice, the email, or the customer.
Asserted a memory that was never stored. Zero for Cortex, and zero for every other commercial system tested. Only the Claude Code built-in baseline fabricated, in 3.5% of its runs.
An old value came back after an update. Second best in the field. Zep, the worst on this measure, returned stale answers 13.5% of the time.
Never addressed the question asked. Tied second lowest, behind Karpathy Wiki at 1.0%. This is the safe way to fail, and it is still a miss. It is the number we are working on.
Early facts are probed again at five checkpoints as the simulated relationship accumulates. Cortex is the only system in the field still perfect at session 33. It is also the one that falls fastest after that.
Karpathy Wiki ends highest at 95%. Our curve is the cleanest evidence we have for the temporal work already underway, recency weighting and long-horizon pruning. When that ships, this row is where you should check whether it worked.
A “95%” headline often means the right passage appeared somewhere in the top-k results — not that the system answered correctly. Recall@k is structurally the higher number.
Vendor-run harnesses reward benchmark-specific prompt tuning. Under a neutral, independent harness, one leading system’s claimed 93.4% reproduced at 73.8%.
A 20-point gap inside a single system. It is the reason we publish the protocol, not just the score.
An evaluation we would accept from a competitor. Each step removes a way to overstate the result.
Full multi-session histories, original timestamps, isolated workspace. No per-question tuning, no access to the answer key.
The same fused vector, full-text, and knowledge-graph search a live agent uses over MCP. Every retrieved item carries its source.
Claude Sonnet, pinned and verified per call, answers from retrieved context alone. All settings frozen before the run.
LongMemEval’s published judge prompts, used verbatim. Judge model disclosed: Grok 4.5 (academic default is GPT-4o); cross-judge agreement is being measured.
The dataset is public and MIT-licensed. The judge prompts are published. We used both without modification, against the production system. A different result under the same protocol is exactly the finding this page exists to invite.
Wu, Wang, Yu, Zhang, Chang, Yu. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025. arXiv:2410.10813 · dataset & judge prompts
Supermemory. State-of-the-Art on LongMemEval. Vendor research page; end-to-end figures. supermemory.ai/research
Mem0. State of AI Agent Memory 2026. Claimed scores and reported latency. mem0.ai/blog
Maximem. State of AI Memory 2026: Claimed vs. Observed. Independent reproduction under a neutral harness (93.4% → 73.8%). maximem.ai/blog
Zep. Lies, Damn Lies, and Statistics. Corrected end-to-end and p95 search-latency figures; plus arXiv:2501.13956. blog.getzep.com
Liu et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. arXiv:2307.03172
Maharana et al. LoCoMo: Evaluating Very Long-Term Conversational Memory. ACL 2024. arXiv:2402.17753
Overing, M. (Verging Labs). Inside the Agentic Memory Index. Independent evaluation of 8 agent memory systems plus a Claude Code baseline; 272 tasks per system over 56 simulated sessions. Revision dated 3 August 2026, superseding the v0.1 figures previously shown on this page. Every figure in section 03. x402oracle.com/deck
Last updated 3 August 2026 · Agentic Memory Index figures are third party · LongMemEval figure is ours and preliminary · versioned as the full evaluation completes.