Mitosis Cortex, evaluated on LongMemEval — the standard academic benchmark for long-term agent memory. End-to-end grading, the authors’ own judge prompts, every source cited.
LongMemEval (Wu et al., ICLR 2025) is the field’s standard test of long-term memory: 500 questions over long, timestamped chat histories. Frontier assistants lose ~30 points of accuracy on it — that difficulty is what makes it worth measuring.
Two axes decide a production memory system: how often it is right, and how fast it answers. Every figure below is end-to-end — an LLM judge grades the final answer, never a retrieval-only proxy.
Sources: vendor publications and independent reproductions, cited in full below. Cortex accuracy is preliminary — scope stated under Method.
A “95%” headline often means the right passage appeared somewhere in the top-k results — not that the system answered correctly. Recall@k is structurally the higher number.
Vendor-run harnesses reward benchmark-specific prompt tuning. Under a neutral, independent harness, one leading system’s claimed 93.4% reproduced at 73.8%.
A 20-point gap inside a single system. It is the reason we publish the protocol, not just the score.
An evaluation we would accept from a competitor. Each step removes a way to overstate the result.
Full multi-session histories, original timestamps, isolated workspace. No per-question tuning, no access to the answer key.
The same fused vector, full-text, and knowledge-graph search a live agent uses over MCP. Every retrieved item carries its source.
Claude Sonnet, pinned and verified per call, answers from retrieved context alone. All settings frozen before the run.
LongMemEval’s published judge prompts, used verbatim. Judge model disclosed: Grok 4.5 (academic default is GPT-4o); cross-judge agreement is being measured.
Scope. 91.7% is a preliminary figure from a stratified sample — two questions per category — and carries a wide confidence interval. The full 500-question run is underway and will be published here under the identical protocol, regardless of outcome.
The dataset is public and MIT-licensed. The judge prompts are published. We used both without modification, against the production system. A different result under the same protocol is exactly the finding this page exists to invite.
Wu, Wang, Yu, Zhang, Chang, Yu. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025. arXiv:2410.10813 · dataset & judge prompts
Supermemory. State-of-the-Art on LongMemEval. Vendor research page; end-to-end figures. supermemory.ai/research
Mem0. State of AI Agent Memory 2026. Claimed scores and reported latency. mem0.ai/blog
Maximem. State of AI Memory 2026: Claimed vs. Observed. Independent reproduction under a neutral harness (93.4% → 73.8%). maximem.ai/blog
Zep. Lies, Damn Lies, and Statistics. Corrected end-to-end and p95 search-latency figures; plus arXiv:2501.13956. blog.getzep.com
Liu et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. arXiv:2307.03172
Maharana et al. LoCoMo: Evaluating Very Long-Term Conversational Memory. ACL 2024. arXiv:2402.17753
Last updated 20 July 2026 · accuracy figure preliminary · versioned as the full evaluation completes.