How we raised LongMemEval Recall@20 to 99.4% — and the experiments that didn't work

Supermemory's public bar is 97% Recall@20 on LongMemEval-S. We rebuilt the harness to measure the same thing at the same K, then ran all 500 questions. Here is the full result, the changes that moved it, and the ones that were noise.

The scoreboard, stated precisely

Supermemory's headline is 97% overall Recall@20 with aggregation on LongMemEval-S. Recall@20 means: for each question, does the labelled evidence session appear in the first 20 retrieved results? An item is one haystack session, not one message. We adopted that unit exactly — one memory node per session, with the official sessionId in metadata — so the comparison is retrieval-for-retrieval, not our favourite metric against theirs.

The earlier MemOS harness only printed @5 and @10. That is a different question. The harness now reports metrics at an arbitrary K, including metrics.atK and perCategoryAtK, and the full run below is scored at K=20.

On the full 500-question set the run reached 99.4% Hit@20, 98.3% evidence recall@20, 96.4% all-evidence@20, with MRR 0.934 and nDCG 0.929. Every category is at or above the published vendor row on hit and evidence recall: knowledge update 100%, single-session user 98.6%, single-session assistant 100%, single-session preference 96.7%, temporal reasoning 99.2%, multi-session 100%.

What actually moved the number

Most of the gain came from measuring the right thing and not throwing evidence away. Four changes did the work:

• One session, one memory. A message-pair unit fragments the evidence and makes a "recall" score depend on whether the retriever found the right sentence inside the right session. Ingesting the session as the retrieval unit matches LongMemEval's annotation and Supermemory's own protocol.
• Context in the vector, not in the answer. The embedder sees a compact[session · date · speaker] header. For conversational data we also put the previous turn in the embedded context: the answer to "what are your summer plans?" is often "researching adoption agencies" and shares none of the question's words. This was worth roughly +7 to +11pp Hit@10 on LoCoMo and is enabled in the LongMemEval run.
• Don't let the write gate delete evidence. The prompt-injection screen classified about 150 ordinary chat sessions in the haystack as instruction-like and excluded them from recall. The benchmark harness now disables the screen for the trusted corpus. The recorded 99.4% run predates that change, so it is a lower bound: a clean rerun can only add candidates, not remove them. A memory system that silently drops evidence is not more secure; it is less correct.
• Measure at the depth the benchmark uses. All of the @5/@10 numbers in the repo were true and irrelevant to the 97% claim. At K=20 the same retriever has more room, and the numbers are directly comparable.

What didn't work (and stayed out of the defaults)

Negative results are the useful part of a benchmark. These were measured on the same 100-question stratified sample and did not improve recall at K=20:

• OR-token BM25. Forcing every query term into an OR match raised MRR slightly (0.914 vs 0.901) but lowered evidence recall (97.1% vs 97.6%) and all-evidence (93.0% vs 94.0%). More lexical recall is not free when it displaces precise sessions.
• Chunked sessions. Splitting a session into 6-message windows and aggregating back to the session scored 97.5% evidence / 95.0% all-evidence on 20 questions, below the whole-session 100% / 100%. A cross-encoder or atomic fact extraction might make chunking worthwhile; naive windows do not.
• Key expansion and LLM query decomposition. Selecting fact-like sentences into the summary was neutral (96.8% / 93.0%). Decomposing a question into sub-queries with MiniMax M3.1 improved MRR (0.899 vs 0.872) but not recall. The missing evidence in multi-session questions is usually incidental in a session about another topic; a better query does not create the missing lexical or semantic hook.
• MMR and session diversity. Both reduce near-duplicates in the head, and both lowered recall on LoCoMo. Precision work belongs after the evidence is retrieved, not instead of it.

What this run does not claim

This is retrieval-only, deterministic evidence matching. It is not an LLM-judge accuracy score, and it is not comparable to a vendor number produced with a different judge, answer model or category set. Supermemory's 97% is one aggregated recall figure; our table reports hit and evidence recall separately. The full set was one run; the LoCoMo numbers in the docs are reported as a 3-run mean because single runs move by ~2pp. The harness, the raw JSON and the exact commands are in the repository.

Reproduce it

The full run is resumable: --db keeps the SQLite store and skips namespaces that are already populated, so a killed 500-question run does not re-embed 24k sessions.

npx tsx scripts/bench-longmemeval.ts --provider=ollama --model=nomic-embed-text --dimensions=768 --topk=20 --retrieve-only --keyword-weight=0.5 --semantic-weight=0.5 --contextual --ingest-concurrency=16 --db=/tmp/lme-full.db --out=/tmp/lme-full.json

Raw result: docs/artifacts/longmemeval-s-full-500-retrieval.json. The same harness measures the lexical ceiling (--no-embeddings), chunking, key expansion and LLM query decomposition, so the failures above can be re-run instead of taken on faith.

MemOS is MIT-licensed, local-first and SQLite-backed; the benchmark scripts are part of the repo, not a separate evaluation service. If a number here cannot be traced to a committed artifact or a command in docs/benchmark-comparison.md, treat it as marketing — including ours.