Benchmarks

LongMemEval-S full 500 at K=20, full-dataset LoCoMo, ConvoMem, HotPotQA multi-hop, BEAM-1M, and a 10k-memory haystack — local embeddings and hybrid retrieval, reproducible from the repo.

Verified runs

Regenerated automatically from result JSONs committed to the repo — if a number on this page can't be traced to a run below, treat it as marketing.

LoCoMo · full dataset · retrieval-only evidence matching

All 10 conversations, 1,986 questions, scored by direct evidence-ID matching — no LLM judge, no fuzzy matching. This run uses BAAI/bge-small-en-v1.5 with no reranker, which is what `scripts/bench-locomo-noapi-results.json` reproduces. Per category: single-hop 68.4% Hit@10, multi-hop 53.9%, temporal 67.9%, adversarial 70.2%, open-domain 33.7%. An earlier operating point with LFM2.5-Embedding-350M plus a bge-reranker reported 83.2%, but that artifact is not checked in, so it is not claimed here.

LoCoMo · span aggregation · post-hoc on the same retrieved lists

No re-retrieval and no model calls: the recorded top-10 lists are re-scored after merging adjacent turns. Filling the interior of a span changes nothing — the top-10 hits are already contiguous — but expanding the best hit by one turn on each side adds 5.5pp Hit@10 for 0.7 extra turns and does not lower precision. Expanding every hit adds 13.7pp but nearly doubles the injected context. The recommended default is top-1 expansion; top-3 if the budget allows. Raw artifact: docs/artifacts/locomo-span-aggregation.json.

LongMemEval-S · full 500 · retrieval-only · K=20

500 questions, one memory per haystack session (the benchmark's own retrieval unit), nomic-embed-text 768d, contextual session/date header, 0.5/0.5 hybrid fusion, no reranker. This is the same K and retrieval protocol as Supermemory's published 97% overall Recall@20: MemOS reaches 99.4% Hit@20 and 98.3% evidence recall@20. All-evidence recall@20 (every labelled evidence session in the top 20) is 96.4%, a stricter secondary metric. Retrieval now takes ~1 min for all 500 (down from ~21 min before the namespace index); a warm 100-question rerun is ~20 s, and --fast-ingest cuts setup work with identical scores. Raw run: docs/artifacts/longmemeval-s-full-500-retrieval.json.

ConvoMem · 525 sampled / 387 scored · retrieval-only · K=10

Salesforce/ConvoMem, one JSON file per evidence category: 525 sampled questions, 387 with annotated message evidence and scored; 138 abstention/no-answer items cannot be scored by retrieval-only matching. All-minilm 384d, contextual speaker/conversation header, 0.5/0.5 fusion, top 10. ConvoMem's official metric is LLM-judge answer accuracy; this is the deterministic retrieval layer. Reproduce with scripts/download-convomem.ts and scripts/bench-convomem.ts.

LongMemEval-S — end-to-end QA — independent judge

Cost per correct = tokens per correct answer × price / 1,000,000. The prices are examples, not a quoted rate — plug in your provider's blended input/output price. Tokens are the measured answer and judge total from the run, so the token side is real even where the price is an assumption.

The retrieval tables above say the evidence was returned. This says a reader could answer from it. DeepSeek V4.1 Flash answers at high reasoning and GLM-5.3 grades — an independent model, not the answerer. A self-judged run (DeepSeek as both) produced the same 78.0%, so the result is not judge-inflated. n=50 gives a ±13.9pp interval; treat it as a sanity check, not a leaderboard number. Raw run: docs/artifacts/qa-glm-judge-50.json.

Three arms: floor, memory, ceiling

The floor and ceiling are same-model runs (deepseek-v4-flash:free answering, deepseek-v4.1-flash:free judging, 0 errors): the model knows almost none of the personal facts without memory, and stuffing every haystack session into the prompt only reaches 16% while spending 2.07M tokens. The memory row uses the paid DeepSeek V4.1 Flash + GLM-5.3 pair, so do not subtract the rows — they bracket the problem, they are not a controlled ablation. A same-model memory run was blocked by free-tier rate limits; that gap is stated rather than papered over.

HotPotQA · multi-hop retrieval

500 validation questions, ~4,900 deduplicated Wikipedia paragraphs ingested corpus-wide, scored by supporting-paragraph title matching. Comparison questions are perfect across every metric; bridge questions sit at 95.4% evidence recall. This is the dataset class Cognee uses in its published memory evals.

Memory haystack · precision at scale

The needle-in-a-haystack test adapted to memory: unique facts planted at random depths in a growing distractor corpus, one natural-language query per needle. Deterministic, CPU-only — precision holds at 10k memories with an honest near-linear latency curve.

BEAM-1M · retrieval at scale