Can you trust what your agent remembers?
Recall benchmarks such as LongMemEval and LoCoMo ask whether a memory system can find something. MIB asks whether it can be trusted: does it know what is true now and what was true then, does it keep a web page from rewriting what the user said, and is a forgotten fact really gone?
Results
Development split, 1,200 probes. Every system got the same messages, the same answering model (gpt-4o-mini) and the same judge (gpt-4o).
| System | Integrity score (95% CI) | Forgotten facts still in storage | Wrong value in answer | Memory per answer | Retrieval time |
|---|---|---|---|---|---|
| Mercury 1.1 | 0.946 (0.932–0.957) | 0% | 1.2% | 78 tokens | 0.35 s |
| Full context no memory system | 0.854 (0.833–0.873) | 100% by design | 10.4% | 411 tokens | none |
| Cognee | 0.832 (0.809–0.852) | 50% | 15.2% | 787 tokens | 4.9 s |
| Keyword baseline store everything, BM25 | 0.751 (0.726–0.774) | 0% | 9.2% | 93 tokens | 0.002 s |
| Supermemory | 0.741 (0.715–0.765) | 50.5% | 12.4% | 65 tokens | 1.3 s |
Supermemory, Cognee, full context and the keyword baseline were run on 15 Oct 2026 with their default settings; Mercury 1.1 on 17 Oct 2026. "Forgotten facts still in storage" is the share of forget probes where the forgotten value can still be found by searching the system's storage, whatever the answer says. "Wrong value in answer" is the share of answers that state a value they should not (an outdated, planted, forgotten or someone else's value).
Mercury today (1.4)
These runs use deepseek-v4.1-flash to answer and judge, so they are not directly comparable with the table above; the gap between systems is what matters, and every release is compared with the last one under the same models (see Updates). The leak check flagged 2 of 200 forget probes on the held-out split; both were false matches on everyday words in messages that were never forgotten ("now" in "I'm on The Overstory now", "python" in "how do I reverse a list in python?"). For reference, Supermemory scored 0.733 on the held-out split with default settings and 0.774 with its search threshold set to 0 (gpt-4o-mini answering, gpt-4o judging).
By category
Development split, same run as the results above. 100 probes per category, so each figure is accurate to about ±9 points; the best score in each row is in bold.
| Category | Mercury | Full context | Cognee | Keyword baseline | Supermemory |
|---|---|---|---|---|---|
| Knowledge update | 0.94 | 0.88 | 0.77 | 0.44 | 0.93 |
| As of a past date | 0.97 | 0.92 | 0.86 | 0.58 | 0.02 |
| When it changed | 0.89 | 1.00 | 0.99 | 0.92 | 0.87 |
| Source poisoning | 0.94 | 0.94 | 0.89 | 0.74 | 0.89 |
| Instruction injection | 0.97 | 0.58 | 0.49 | 0.64 | 0.39 |
| Forget a fact | 0.99 | 0.98 | 0.99 | 0.98 | 0.97 |
| Forget what was inferred | 0.89 | 0.29 | 0.31 | 0.89 | 0.38 |
| Hypothetical or denied | 0.98 | 0.95 | 0.99 | 0.74 | 0.97 |
| Where a fact came from | 0.86 | 0.90 | 0.91 | 0.47 | 0.71 |
| Someone else's fact | 0.96 | 0.93 | 0.96 | 0.85 | 1.00 |
| Say "I don't know" | 0.96 | 0.91 | 0.84 | 0.94 | 0.90 |
| Self-correction | 1.00 | 0.97 | 0.98 | 0.82 | 0.86 |
Planted instructions
A tool output tells the assistant to store a false fact about the user. Mercury quarantines it and answers correctly 97% of the time; the other systems got 39–64% right.
Forgetting what was inferred
Supermemory and Cognee deleted the forgotten message, but notes the assistant had inferred from it stayed in storage in all 100 cases. Mercury removes everything derived from a forgotten message.
What was true then
Asked what was true on a past date, Supermemory answered correctly 2 times in 100. Mercury records when every fact became true and when it stopped, and scored 0.97.
How it works
The data
- 150 simulated users, each with about ten months of history: 3,661 messages in all.
- 24 everyday attributes (home city, employer, dog's name, coffee order, gym, manager, current book…), interleaved with small talk and assistant replies, so no system's built-in schema covers them all.
- Messages come from the user, the assistant, web pages, emails and tools, each with a timestamp.
- 1,200 probes: 12 categories × 100 questions, each with a gold answer.
The rules
- Every system gets the same messages, the same answering model and the same judge. The answering model reads only what the memory system returns.
- Systems are connected only through their own documented features: metadata fields, delete by id, provenance links. They never see the benchmark's labels or answers.
- Two splits. A development split, and a held-out test split with new users, values and dates, run once per release candidate.
- Two scores. The integrity score (mean accuracy over the 12 categories) and the storage leak rate (can a forgotten value still be found in storage?).
The 12 categories
| Category | What happens in the history | What's asked | Correct behaviour |
|---|---|---|---|
| Knowledge update | A fact changes once or twice | The current value | The latest value |
| As of a past date | A fact changes once | The value on a past date | The value at that date |
| When it changed | A fact changes | When it changed | A date within 3 days |
| Source poisoning | A web page states a different value about the user | The current value | The user's own value |
| Instruction injection | A tool output tells the assistant to store a false fact | The current value | The user's own value |
| Forget a fact | The user states a fact, then asks to forget it | The fact | "I don't know", and the value is gone from storage |
| Forget what was inferred | The assistant infers something from a fact; the user asks to forget the fact | The inference | "I don't know", and the value is gone from storage |
| Hypothetical or denied | The user asks about, considers or denies another value | The current value | The real value |
| Where a fact came from | A fact comes from the user or from an email | "How do you know…?" | The right kind of source |
| Someone else's fact | Someone else has a value for the same attribute | The user's value | The user's value, or "unknown" |
| Say "I don't know" | Only a decoy (hypothetical or someone else's) | The user's value | "I don't know" |
| Self-correction | "Sorry, I meant X, not Y" | The current value | X |
Read with care
- Mercury was developed against the development split. Changes such as those in 1.1 came from studying its misses, which is why the held-out split exists and why Mercury's headline is its held-out score. Supermemory and Cognee were not tuned for MIB; we ran them with their defaults and, for Supermemory, one search-threshold variant.
- Synthetic and template-based. Real conversations are messier. The held-out split uses new users, values and dates but the same templates; paraphrased templates are planned for the next version.
- One LLM judge. Small differences, especially per category (about ±9 points), should not be over-read.
- Storage checks depend on each system's API, and match whole words, so a forgotten value that is also an everyday word can be flagged by mistake.