MemCorrect: benchmarking what happens after you say 'that's wrong'
Recall benchmarks measure what a memory system finds. MemCorrect measures whether a correction sticks, and what it breaks on the way.
Every memory benchmark we know of asks some version of the same question: given a long history, can the system retrieve the right fact? That is worth measuring. It is also not where memory systems hurt people.
The painful failures happen after the user corrects the system. You tell your agent the staging host changed. It agrees. Two weeks later the old host is back in its answers, resurrected from a document it re-ingested. Nothing in a recall benchmark catches this.
MemCorrect is our attempt to benchmark the correction path itself.
The four axes
MemCorrect stresses a memory system on four questions:
- Does the correction stick across sessions and agents?
- Does the old fact resurrect when the source material is re-ingested?
- Does neighboring, still-valid memory survive the correction, or does the fix take collateral damage?
- Does the write path apply corrections it should have refused?
These map to real failure modes we hit while building Remnic: resurrection under re-ingest, collateral deletion, and false-apply on the write path. Published systems do not report numbers on any of them, which is exactly why we built the benchmark.
How Remnic runs it
Since v9.6.11, MemCorrect routes Remnic’s own corrections through the public correction contract, the same surface users hit. No privileged path, no benchmark-only shortcuts. The harness records dataset versions, model ids, judge ids, and commit SHAs so a result can be reproduced or challenged.
MemCorrect is open source and lives in the Remnic repo. It is designed so other memory systems can be scored on the same axes through a generic adapter.
Our full posture on measurement, including the numbers Remnic does not win yet, is on the benchmarks page. The team-scale extension of this work is Remnic Relay.