Benchmarks

Memory benchmarks that test the hard parts.

Remnic's stance is measured, not asserted. The harness covers published memory-agent suites and Remnic-specific regression packs, and the results below include the numbers Remnic does not win yet. Trust in the good numbers requires publishing the bad ones.

Runbook Readiness audit
MemCorrect

The correction benchmark.

Most memory benchmarks measure what a system can recall. MemCorrect measures what happens after you tell it something was wrong. A memory layer that cannot take a correction is a liability, so Remnic built and published a benchmark that stresses exactly that path.

MemCorrect is open source and lives in the Remnic repo. It is designed so other memory systems can be scored on the same axes.

  • Does the correction stick across sessions and agents?
  • Does the old fact resurrect when the source material is re-ingested?
  • Does neighboring, still-valid memory survive the correction?
  • Does the write path apply corrections it should have refused?
Latest AMB work

Provider bridge, verifier, and PersonaMem evidence.

Remnic now ships a native Agent Memory Benchmark provider bridge plus a SOTA verifier. The current PersonaMem 128k artifact is useful evidence, but it is intentionally not published as a SOTA claim until clean Remnic and AMB git provenance is rerun.

  • Full PersonaMem 128k artifact: 2,727 queries, 1,660 correct, 60.87% accuracy.
  • Target beaten: 52% Gemini-1.5-Flash baseline from the PersonaMem paper.
  • SOTA label withheld: the original verifier manifest recorded dirty or missing provenance.
  • Current verifier fails publishable runs unless result rows, LLM IDs, query count, and git provenance are clean.
Where Remnic trails today

The numbers we are not proud of, published anyway.

These are current, tracked, and public. Each one links the GitHub issue where the work to close the gap is happening.

  • LoCoMo: the real behavioral profile currently scores 0.444 against a 0.474 LCM-only baseline. Issue #1879
  • LoCoMo adversarial abstention is the largest single accuracy lever, at 0.245 on 446 items. Issue #1878
  • Judge calibration varies by slice across the same judge pair, so absolute accuracy claims are held loosely. Issue #1877
  • Remnic does not claim a LoCoMo leaderboard position, and will not until full-provenance runs support one.
Published suite

Coverage across nine public memory benchmarks.

AMA-Bench

Long-horizon action/observation trajectory memory

MemoryArena

Interdependent multi-session planning and dependency recall

AMemGym

Interactive personalization and latest-state recall

LongMemEval

Long-term conversational memory across temporal sessions

LoCoMo

Long conversation QA with dialogue, speaker, and session cues

BEAM

Extreme-scale conversation memory with plan and chat references

PersonaMem-v2

Implicit preference learning and preference updates

MemoryAgentBench

Event, date, keypoint, and conflict-resolution memory

MemBench

Factual and reflective step/time recall

Leaderboard safety

Exact cue recall without answer leakage.

Remnic can retrieve exact evidence from turn numbers, dates, speakers, plan ids, field names, preference updates, and keypoints when those cues are visible in the user question or were stored in memory. The harness keeps hidden scoring metadata out of answering recall, so high scores represent Remnic's retrieval behavior rather than benchmark leakage.

  • Full runs use isolated benchmark memory stores, not production user memory.
  • Artifacts record dataset versions, seed, model ids, judge ids, runtime profile, commit SHA, and manifest data.
  • Hidden gold answers, target ids, final state, and evidence labels stay out of answering recall.
  • Visible cue anchors are derived from stored memory or user-visible prompts and stripped before answer scoring when needed.
  • Quick mode is treated as smoke testing only; full mode is required for public or leaderboard-style claims.
How to read results

Quick runs are smoke tests. Full runs are the credible numbers.

A public result should name the dataset version, Remnic commit, model, judge, seed, runtime profile, and artifact manifest. It should also say whether QMD, graph recall, temporal supersession, and explicit cue recall were enabled. Remnic.ai will label smoke fixture results separately from full published-dataset runs.