Blog

Action-site delivery: same failure memory, 35.6 points fewer repeats

A preregistered study on where failure memories should land in an agent's loop. Turn-start recall cut repeated failures from 44% to 36%. The identical fact delivered at the action site cut them to zero.

Updated August 13, 2026

Most agent memory systems, Remnic included, retrieve context at the start of a turn. The agent begins, the system injects whatever looks relevant, and work happens after. Storage formats and retrieval strategies differ. The delivery point rarely does.

We ran a preregistered experiment on whether that delivery point is the problem. The short version: a memory of failing barely helps at turn start. The same fact, delivered at the moment the agent proposes the action that failed, removed the repeated failure.

The study

Thirty synthetic TypeScript repair tasks, each with a deliberate trap: a wrong fix that looks more attractive than the right one. Mark the failing test as flaky instead of fixing the state bug. Patch the call site instead of the module that owns the bug. Six trap classes, five tasks each.

Two episodes per task. In episode one the agent falls into the trap, and the harness freezes that failure as a memory: the fact text plus a fingerprint of the trapping action. Episode two returns the same task with the agent’s own failure memory. The only variable is when it arrives.

One arm gets the memory at turn start, the way memory systems deliver it today. The other arm gets the same fact, same wording and token count, at the action site. A small advisory gate watches proposed tool calls, matches them against the fingerprint, and speaks up right then. The matched proposal does not run, the warning lands as a direct response to it, and the model decides again with the memory in front of it. Nothing is ever blocked.

The registration was hash-bound before data collection: tasks, arms, seeds, outcome definitions, statistical tests, and decision thresholds. The harness refuses to run if the documents drift.

The results

Without any memory, the model repeated its own known failure in 44% of episodes. The traps work.

With the failure memory at turn start, repeats fell to 36%. Eight points. A matched success memory, telling the agent what worked in a similar repo, actually beat the failure memory on task completion. The registered claim that failure wording beats success wording at turn start was rejected by the data.

With the same fact delivered at the action site: zero repeats in 270 raw episodes. On the preregistered task-level test, that is a 35.6 percentage point drop, p = 0.0019, on all 18 tasks with no excluded data. We measured the effect three times across the pilot and both registered runs. It landed between 35 and 38 points every time.

The honest part

Preventing the known failure did not make the agent finish more tasks. The completion estimate moved one and a half points with an interval touching zero. Agents blocked from the trap mostly wandered to a different wrong answer. The claim is narrow: action-site delivery removes a known failure mode. It does not make your agent smart.

The study compares two complete delivery mechanisms, not timing in isolation. The action-site arm includes the advisory, the interruption, and the reconsideration. Separating the effect of timing from the effect of the interruption is the follow-up experiment. And the tasks are synthetic traps, not real repositories. Both limits are drawn in the paper.

One registered run was voided under its own rules: 1,890 episodes completed, two unclassifiable, and the preregistered rule allowed zero exceptions in the timing comparison. The rule was fixed and re-registered before any new episode ran. Both raw logs are public. The kept confirmatory result is the smaller of the two runs.

What it means for Remnic

The takeaway is a routing rule, not a new architecture. Store failure memories with a fingerprint of the action that failed. Check the match at tool-proposal time. Inject the memory as an advisory at the moment of temptation, and fail open if anything goes wrong. Semantic recall and preference context stay at turn start, where they belong.

The gate primitive already exists in Remnic core. The production design is tracked in a public issue. When it ships, the delivery-timing result becomes a default behavior rather than a paper.

This study extends the measurement posture behind MemCorrect: benchmark the paths where memory actually hurts people, publish the losses, keep the registration honest.

Everything is open

The analysis replays from the public logs with zero model calls.