What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory

Agent memory systems operate under a token budget: when accumulated history exceeds the window, something must be forgotten. Yet budget–accuracy frontiers report that accuracy falls as the budget tightens without saying what kind of failure the lost tokens cause. A budget-induced error is one of two kinds with opposite remedies. A loss is recoverable when the needed memory survived but retrieval missed it (fix the retriever), and irreversible when eviction destroyed it (only retention helps). We introduce the restore counterfactual, a per-question paired intervention that reinstates the specific evidence the question needs at read time, measures the change in end-task accuracy, and uses that change to decompose each policy's error mass into recoverable, irreversible, and residual (utilization-failure) shares on an oracle-answerable denominator. On LongMemEval-S, with GPT-4o-mini as reader/judge and a GPT-5.4-mini robustness reader, we evaluate {FIFO, random, redundancy-aware, LLM-importance} eviction at three budgets and two retrieval regimes. We find: (i) under tight budgets eviction loss is dominated by destruction — the two-bin irreversible share $\text{irreversible}/(\text{irreversible}+\text{recoverable})$ is 0.67–0.73 at 80k and rises to 1.00 as the budget tightens to 8k. The LLM-importance arm is the exception at 0.60; the three-bin share over all oracle-answerable errors is $\approx$0.40–0.44 at 80k; no-eviction is zero by construction. (ii) The recoverable component is real but is exposed or hidden by the retrieval regime, so budget–accuracy costs are not comparable unless the retrieval regime is reported. (iii) At matched accuracy the policies (even a content-aware LLM-importance arm) show no dissociation in destructiveness on a non-degenerate measure, detected at a 1.2–6 pp resolution (the same test resolves a deliberately destructive control). To our knowledge this is the first per-item, per-question restore-counterfactual audit of eviction for external agent-memory stores, on a standard conversational benchmark.