The Read Side Decides: Where Agent Memory Accuracy Actually Lives

In ten days, seven papers landed on arXiv about agent memory, from groups that overlap — Fudan appears twice, on AgentMemRec and on BehaviorMem, and Meta FAIR is among them. Each defends its own system. Read together, they point somewhere none of them says out loud: the accuracy of your memory lives on the read side — in what enters the context window — and almost everything the industry sells you happens on the wrong side of that line. Here is the argument assembled from the seven: every number re-verified against the paper itself, the tension it cannot dissolve kept visible.

Designed memory loses twice, and the second loss is the one nobody names

The pitch of a designed store — Mem0's extracted facts, Zep's temporal graph — is that it curates your past so your future questions find what they need. The measurements say curation costs you the answer twice.

First loss: whatever it discards at write time stays gone. MemoryTax froze a store and swapped readers: on a Zep graph written by GPT-4o-mini, three readers of increasing strength — mini, GPT-6 Luna, Claude Opus 5.5 — scored 43%, 43% and 45%. On the same graph written by Opus: 66%, 71%, 87%. A stronger reader cannot recover what a weaker writer threw away. The write side is a trapdoor.

Second loss: even what survives storage doesn't get gathered when a question needs it. MemDerive built 1,503 questions over 4,692 events whose answers are derived state — computable from the history by rules, stated in no record. Four designed systems answered 30.41–48.30%. Plain BM25 answered 58.28%. The whole history in the window answered 79.84%. And the diagnostic that matters: asked directly, Mem0, Cognee and BM25 return 86–94% of the intermediate states one level above the evidence — but inside the context of the request that needs them, only 54–74%. Between 61 and 85% of the designed systems' wrong answers contain a state the system demonstrably has and did not fetch. On the same paper's ranking panel, Mem0 beats BM25 84.00% to 78.60% on LongMemEval — and BM25 beats all four systems on the questions that resemble what agents actually have to know.

AgentMemRec gave the read side its accounting identity. Memory selection is budgeted slate recommendation: request as user, stored turns as items, context window as slate. Accuracy factorizes into what recall and ranking deliver: get the evidence in, and the answering model does the rest. Slate completeness — how often the selected context holds all the annotated evidence — is 0.94 for their cascade and 0.50 for HippoRAG 2 and A-Mem. On the question-and-budget pairs where both happened to deliver complete slates, the accuracy difference was −2.8 to +2.2 points: not significant. Once the evidence is in the window, on that panel, the systems answered within noise of each other. The entire gap is the evidence you left outside.

Cheap selection is enough — but cheap in calls is not free

If accuracy lives in completeness, the money should go into completeness, not cleverness.

AgentMemRec's selector makes zero LLM calls: BM25 and dense retrieval fused by reciprocal rank fusion into a 128-turn pool, a learned ranker over ten cheap features, greedy fill to 2,700 tokens. On LongMemEval-S multi-session questions it scores 80.7 and 89.4 with two answering models, 10.4 to 31.2 points above T-Mem, which spent about 2,270 LLM calls per history building its store. It matches a 64-calls-per-query LLM reranker at statistical parity, at 3.2× lower serving cost — 280 against 891 GPU-milliseconds per request, roughly $69 against $220 per million requests. TrawMem attacks the same hole from the other side: select by the role a memory plays — anchor, bridge, context — not by how much it resembles the query; a submodular greedy pass fills the budget with coverage, and no LLM sits in the selection loop. It posts the best LoCoMo judge scores across four backbones, +2.23 to +3.14 macro-F1 over the second-best system, with ~12.8% and ~5.1% fewer context tokens than SimpleMem.

Two honest qualifiers on "cheap". TrawMem's price is wall clock, which its abstract omits and its Table 3 states: 699.9 seconds per conversation against SimpleMem's 605.3, and the bill is retrieval, not construction — 603.8 seconds against 501.4. And TrawMem beats the designed systems; it never runs the raw full-context baseline in the same table. Cheap-in-calls, expensive-in-seconds: know which currency your bottleneck is priced in.

One shared history is a leak. Partition it

The Meta FAIR paper started from an uncomfortable fact: CL-Bench, running its six task families as separate streams, found naïve in-context learning — raw trajectories back in the prompt — beat every dedicated memory system. Routed Continual Learning interleaved the families' 301 instances with 97 τ³-Bench banking instances, 398 in one stream, labels hidden. Under interleaving, the gain of one global ICL history over running stateless collapses to +0.015 normalized reward. On Opus 5, a global history is worse than nothing for the entire stream.

Partition, and the reward returns: Routed CL beats global ICL by 0.151 to 0.296 normalized reward across four models — and cuts estimated API cost 65–72%. Opus 5: $914 per rollout global, $317 routed. GPT-5.6 Sol: $456 to $126. Where does the win come from? Routing the raw histories alone — clustering, no clever representation — contributes 0.117 to 0.241; the exotic memory library adds a further 0.002 to 0.077, and on Gemini nothing. Partition first, represent second. A prompted router is already near its ceiling: oracle assignment with hidden labels adds at most 0.071, and the router picks the ground-truth memory system in 86% of cases, reproducing the true partition in 91%. One warning: routing ReasoningBank — a system that already retrieves selectively — produced no consistent gain and raised cost 8–29%. Some memories sort themselves; don't fragment them.

The format of a memory is not neutral metadata

The last failure mode is the cheapest to fix and the least suspected. CrossProfile crossed who-holds-a-fact against what-a-fact-does across 480 decision tasks with identical texts. Both natural compression policies fail where the design predicts: relevance-first loses 20 to 32 points in the classic hidden-profile cell, and an oracle relevance filter loses just as much — 23 to 31 — because decoys are on-topic by construction. Rarest-first averages to no better than random. Across all of CrossProfile's cells, accuracy tracks one thing: the net balance of supporting versus decoy evidence the memory kept; the policy that kept it adds almost nothing.

Then the tags. Every memory line carried the names of who stated it — and the decider systematically overweighted facts attributed to many agents, though sharedness was assigned, not observed. Stripping the holder tags raised classic-cell accuracy by +14.4 points on Qwen2.5-7B, +13.3 on Llama-3.1-8B, and +24.4 on DeepSeek-V4.1-Flash, from 72.8% to 97.2%. Reassigning the tags at random, order fixed, recovers the same amount: it is the attribution itself, read as corroboration. "Three agents said it" is one fact told three times, and your decider pays for it like three observations. The natural-discussion arm is worse and honest about it: the full transcript scored below an empty memory on all three models — private facts surface in full in only 1.7% of discussions. Evidence dies upstream of any memory.

The scoreboard

Read the raw side of that table as one column: the raw or simpler option wins on accuracy in five of the seven rows — in MemoryTax only at the strong models, losing at GPT-4o-mini. Where it loses, look at what beats it — in TrawMem, a simpler designed system (no raw baseline was run there); in Routed CL, routing: read-time partitioning, not write-time curation. No row shows write-time design beating raw history on accuracy. And read the limits column as one caveat: none of these results transfers across tables — the judges differ, the re-runs sit at different levels, several are single runs. What transfers is the direction.

What designed memory still buys — and where the argument bites itself

Here is the tension, in plain sight. If BM25 beats the designed stores, why did MemDerive's own prescription read like a love letter to structure? Split every request for derived state along its derivation; store rules as first-class versioned objects; record dependencies the way incremental view maintenance does; build structure at ingestion, not summaries. Five of this month's papers record that structure-as-storage didn't pay — MemoryTax with its write-time trapdoor, AgentMemRec's cascade beating T-Mem by 10.4 to 31.2 points, BehaviorMem behind raw history, CrossProfile's failed filters, and Routed CL, where the exotic memory library adds 0.002 to 0.077 and on Gemini nothing. MemDerive joins that tally with its own numbers — designed systems answering 30.41–48.30% where BM25 answers 58.28% — while being the one paper of the seven that says structure is exactly what's missing. TrawMem takes neither side: it structures only at read time, by role, and wins.

Both can be true, and the resolution is a distinction: structure as storage versus structure as maintained state. What failed this month is structure as storage format — distilled notes and graphs that answer 30–48% of the questions BM25 answers at 58%, that discard through a trapdoor at write time, that refresh only 41.1% of implicitly-propagated changes while raw retrieval refreshes 63.1% of the same. On propagation — the axis maintained state is pitched on — today's designed stores lose. But the surviving half of the argument was never about the graveyard of current stores. It is about what has to exist the moment the history stops fitting the window — and MemoryTax conditions its full-context win on exactly that: "while the whole history fits." When it no longer fits, something must decide what comes in, and the only designs that earn their cost are the ones the read side can audit: completeness-measurable selection, partitioned histories, recorded dependencies so staleness is computable. Structure is the prescription for a memory system that does not exist yet.

The two purchases that genuinely close today: token compression, and weak models. BehaviorMem supplies the first honestly: 2,018 tokens of memory readout against 7,338 for long context — over 70% less text at leading agentic recall, though raw history still beats it at remembering, 0.647 to 0.612: you are buying the bill, not the win. MemoryTax supplies the second: Mem0 beats full context at exactly one of three models (78% to 74% at GPT-4o-mini), its build alone costs more than whole full-context rollouts, and break-even needs 372–596 questions on one history where LoCoMo asks 152. A patch for weak models, amortized over traffic that may never come.

What to do differently, starting with your next memory decision

Seven changes, in order:

1. Keep the raw record as the source of truth. Write-through anything; delete nothing that isn't contradicted — BehaviorMem's retirement rule, absence never justifies deletion, is the one piece of designed memory to copy today.
2. Run the cheap test before buying a store. Can full context or BM25 over the raw record answer your real questions? On the month's evidence, the default assumption is yes, until your history measurably stops fitting.
3. Put your budget in completeness, not cleverness. Measure how often your selected context holds all the evidence, the way AgentMemRec scores slates at 0.94. A missing gold turn costs tens of points.
4. Select with zero LLM calls first. BM25 + dense + rank-fusion + a learned ranker matched a 64-call reranker at a third of the cost. If you add structure — roles, threads — price it in wall clock, not just calls.
5. Partition your memory by kind of work. A global history across heterogeneous tasks is worth +0.015 over amnesia. Cluster, route with a prompt, and route only what doesn't already sort itself.
6. Strip or neutralize attribution tags in memories that feed decisions. Preserve the balance of evidence, not the gist and not the topic hits — on-topic is not on-side. Names stripped is a change you can A/B this week.
7. Score raw-history baselines in every evaluation, and score remembering and inference separately. BehaviorMem's two pools ranked the same systems differently. A table without a raw baseline is a sales sheet.

The burden of proof has moved: not "why did you skip the memory system?", but "what does your memory system beat raw history at, measured, on your questions?" Until that answer has a number in it, the seven papers say: keep the record, select cheap, and let the model search.