Applied ML · 2026
Agent Memory Bench
An open benchmark harness comparing memory strategies for LLM agents — no memory, vector recall, summarization, and a future-value-scored memory policy — with a public leaderboard.
PythonOpenAI APIDuckDB
Problem
Agent memory strategies are compared anecdotally. There is no standard harness measuring which approach wins where, at what cost, or how robust it is to memory poisoning.
Why I built it
My continual-learning research asks which structural changes will be valuable in the future. This ports that question to agent memory: which memories will be valuable — and can we predict it?
Architecture
- A task suite of multi-session scenarios tests recall, consolidation, transfer, and robustness to injected false memories.
- Each strategy runs against every task with fixed seeds; full traces log what was stored, retrieved, and at what cost.
- Results land in a warehouse-style schema; a static site regenerates the leaderboard from it.
Implementation
- Strategy plugins implement a common memory interface (write, retrieve, budget).
- Deterministic runner with fixed seeds and full trace logging.
- Metrics: accuracy, retention, forward transfer, token cost, memory size, poisoning success rate.
What I learned
- Too early for conclusions — the harness exists to produce them.
Next questions
- Does future-value-scored memory beat RAG at a fixed budget?
- How susceptible is each strategy to memory poisoning?
- Can the value scorer be learned from the harness's own counterfactual labels?