EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

Problem
This paper addresses a significant gap in the evaluation of memory mechanisms in Large Language Model (LLM) agents, which has been largely overlooked in existing benchmarks that focus primarily on reasoning, planning, and execution. The authors note that current benchmarks lack a systematic approach to assess the memory capabilities of agents, which are crucial for storing, updating, and retrieving information over time. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors introduce EvoMemBench, a unified benchmark designed to evaluate agent memory from a self-evolving perspective. The benchmark is organized along two axes: memory scope (in-episode vs. cross-episode) and memory content (knowledge-oriented vs. execution-oriented). The study compares 15 representative memory methods against strong long-context baselines using a standardized evaluation protocol. The evaluation includes various memory architectures, such as retrieval-based, procedural, and long-term memory methods, although specific architectural details and loss functions are not disclosed. The training compute used for the experiments is also unspecified.

Results
The results indicate that current memory systems are not yet a comprehensive solution for LLM agents. Long-context baselines demonstrate competitive performance across various tasks. Memory mechanisms are shown to be most beneficial when the current context is insufficient or when tasks are particularly challenging. Notably, no single memory form consistently outperforms others across all settings. Retrieval-based methods excel in knowledge-intensive scenarios, while procedural and long-term memory methods are more effective for execution-oriented tasks, particularly when their stored experiences align with the task structure. The paper provides quantitative results, although specific effect sizes and numerical comparisons against named baselines are not detailed in the summary.

Limitations
The authors acknowledge that their benchmark may not capture all aspects of memory performance and that the evaluation is limited to the selected memory methods. They also note that the effectiveness of memory systems can vary significantly depending on task complexity and context. An obvious limitation not discussed by the authors is the potential overfitting of memory methods to specific tasks, which may not generalize well to unseen scenarios. Additionally, the lack of detailed architectural and training compute information limits reproducibility and deeper analysis.

Why it matters
EvoMemBench has significant implications for future research on memory systems in LLM-based agents. By providing a structured framework for evaluating memory capabilities, it encourages the development of more effective memory architectures that can enhance the performance of agents in complex tasks. This work lays the groundwork for further exploration into how memory can be optimized and integrated into LLMs, potentially leading to advancements in areas such as conversational agents, automated reasoning, and task execution.

Authors: Yuyao Wang, Zhongjian Zhang, Mo Chi, Kaichi Yu, Yuhan Li, Miao Peng, Bing Tong, Chen Zhang et al.
Source: arXiv cs.CL
URL: https://arxiv.org/abs/2605.18421v1
arXiv ID: 2605.18421