LongMINT: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems

Problem
This paper addresses a significant gap in the evaluation of memory-augmented agents operating in long-horizon scenarios characterized by multi-target interference. Existing benchmarks predominantly focus on static recall tasks that do not account for the dynamic interactions and updates that occur in real-world applications. The authors highlight that current methodologies fail to capture the complexities of memory interference, which is critical for tasks requiring accurate recall and reasoning over multiple pieces of information. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors introduce LongMINT (Long-Horizon Memory under INTerference), a benchmark designed to evaluate the performance of memory-augmented systems in interference-heavy environments. LongMINT consists of 15.6k question-answering pairs derived from long, interconnected contexts, with an average of 138.8k tokens and instances extending up to 1.8M tokens. The benchmark encompasses diverse domains, including state tracking, multi-turn dialogue, Wikipedia revisions, and GitHub commits, facilitating domain generalization assessments. It features two primary question types: (i) single-target recall tasks that require the retrieval of specific targets from extensive contexts, and (ii) multi-target aggregation tasks that necessitate reasoning over multiple relevant pieces of information. The evaluation includes seven representative systems, such as vanilla long-context LLMs, Retrieval-Augmented Generation (RAG), and various memory-augmented frameworks.

Results
The evaluation reveals that all tested systems exhibit consistently low performance, with an average accuracy of 27.9%. Notably, the systems struggle significantly with questions that require aggregated reasoning over multiple pieces of evidence. The analysis indicates that the primary limitations stem from deficiencies in retrieval mechanisms and memory construction processes. Performance degrades as the number of intervening updates increases, particularly affecting the systems’ ability to recall and reason about earlier facts that have been revised or interfered with by subsequent context.

Limitations
The authors acknowledge that the performance of current memory systems is hindered by their inability to effectively manage interference and recall relevant information in the presence of updates. They do not explicitly address potential biases in the dataset or the representativeness of the evaluated systems, which may limit the generalizability of the findings. Additionally, the benchmark’s reliance on specific domains may not fully capture the breadth of real-world applications.

Why it matters
The introduction of LongMINT has significant implications for the development of memory-augmented agents, as it highlights the necessity for benchmarks that reflect the complexities of real-world information processing. By exposing the limitations of current systems in handling multi-target interference, this work encourages further research into improving memory retrieval and reasoning capabilities. The findings underscore the importance of developing more sophisticated architectures that can effectively manage dynamic memory interactions, which is crucial for advancing the state of the art in long-horizon agent systems.

Authors: Hyunji Lee, Justin Chih-Yao Chen, Joykirat Singh, Zaid Khan, Elias Stengel-Eskin, Mohit Bansal
Source: arXiv:2605.18565
URL: https://arxiv.org/abs/2605.18565v1