MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems

Problem
This paper addresses the gap in understanding and debugging memory systems in large language models (LLMs), which are critical for long-horizon reasoning tasks. Existing memory systems are often unreliable, and there is a lack of methodologies for tracing and attributing errors within these systems. The authors propose a novel framework to systematically analyze memory failures, which is particularly relevant given the increasing complexity of LLM architectures. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors introduce a framework that transforms memory pipelines into executable memory evolution graphs, allowing for fine-grained tracing of information flow within memory systems. This framework is implemented in a benchmark called MemTraceBench, which includes data from various memory systems such as Long-Context, RAG, Mem0, and EverMemOS. The benchmark is designed to systematically study different memory failure modes. Additionally, the authors propose an automatic attribution method that iteratively traces operation subgraphs to identify the root causes of failures. This method focuses on operation-level issues, such as information loss and retrieval misalignment, and integrates with a closed-loop system for prompt optimization, which enhances end-task performance.

Results
The proposed framework and methods were evaluated on MemTraceBench, revealing that memory failures are systematic and can be attributed to specific operational issues. The closed-loop system for prompt optimization demonstrated a performance improvement of up to 7.62% on downstream tasks compared to baseline models without this optimization. The authors do not specify the exact baseline models used for comparison, nor do they provide detailed metrics beyond the percentage improvement.

Limitations
The authors acknowledge that their framework is still in the early stages and may not cover all possible memory failure modes. They also note that the effectiveness of the automatic attribution method may vary depending on the complexity of the memory system being analyzed. An obvious limitation not discussed by the authors is the potential computational overhead introduced by the tracing and optimization processes, which could impact the efficiency of real-time applications.

Why it matters
This work has significant implications for the development and debugging of memory systems in LLMs. By providing a systematic approach to error tracing and attribution, it enables researchers and engineers to better understand the dynamics of memory operations and their impact on model performance. The integration of fine-grained attribution signals into prompt optimization represents a novel approach to enhancing LLM capabilities, potentially leading to more reliable and efficient models for long-horizon reasoning tasks. This research could pave the way for future advancements in memory architectures and their applications in various NLP tasks.

Authors: Xinle Deng, Ruobin Zhong, Hujin Peng, Xiaoben Lu, Yanzhe Wu, Guang Li, Buqiang Xu, Yunzhi Yao et al.
Source: arXiv:2605.28732
URL: https://arxiv.org/abs/2605.28732v1