M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

Problem
The paper addresses a significant gap in the evaluation of memory capabilities in multi-modal models, particularly in the context of long-form video understanding. While existing benchmarks focus on perception and reasoning, they lack systematic assessments of memory retention, fidelity of information preservation, and robustness against interference. This work is particularly relevant as it is a preprint and has not undergone peer review, indicating that the findings should be interpreted with caution.

Method
M$^3$Eval is introduced as a comprehensive evaluation framework designed to probe various dimensions of memory in multi-modal models. The framework is grounded in cognitive psychology, featuring tasks that isolate critical aspects of memory, such as retention and interference. The authors conduct extensive experiments using representative multi-modal models, although specific architectures and training compute details are not disclosed. The tasks are designed to evaluate how well models maintain disentangled representations when processing parallel video streams and how they ground memory sources in spatial versus temporal domains.

Results
The experiments reveal several consistent weaknesses across the evaluated models. Notably, models struggle to maintain disentangled representations when processing multiple video streams, indicating a limitation in their memory architecture. The interference patterns observed in these models differ significantly from those found in human memory, suggesting a fundamental misalignment in how these models process information. Additionally, the models demonstrate a stronger grounding of memory sources in the spatial domain compared to the temporal domain, and they exhibit limited capabilities in symbolic memory retention. The paper does not provide specific quantitative results or effect sizes against named baselines, which could enhance the understanding of the models’ performance.

Limitations
The authors acknowledge that their evaluation framework is the first of its kind, which may limit its immediate applicability across all multi-modal tasks. They also note that the findings are based on a limited set of models and tasks, which may not fully represent the diversity of multi-modal architectures. Furthermore, the lack of detailed information regarding the architectures and training processes of the models evaluated may hinder reproducibility and broader applicability of the results.

Why it matters
The introduction of M$^3$Eval provides a valuable resource for future research aimed at enhancing memory mechanisms in multi-modal models. By highlighting the underexplored nature of memory capabilities, this work encourages the development of more effective memory architectures that can better mimic human cognitive processes. The findings have implications for the design of future multi-modal systems, particularly in applications requiring robust memory retention and retrieval. This research contributes to the ongoing discourse in the field, as published in arXiv cs.AI.