Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events

Problem
This work addresses a significant gap in the evaluation of video multimodal large language models (MLLMs) regarding their ability to capture momentary visual events, which are critical for answering specific questions about transient actions or state transitions. While existing models have shown progress in general video understanding, their performance on brief, localized visual evidence remains underexplored. The authors highlight that current methodologies often overlook these fleeting events due to sparse frame sampling, visual-token compression, and coarse temporal aggregation. This paper is a preprint and has not undergone peer review.

Method
The authors propose the Moment-Video benchmark, which consists of 1,000 human-verified video-question-answer (QA) pairs across seven domains and 25 fine-grained subcategories. The benchmark encompasses four task types: Temporal Occurrence, Temporal Counting, Action Description, and Temporal Reasoning. Each question is designed to require models to engage with momentary visual events, necessitating a focus on transient evidence rather than relying on persistent objects or global context. The evaluation involved 33 proprietary and open-source MLLMs, with a specific focus on their ability to process and reason about momentary visual events. The authors conducted diagnostic analyses to assess the impact of denser frame sampling and video length on model performance.

Results
The best-performing model, Seed-2.0-Pro, achieved an overall accuracy of only 39.6% on the Moment-Video benchmark. In contrast, most open-source models performed below 25%, indicating a substantial deficiency in momentary visual event understanding across the evaluated models. The diagnostic analyses revealed that while denser frame sampling improved performance for some models, it did not fully address the limitations in temporal fidelity. Additionally, longer videos presented increased challenges for temporal localization, further complicating the models’ ability to capture brief visual evidence.

Limitations
The authors acknowledge that the benchmark’s reliance on human-verified QA pairs may introduce biases based on subjective interpretations of momentary events. They also note that while denser frame sampling can enhance model performance, it does not resolve the fundamental issues related to temporal fidelity. Furthermore, the benchmark’s focus on momentary events may not encompass all aspects of video understanding, potentially limiting its applicability to broader tasks.

Why it matters
The findings from this research underscore the critical need for improved temporal fidelity in video MLLMs, particularly for applications requiring precise understanding of transient visual events. The Moment-Video benchmark serves as a valuable tool for diagnosing and addressing these limitations, paving the way for future advancements in video understanding. This work highlights the importance of developing models that can effectively capture and utilize brief visual evidence, which is essential for a wide range of applications in AI and machine learning, as published in arXiv cs.AI.