Audio-Visual Intelligence in Large Foundation Models

Problem
This paper addresses the fragmented state of the literature on Audio-Visual Intelligence (AVI) in the context of large foundation models. Despite significant advancements in unified audio-vision architectures, existing research lacks a cohesive framework for systematic comparison across diverse tasks, taxonomies, and evaluation practices. The authors present this work as a comprehensive survey, which is a preprint and unreviewed, to consolidate knowledge and provide a structured reference for future research.

Method
The authors propose a unified taxonomy for AVI tasks, categorizing them into three main areas: understanding (e.g., speech recognition, sound localization), generation (e.g., audio-driven video synthesis, video-to-audio), and interaction (e.g., dialogue systems, embodied agents). They synthesize methodological foundations that include:

• Modality Tokenization: Techniques for representing audio and visual data in a unified format.
• Cross-Modal Fusion: Strategies for integrating information from both modalities to enhance understanding and generation.
• Generative Models: Discussion of autoregressive and diffusion-based generation methods tailored for multimodal outputs.
• Large-Scale Pretraining: Insights into pretraining strategies that leverage extensive multimodal datasets.
• Instruction Alignment and Preference Optimization: Approaches to align model outputs with user preferences and instructions.

The paper also curates a selection of representative datasets, benchmarks, and evaluation metrics, facilitating structured comparisons across task families.

Results
While specific quantitative results are not provided in the abstract, the authors emphasize the importance of establishing benchmarks and evaluation metrics to assess performance across various AVI tasks. They identify open challenges in synchronization, spatial reasoning, controllability, and safety, which are critical for advancing the field. The survey serves as a foundational reference, suggesting that future work can build upon the established taxonomy and methodologies.

Limitations
The authors acknowledge the inherent challenges in synchronizing audio and visual data, as well as the complexities involved in spatial reasoning and controllability in AVI systems. They also note the need for improved safety measures in multimodal interactions. However, the paper does not address potential biases in the datasets used for training or the implications of model interpretability in AVI applications, which are critical considerations for responsible AI deployment.

Why it matters
This survey is significant as it consolidates a rapidly evolving field into a coherent framework, providing a structured reference for researchers and practitioners in AVI. By establishing a unified taxonomy and synthesizing methodological foundations, the work lays the groundwork for systematic exploration and comparison of AVI tasks. This can facilitate advancements in multimodal AI systems, enhancing their ability to perceive, generate, and interact in complex environments, which is crucial for applications in robotics, virtual assistants, and immersive media.

Authors: You Qin, Kai Liu, Shengqiong Wu, Kai Wang, Shijian Deng, Yapeng Tian, Junbin Xiao, Yazhou Xing et al.
Source: arXiv:2605.04045
URL: https://arxiv.org/abs/2605.04045v1