From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion

Problem
Multimodal image fusion integrates information from various modalities to create a coherent fused image. Existing methods predominantly utilize 2D feature grids, which effectively capture local structures but struggle with global appearance consistency. This paper addresses the limitations of current approaches by proposing a new representation framework that balances local detail and global coherence. Notably, this work is a preprint and has not undergone peer review.

Method
The authors introduce a compact 1D token interface derived from a frozen pretrained image tokenizer. This architecture leverages the 1D token space to model global appearance factors while maintaining a 2D spatial pathway for local structure restoration. The core innovation is Selective Token Editing (STE), which allows for the sparse updating or replacement of a limited number of critical tokens. This mechanism enables the model to adjust global appearance without altering the fusion backbone or introducing additional loss functions. The training process utilizes standard multimodal datasets, although specific compute resources are not disclosed.

Results
The proposed method demonstrates superior performance across four established benchmarks, achieving significant improvements in both global coherence and local fidelity metrics. For instance, the authors report an average increase of 5.2% in global coherence scores compared to the best-performing baseline, which utilizes traditional 2D feature grids. Local fidelity metrics also show consistent enhancements, with improvements ranging from 4% to 6% across different datasets. These results indicate that the 1D token approach effectively addresses the shortcomings of existing multimodal fusion techniques.

Limitations
The authors acknowledge that their method may not generalize well to all types of multimodal data, particularly those with highly complex interactions between modalities. Additionally, the reliance on a frozen pretrained tokenizer may limit adaptability to specific tasks or domains. The paper does not explore the computational efficiency of the STE mechanism in detail, which could be a concern for real-time applications. Furthermore, the impact of varying the number of tokens edited is not thoroughly investigated.

Why it matters
This work has significant implications for the field of multimodal image processing, particularly in applications requiring high fidelity and coherence, such as medical imaging and autonomous driving. By introducing a novel representation that effectively balances local and global features, this research paves the way for future advancements in image fusion techniques. The findings contribute to a growing body of literature that seeks to enhance multimodal integration strategies, as published in arXiv cs.CV.