Discovering Functionally Selective Brain Regions with a Deep Topographic Multimodal Model

Problem
This work addresses the limitations of existing topographic models that are unimodal and spatially constrain each layer separately, resulting in fragmented cortical maps. The authors highlight the need for a model that captures the contiguity of cortical processing streams and integrates multiple modalities, such as visual, auditory, and cognitive functions. This paper is a preprint and has not undergone peer review.

Method
The authors introduce Topo-Omni, a topographic multimodal model that utilizes a single contiguous in-silico sheet to represent various modalities. The architecture is built by fine-tuning a pretrained foundation model with a spatial smoothness objective, which encourages the development of clusters across modalities. This model is designed to reflect the systematic spatial organization observed in human neuroimaging data. The training process involves optimizing the model to ensure that driving or suppressing a cluster can selectively bias or impair perception, mirroring findings from human intervention studies.

Results
Topo-Omni demonstrates significant improvements over existing unimodal models. The authors validate their model against human neuroimaging data, showing that the clusters formed are consistent with known sensory and cognitive processing networks. They report the discovery of novel clusters related to natural landscapes and animal networks, which were validated through human data. The paper does not provide specific quantitative metrics or effect sizes against named baselines, focusing instead on qualitative validation through neuroimaging consistency.

Limitations
The authors acknowledge that while Topo-Omni successfully integrates multiple modalities, the model’s reliance on a pretrained foundation may limit its adaptability to entirely novel data distributions. Additionally, the spatial smoothness objective, while beneficial for clustering, may oversimplify complex cortical interactions. The lack of quantitative performance metrics against established benchmarks is also a noted limitation, as it makes it difficult to assess the model’s efficacy in comparison to existing methods.

Why it matters
The implications of this work are significant for understanding cortical organization and the integration of sensory modalities. By providing a unified framework for multimodal processing, Topo-Omni opens avenues for further research into the neural basis of perception and cognition. The model’s ability to generate testable hypotheses about cortical organization can inform future experimental designs and neuroimaging studies. This research contributes to the growing body of literature on multimodal neural representations, as discussed in related works available on arXiv.