Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model-Brain Alignment

Problem
This preprint addresses a significant gap in the evaluation of artificial vision models against human brain responses, specifically the reliance on prediction accuracy as the sole metric for model-brain alignment. Traditional methods do not elucidate which specific dimensions of the brain’s response space are effectively captured by these models. The authors propose a novel framework that not only assesses model-brain alignment but also facilitates brain-brain comparisons, thereby providing a more nuanced understanding of how well models replicate human visual processing.

Method
The authors introduce a unified framework that quantifies the recovery of target-brain response dimensions from both human brain responses and artificial vision model representations. They utilize repeated fMRI measurements to identify reproducible response dimensions across independent trial splits. The framework involves two main components: (1) predicting target-brain responses from another subject’s brain data and (2) predicting responses from a vision model’s internal representations. The strength of recovery for each reproducible dimension is quantified, allowing for a detailed analysis of alignment. The experiments are conducted on a subset of the Natural Scenes Dataset, where eight subjects viewed identical natural images during fMRI scanning. The authors compare pretrained models against randomly initialized models to assess differences in recovery profiles despite similar prediction accuracies.

Results
The findings reveal that early-to-intermediate visual cortex responses are characterized by a low-dimensional set of reproducible response dimensions. The framework demonstrates that brain-to-brain comparisons can identify consistently recoverable dimensions, providing a more robust diagnostic reference than scalar benchmarks. Notably, the authors report that pretrained and randomly initialized models achieve comparable prediction accuracies, yet exhibit distinct recovery profiles across the identified response dimensions. This indicates that high prediction accuracy does not necessarily correlate with effective model-brain alignment, highlighting the importance of the proposed evaluation framework.

Limitations
The authors acknowledge that their framework is contingent on the quality and reproducibility of fMRI data, which can vary across subjects and sessions. Additionally, the study is limited to a specific dataset and may not generalize to other visual stimuli or tasks. The authors do not address potential confounding factors such as individual differences in brain anatomy or function that could influence the recovery profiles. Furthermore, the framework’s reliance on dimensionality reduction techniques may obscure complex interactions within the response space.

Why it matters
This work has significant implications for the development and evaluation of artificial vision models. By shifting the focus from mere prediction accuracy to a more comprehensive understanding of model-brain alignment, researchers can better diagnose and improve model performance. The framework encourages the exploration of specific response dimensions that are critical for replicating human visual processing, potentially guiding future model architectures and training strategies. This approach could lead to more effective artificial vision systems that align closely with human perceptual mechanisms, enhancing applications in areas such as computer vision, neuroscience, and cognitive modeling.

Authors: Ken Nakamura, Tomoya Nakai, Ryuto Yashiro, Ayumu Yamashita, Kaoru Amano
Source: arXiv:2605.20127
URL: https://arxiv.org/abs/2605.20127v1