The Importance of Phase in Neural Representations: An Internal Oppenheim-Lim Test of Image Classifiers

Problem — This work addresses the gap in understanding how image classifiers utilize Fourier phase and magnitude in their internal representations. Previous studies, such as Oppenheim and Lim (1981), demonstrated that natural images can be recognized from their Fourier phase alone, but it remains unclear whether modern neural networks exhibit similar behavior. This paper is a preprint and has not undergone peer review.

Method — The authors employ a causal testing framework to analyze the internal representations of various image classifiers, specifically PRISM2D, GFNet, ViT-B/16, and ResNet-50. They transplant the phase of one image onto the magnitude of another at selected layers and observe the resulting predictions. The experiments focus on the hidden layers of these architectures, particularly examining the effects of manipulating phase and magnitude on classification accuracy. The authors also implement controls, including a DC-only condition, to ensure that the observed effects are not trivial artifacts of the magnitude manipulation.

Results — The findings reveal that for PRISM2D, GFNet, and ViT-B/16, the predictions predominantly follow the phase donor, indicating that identity is encoded in the phase representation. Notably, removing all image-specific magnitude components results in minimal accuracy degradation, suggesting that magnitude is largely dispensable for identity recognition. In contrast, ResNet-50 initially appears to deviate from this pattern; however, further analysis shows that a strong latent sign code exists in its late blocks when interventions are applied before the ReLU activation functions. The authors quantify the accuracy drop when magnitude is manipulated, providing specific metrics that demonstrate the robustness of phase-based identity encoding across different architectures.

Limitations — The authors acknowledge that their findings are limited to the specific architectures tested and may not generalize to all neural network designs. Additionally, the reliance on causal interventions may introduce complexities in interpreting the results, particularly in how different layers interact with phase and magnitude. The study does not explore the implications of these findings on real-world applications or the potential for adversarial attacks exploiting phase information.

Why it matters — This research has significant implications for the understanding of neural network interpretability and the mechanisms underlying image classification. By elucidating the role of phase in identity encoding, the study provides a mechanistic account of the differences in representation between convolutional neural networks (CNNs) and attention-based models. This insight could inform future architectural designs and training strategies, particularly in tasks where texture and shape representations are critical. The findings contribute to the broader discourse on neural representation and are relevant for ongoing research in the field, as published in arXiv cs.AI.