Spectral Audit of In-Context Operator Networks

Problem
Existing evaluations of neural operators and in-context operator learning predominantly focus on prediction error, which does not ensure the accurate representation of local dynamical structures. This paper addresses the gap in the literature regarding the assessment of learned operators, particularly in their ability to capture the underlying mechanisms of partial differential equations (PDEs). The authors highlight that conventional metrics may overlook critical operator-level phenomena, leading to misleading conclusions about model performance. This work is presented as a preprint and has not undergone peer review.

Method
The authors propose a Jacobian-based spectral audit that differentiates the network output concerning the query function, treating the resulting Jacobian as a learned tangent operator. This operator is then projected onto Fourier modes to derive a local spectral characterization of the inferred operator. Key aspects evaluated include frequency-dependent gains, phase structure, and cross-mode coupling. The framework is designed to complement standard prediction metrics by assessing whether the model accurately reproduces local mechanisms of the underlying PDE operator. The authors utilize various benchmarks to demonstrate the effectiveness of their approach, although specific datasets and training compute details are not disclosed.

Results
The spectral audit reveals distinct operator-level phenomena across benchmarks, such as phase transport, viscosity-dependent damping, nonlinear mode coupling, and reaction-diffusion stability structures. Notably, the audit identifies failures that are not apparent through traditional prediction-error metrics, including high-frequency degradation, incorrect phase recovery, and inconsistencies between prompts and operators. The authors report that even when pointwise predictions are partially accurate, corrupted or internally inconsistent prompts can lead to significant degradation in the tangent-operator structure. The results suggest that prediction accuracy and local operator fidelity are distinct properties, with implications for the robustness of learned neural operators.

Limitations
The authors acknowledge that their framework may not capture all aspects of operator fidelity and that the spectral audit is dependent on the quality of the prompts used. They also note that while the audit can reveal inconsistencies, it may not provide a complete picture of model performance in all scenarios. Additionally, the paper does not disclose specific datasets or the computational resources used for training, which limits reproducibility.

Why it matters
This work has significant implications for the evaluation of neural operators, suggesting that traditional metrics may be insufficient for assessing model fidelity in dynamical systems. By introducing a method to audit local operator behavior, the authors provide a valuable diagnostic tool for researchers working on neural operators and PDEs. This framework can enhance the understanding of model stability, sensitivity, and consistency, paving the way for more reliable applications in scientific computing and engineering. The findings are relevant for future research in operator learning and are available on arXiv.