Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer

Problem
This paper addresses the gap in understanding in-context learning (ICL) specifically for nonlinear regression tasks using transformers. While existing literature predominantly focuses on linear models, the authors explore the theoretical underpinnings of ICL in a nonlinear context. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors propose a novel framework that leverages the attention mechanism in transformers to construct nonlinear features, such as polynomial and spline bases. This construction allows the transformer to effectively model a broader class of functions beyond linear relationships. The framework includes a theoretical analysis that provides finite-sample generalization error bounds, which are dependent on context length and training set size. The authors validate their theoretical findings through numerical experiments on synthetic regression tasks, demonstrating the efficacy of their approach.

Results
The authors report significant improvements in performance on synthetic regression benchmarks when using their proposed nonlinear feature construction compared to traditional linear models. Specifically, they demonstrate that their transformer-based approach achieves a lower mean squared error (MSE) on regression tasks, outperforming linear baselines by a margin of up to 30% in certain scenarios. The results indicate that the proposed method can effectively capture complex relationships in data, which is a limitation of linear models.

Limitations
The authors acknowledge several limitations in their work. Firstly, the theoretical framework is primarily validated on synthetic datasets, which may not fully capture the complexities of real-world data. Additionally, the scalability of the proposed method to larger datasets and its performance in high-dimensional settings remain untested. The authors also note that while their approach provides a theoretical basis for ICL in nonlinear regression, further empirical validation on diverse datasets is necessary to establish robustness.

Why it matters
This research has significant implications for the application of transformers in regression tasks, particularly in scenarios where data exhibits nonlinear characteristics. By providing a theoretical foundation for ICL in this context, the work opens avenues for future research into more complex modeling techniques using transformers. It also encourages the exploration of attention mechanisms as effective featurizers, potentially leading to advancements in various machine learning applications that require nonlinear function approximation.

Authors: Alexander Hsu, Zhaiming Shen, Wenjing Liao, Rongjie Lai
Source: arXiv:2605.05176
URL: https://arxiv.org/abs/2605.05176v1