Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization

Problem
This paper addresses a significant gap in the theoretical understanding of weight decay as a regularization technique in Transformer models, specifically its impact on the loss landscape. While weight decay is commonly employed in large language models, its precise role and the underlying mathematical properties have not been rigorously characterized. This work presents the first functional-analytic framework for the standard Transformer objective, which is cross-entropy loss augmented with (L^2) regularization. The authors provide a theoretical foundation that satisfies Villani’s criteria for coercive energy functions, which is crucial for understanding optimization dynamics and generalization in deep learning. This is a preprint and has not yet undergone peer review.

Method
The authors establish that the regularized loss function (\mathcal{F}) is infinitely differentiable and exhibits quadratic growth, with Gaussian-integrable tails. They prove that it satisfies the differential growth condition (-\Delta\mathcal{F} + \frac{1}{s}|\nabla\mathcal{F}|^{2} \to \infty) as (|\theta| \to \infty) for all (s > 0). From this characterization, they derive explicit log-Sobolev and Poincaré constants, linking the regularization strength (\lambda) and model dimension (d) to finite-time convergence guarantees for noisy stochastic gradient descent (SGD) and PAC-Bayesian generalization bounds that improve with increasing (\lambda). To validate their theoretical findings, the authors introduce a scalable diagnostic (Ψ_s(θ) = -\Delta\mathcal{F} + s^{-1}|\nabla\mathcal{F}|^2) and utilize Hutchinson trace probes for efficient estimation in large models (over 100M parameters).

Results
The empirical validation is conducted on the GPT-Neo-125M model using the Penn Treebank and WikiText-103 datasets. The experiments confirm the predicted quadratic growth of the diagnostic (Ψ_s), alongside spectral inflation of the Hessian matrix. The convergence behavior observed aligns with the theoretical predictions from the log-Sobolev analysis, demonstrating exponential convergence rates. The authors report that weight decay not only enhances generalization performance but also establishes the necessary mathematical conditions for accelerated Langevin mixing and curvature-aware optimization.

Limitations
The authors acknowledge that their theoretical framework is based on specific assumptions regarding the loss landscape and regularization strength, which may not hold universally across all model architectures or datasets. Additionally, while the empirical results are promising, they are limited to a single model (GPT-Neo-125M) and specific datasets, which may restrict the generalizability of the findings. The scalability of the proposed diagnostic in even larger models or different architectures remains to be explored.

Why it matters
This work has significant implications for the optimization and generalization of deep learning models, particularly in the context of large language models. By providing a rigorous theoretical foundation for weight decay, the authors pave the way for more informed choices in regularization strategies and optimization techniques. The established link between regularization strength and convergence guarantees could lead to improved training methodologies and better performance in practical applications.

Authors: Abhijit Das, Sayantan Dutta
Source: arXiv:2605.06599
URL: https://arxiv.org/abs/2605.06599v1