A Spectral Theory of Grokking: Weight Decay induces Feature Learning

Problem

The paper addresses the phenomenon of delayed generalization in neural networks during training, a gap in understanding how certain training dynamics affect model performance. This work is particularly relevant as it investigates the spectral properties of neural networks, which have not been thoroughly explored in the context of weight decay and feature learning. The authors present their findings in a preprint format, indicating that the work is unreviewed.

Method

The authors utilize homogeneous networks, specifically a multi-layer perceptron (MLP) and a one-block Transformer architecture, to study the effects of weight decay on generalization. The loss function employed is squared loss, and L2 weight decay is applied to the models. The data used for training consists of a modular addition task, which serves as a benchmark for evaluating the networks' performance. The training compute involved an extensive setup, with an 84x90 grid of trained networks for the MLP and a 42x45 grid for the Transformer. The dynamics of the networks are analyzed through the lens of Neural Tangent Kernel (NTK) dynamics, projected onto task-relevant spectral directions, allowing for a deeper understanding of how weight decay influences feature learning.

Results

The paper reports that generalization time scales inversely with the product of learning rate and weight decay, aligning with the predicted phase geometry. Additionally, the emergence of feature learning is observed, characterized by a task-aligned Fourier structure that persists even after the saturation of training accuracy. However, the available text does not report quantitative results, limiting the ability to compare these findings against specific baselines or benchmarks.

Limitations

The authors note that while weight decay can facilitate feature learning, excessively strong weight decay may hinder the model's ability to fit the training data altogether. There exists a critical threshold of weight decay beyond which the task-aligned NTK structure fails to support generalization, indicating a delicate balance that must be maintained during training.

Why it matters

This research has significant implications for the design and training of neural networks, particularly in understanding how regularization techniques like weight decay can influence feature learning and generalization. By elucidating the spectral dynamics at play, this work paves the way for future investigations into optimizing training strategies and improving model performance in various tasks.