Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization

Problem
This work addresses the gap in understanding how different transformer modules may benefit from distinct weight-space geometries during optimization. The authors highlight that existing literature typically applies uniform manifold constraints across all weight matrices, which may not be optimal. This research is presented as a preprint and has not undergone peer review.

Method
The authors introduce a novel approach termed Manifold Muon, applied to the pretraining of the GPT-2 model. They explore the effects of assigning different geometric constraints—specifically Stiefel and DGram geometries—on the attention and MLP (multi-layer perceptron) blocks of the transformer architecture. The study involves a systematic comparison of layer-wise assignments, where attention layers are constrained with Stiefel geometry while MLP layers are assigned DGram geometry. The training process utilizes a shared hyperparameter setting across configurations to ensure consistency in evaluation.

Results
The findings indicate a clear performance asymmetry based on the geometric constraints applied to different modules. The configuration that constrains attention layers with Stiefel geometry and MLP layers with DGram geometry outperforms other tested configurations, achieving significant stability and performance improvements. In contrast, the inverted assignment (DGram for attention and Stiefel for MLP) and an all-DGram configuration led to instability during training, attributed to singular value growth in DGram-constrained attention weights. This growth can amplify attention logits, resulting in softmax saturation and degraded model performance. The paper provides quantitative results, demonstrating that the optimal configuration yields a marked improvement over baseline models, although specific numerical performance metrics are not disclosed.

Limitations
The authors acknowledge that their findings are limited to the specific configurations tested and the GPT-2 architecture. They do not explore the implications of their results on other transformer variants or architectures, nor do they address the scalability of their approach to larger models or different tasks. Additionally, the reliance on a shared hyperparameter setting may not generalize across diverse training scenarios.

Why it matters
This research underscores the importance of module-specific optimization strategies in transformer architectures, suggesting that a one-size-fits-all approach to weight-space geometry may hinder performance. The implications of these findings could lead to more effective training methodologies for transformers, particularly in applications requiring high stability and performance. Future work may build on these insights to explore geometry-aware optimization across various architectures and tasks, as published in arXiv cs.LG.