Variable-Width Transformers

Problem — This work addresses the limitation of existing transformer architectures that utilize a uniform width across all layers, which may not effectively leverage the distinct computational roles of different layers. The authors highlight that this is an unreviewed preprint, indicating a gap in the literature regarding nonuniform capacity allocation in transformer models.

Method — The authors propose a novel architecture termed the $\times$-shaped > <former, which features a nonuniform width distribution across its layers. Specifically, the architecture maintains wider layers at the beginning and end of the model while narrowing the middle layers. This design is complemented by a parameter-free residual resizing mechanism that adjusts the residual connections to accommodate the varying layer widths. The experiments are conducted on decoder-only language models with parameter counts ranging from 200M to 2B (dense) and up to 3B (Mixture of Experts, MoE). The training process involves standard language modeling tasks, and the authors report a reduction in average layer width, leading to fewer overall floating-point operations (FLOPs) and reduced memory costs.

Results — The > <former architecture consistently outperforms parameter-matched uniform baselines on language modeling loss across all tested configurations. Notably, the model achieves a 22% reduction in FLOPs under fitted loss-matched scaling curves and a 15% decrease in key-value (KV) cache memory and I/O costs. These results indicate that the nonuniform width allocation not only enhances performance but also improves resource efficiency, making it a compelling alternative to traditional transformer designs.

Limitations — The authors acknowledge that their approach may introduce complexity in model design and training due to the nonstandard layer widths. Additionally, they do not explore the impact of this architecture on other tasks beyond language modeling, which could limit the generalizability of their findings. The paper does not provide extensive ablation studies to dissect the contributions of the residual resizing mechanism, which could further clarify the architecture’s advantages.

Why it matters — The implications of this research are significant for the future design of transformer architectures, particularly in resource-constrained environments. By demonstrating that nonuniform width allocation can lead to more efficient scaling, this work opens avenues for further exploration of adaptive architectures in NLP and beyond. The findings suggest that future models could benefit from tailored layer designs that optimize both performance and computational efficiency, as published in arXiv cs.CL.