A Single-Layer Model Can Do Language Modeling

Problem
This preprint addresses the gap in understanding the efficacy of single-layer architectures for language modeling, contrasting with the prevalent multi-layered approaches in contemporary models. The authors investigate whether a single-layer recurrent model can achieve competitive performance in language tasks, challenging the conventional wisdom that deeper architectures are necessary for effective representation learning.

Method
The authors introduce Grounded Prediction Networks (GPN), which utilize a single recurrent block with a shared matrix memory and a feedforward network (FFN). The architecture is designed to revisit a single state vector at each time step, diverging from the multi-layered KV cache approach typical in transformers and other deep models. The GPN is evaluated at 130 million parameters, with a focus on its ability to maintain a persistent state vector that captures both content and context. The training methodology and specific loss functions are not detailed in the abstract, but the model’s performance is benchmarked against established architectures, including a 12-layer Transformer++ and a 10-layer GDN.

Results
The GPN+M achieves a perplexity of 18.06 on the FineWeb-Edu dataset, which is within 13% of the 12-layer Transformer++ (16.05 perplexity) and 18% of the 10-layer GDN (15.34 perplexity). A 2-layer variant of GPN further narrows the performance gap to 6% and 11%, respectively. These results indicate that while the single-layer model does not surpass the deep baselines, it performs competitively, suggesting that depth may not be the sole determinant of language modeling efficacy.

Limitations
The authors acknowledge that their single-layer GPN does not match the performance of deeper models, indicating a limitation in capturing complex dependencies that may require additional layers. They also do not explore the scalability of the model beyond two layers or its performance on a broader range of datasets. Additionally, the abstract does not provide details on the training compute or specific hyperparameters used, which could be critical for replication and further exploration.

Why it matters
This work has significant implications for the design of future language models, suggesting that single-layer architectures can be viable alternatives to deep models, particularly in resource-constrained environments. The findings challenge the assumption that increased depth is necessary for effective language modeling, potentially leading to more efficient architectures that leverage recurrent mechanisms. The insights into the geometry of the state vector and its retention dynamics may inform future research on memory mechanisms in neural networks, paving the way for novel approaches that balance complexity and performance.

Authors: Zanmin Wang
Source: arXiv:2605.10643
URL: https://arxiv.org/abs/2605.10643v1