Language Models Need Sleep

Problem
This preprint addresses the limitations of transformer-based large language models (LLMs) in handling long-context tasks due to the quadratic scaling of their attention mechanism with respect to context length. Existing models struggle with memory efficiency and computational overhead when processing extensive input sequences. The authors propose a novel sleep-like consolidation mechanism to mitigate these issues, which has not been extensively explored in the literature.

Method
The core technical contribution is the introduction of a sleep mechanism that allows the model to convert recent context into persistent fast weights, effectively reducing the need for extensive key-value cache during inference. The model performs $N$ offline recurrent passes over the accumulated context during the sleep phase, updating the fast weights in its state-space model (SSM) blocks using a learned local rule. This approach shifts computational load from the wake phase to the sleep phase, maintaining low latency during prediction. The architecture integrates SSM blocks with traditional transformer components, enhancing the model’s ability to manage long-term dependencies without incurring the typical computational costs associated with large context lengths.

Results
The proposed method was evaluated on several controlled synthetic tasks, including cellular automata and multi-hop graph retrieval, as well as a realistic math reasoning task. The results indicate significant performance improvements over baseline models, including standard transformers and SSM-attention hybrids. Specifically, the authors report that increasing the sleep duration $N$ correlates with enhanced performance, particularly on tasks requiring deeper reasoning. For instance, the model achieved a 15% improvement in accuracy on the math reasoning task compared to the best-performing baseline, demonstrating the effectiveness of the sleep mechanism in consolidating information.

Limitations
The authors acknowledge several limitations, including the dependency on the choice of the local update rule for fast weights, which may not generalize across all tasks. Additionally, the computational overhead during the sleep phase, while reduced, still requires careful tuning of $N$ to balance performance and efficiency. The study primarily focuses on synthetic tasks, which may not fully capture the complexities of real-world applications. Furthermore, the scalability of the proposed method to even larger models or more diverse tasks remains untested.

Why it matters
This work has significant implications for the design of future LLMs, particularly in applications requiring long-context processing and reasoning. By introducing a mechanism that allows for efficient memory management and computational load balancing, the authors pave the way for more scalable and effective models. The sleep mechanism could inspire further research into hybrid architectures that leverage both traditional attention and state-space models, potentially leading to breakthroughs in tasks that demand extensive contextual understanding.

Authors: Sangyun Lee, Sean McLeish, Tom Goldstein, Giulia Fanti
Source: arXiv:2605.26099
URL: https://arxiv.org/abs/2605.26099v1