Solve the Loop: Attractor Models for Language and Reasoning

Problem
This preprint addresses the limitations of recurrent architectures in language modeling and reasoning tasks, specifically their instability during training, high optimization costs, and fixed recurrence depths. The authors propose a novel framework, Attractor Models, to overcome these challenges by enabling adaptive iterations and stable training dynamics.

Method
The core technical contribution is the Attractor Model architecture, which consists of two main components: a backbone module and an attractor module. The backbone generates initial output embeddings, while the attractor refines these embeddings by solving for a fixed point using implicit differentiation to obtain gradients. This approach allows the model to maintain a constant effective depth during training, with the number of iterations determined adaptively based on convergence criteria. The authors leverage large-scale language model pretraining and reasoning tasks to evaluate the performance of their model.

Results
Attractor Models demonstrate significant improvements over standard Transformers and existing looped models. In language modeling, they achieve a perplexity reduction of up to 46.6% and a downstream accuracy increase of up to 19.7%, while also reducing training costs. Notably, a 770M parameter Attractor Model outperforms a 1.3B parameter Transformer trained on double the token count. In reasoning tasks, the model with only 27M parameters achieves 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard, outperforming larger frontier models like Claude and GPT-3, which fail on these tasks. The authors also introduce the concept of equilibrium internalization, where fixed-point training positions the model’s initial output embedding near equilibrium, allowing for the removal of the solver during inference with minimal performance degradation.

Limitations
The authors acknowledge that while Attractor Models show promise, they may still face challenges related to scalability in extremely large models and the potential for overfitting in small datasets. Additionally, the paper does not extensively discuss the computational overhead introduced by the implicit differentiation process, which could impact training efficiency in practice. The generalizability of the results across diverse datasets and tasks remains to be fully explored.

Why it matters
The introduction of Attractor Models represents a significant advancement in the design of iterative refinement mechanisms for language and reasoning tasks. By enabling models to learn adaptive recurrence, this approach could lead to more efficient architectures that maintain high performance with fewer parameters. The findings suggest potential pathways for future research in scalable model design, particularly in contexts where computational resources are limited or where rapid inference is critical. The equilibrium internalization phenomenon may also inspire new methodologies for model deployment and inference optimization.

Authors: Jacob Fein-Ashley, Paria Rashidinejad
Source: arXiv:2605.12466
URL: https://arxiv.org/abs/2605.12466v1