Attention as Frustrated Synchronization

Problem — This work addresses the limitations of existing attention mechanisms that rely on perfect synchronization, which do not facilitate meaningful computation. The authors propose a new architecture, the Frustrated Synchronization Network (FSN), to exploit structured deviations from synchronization to enhance computational capabilities. This paper is a preprint and has not undergone peer review.

Method — The FSN architecture is built on the principles of frustrated synchronization, where token states are represented as phases on a torus. The core technical contribution is a learned complex coupling kernel that incorporates harmonics and a one-step delay. The kernel consists of static Kuramoto-Sakaguchi frustration angles, signed harmonics as repulsive Daido components, and a delay term that couples tokens to their successors. This design allows the FSN to implement next-token prediction as a synchronization process that is frustrated by the data’s transitions. The model is trained with a parameter budget of one million, matching the training compute of a tuned RoPE-SwiGLU transformer for character-level text and code.

Results — The FSN consistently outperforms the baseline transformer across all epochs measured, achieving a validation loss of 1.5953 ± 0.0014 compared to the transformer’s converged loss of 1.611 after fifty epochs. Notably, the FSN’s performance is superior at every thirty-epoch checkpoint during training. Additionally, a variant of the FSN, which replaces every feed-forward block with mean-field coupling to learned collective modes, maintains competitive performance with the transformer. On natural text tasks, the unfrustrated base layer of the FSN lags behind the transformer, particularly in long-range copy events, but the kernel effectively mitigates this deficit at depths of four and beyond. The results are consistent across a scale ladder up to four million parameters, with ongoing work to explore further scaling.

Limitations — The authors acknowledge that the FSN’s performance is evaluated primarily at the one-million-parameter scale, with further investigations needed for larger models. Additionally, while the FSN shows promise in specific tasks, its generalizability across diverse datasets and tasks remains to be fully assessed. The paper does not address potential computational overhead introduced by the complex coupling kernel.

Why it matters — The introduction of the FSN provides a novel perspective on attention mechanisms by leveraging synchronization principles, potentially leading to more efficient architectures for sequence modeling tasks. This work opens avenues for future research into synchronization-based models and their applications in natural language processing and beyond, as discussed in related literature on synchronization in neural networks, such as in arXiv cs.NE.