Transformers with Selective Access to Early Representations

Problem
This paper addresses the limitations of existing Transformer architectures that utilize static value residuals to access early representations. While these methods provide a uniform approach to integrating low-level features, they fail to account for the varying needs of different tokens, heads, and contexts. The authors propose a more dynamic solution, treating early-representation reuse as a retrieval problem rather than a mere connectivity issue. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors introduce the Selective Access Transformer (SATFormer), which enhances the traditional Transformer architecture by incorporating a context-dependent gating mechanism that controls access to early representations. SATFormer retains the first-layer value pathway while allowing for selective reuse of early lexical and semantic information. The architecture is evaluated across models ranging from 130M to 1.3B parameters. The training process involves standard optimization techniques, although specific details regarding the loss function and training compute are not disclosed. The key innovation lies in the dynamic gating mechanism that adapts access based on the context, leading to more efficient utilization of early representations.

Results
SATFormer demonstrates significant improvements over both static value-residual and baseline Transformer models. On retrieval-intensive benchmarks, SATFormer achieves an average improvement of approximately 1.5 points in zero-shot accuracy compared to static value residuals. The validation loss consistently decreases across all model sizes, indicating enhanced performance. Notably, SATFormer maintains throughput and memory usage comparable to baseline Transformers, suggesting that the added complexity of the gating mechanism does not incur substantial overhead.

Limitations
The authors acknowledge that while SATFormer improves access to early representations, it may still be limited by the inherent constraints of Transformer architectures, such as the quadratic scaling of attention mechanisms with respect to input length. Additionally, the paper does not explore the impact of varying the gating mechanism’s complexity or the potential trade-offs between model size and performance. The generalizability of the findings across different tasks and datasets remains to be fully validated.

Why it matters
The introduction of SATFormer has significant implications for future Transformer-based models, particularly in tasks that require nuanced understanding of early representations, such as natural language processing and computer vision. By framing early-representation reuse as a retrieval problem, this work opens avenues for more adaptive architectures that can better leverage low-level features. The findings suggest that context-sensitive access can lead to more efficient and effective models, potentially influencing the design of future architectures and training methodologies.

Authors: Skye Gunasekaran, Téa Wright, Rui-Jie Zhu, Jason Eshraghian
Source: arXiv:2605.03953
URL: https://arxiv.org/abs/2605.03953v1