Geometric Factual Recall in Transformers

Problem
This preprint addresses the gap in understanding how transformer language models memorize factual associations, challenging the prevailing view that internal weight matrices function as associative memories requiring linear scaling with the number of facts. The authors propose a geometric framework for memorization, which has not been thoroughly explored in existing literature.

Method
The authors introduce a theoretical and empirical framework for geometric memorization in transformers. They focus on a single-layer transformer tasked with memorizing random bijections from subjects to a shared attribute set. The key contribution is the demonstration that a logarithmic embedding dimension is sufficient for this task. Subject embeddings are shown to encode linear superpositions of associated attribute vectors, while a small multi-layer perceptron (MLP) serves as a relation-conditioned selector, utilizing ReLU gating to extract relevant attributes rather than functioning as an associative key-value mapping. The authors extend their findings to multi-hop relational queries, providing constructions that exhibit a capacity-depth tradeoff, supported by a matching information-theoretic lower bound. Empirical results indicate that gradient descent effectively discovers the predicted geometric structure, and the MLP demonstrates zero-shot transfer capabilities to new bijections when subject embeddings are re-initialized.

Results
The authors report that their model achieves effective memorization with a logarithmic embedding dimension, significantly reducing the parameter count compared to traditional associative memory approaches. In controlled experiments, the model successfully handles multi-hop queries, demonstrating a capacity-depth tradeoff that aligns with theoretical predictions. The zero-shot transfer capability of the MLP indicates that it has learned a generic selection mechanism, rather than memorizing specific facts. While specific numerical results and comparisons to named baselines are not detailed in the abstract, the implications of the findings suggest substantial improvements in efficiency and generalization over conventional methods.

Limitations
The authors acknowledge that their work is primarily theoretical and empirical, and while they demonstrate the effectiveness of their geometric approach, they do not extensively evaluate performance against a wide range of existing models or benchmarks. Additionally, the scalability of their method to larger, more complex datasets and real-world applications remains untested. The reliance on a single-layer transformer may also limit the generalizability of their findings to deeper architectures commonly used in practice.

Why it matters
This work has significant implications for the design and understanding of transformer architectures, particularly in how they can efficiently encode and retrieve factual information. By shifting the focus from associative memory to geometric memorization, the findings could lead to more efficient models that require fewer parameters while maintaining or improving performance on factual recall tasks. This could influence future research on model architectures, training methodologies, and applications in natural language processing where factual accuracy is critical.

Authors: Shauli Ravfogel, Gilad Yehudai, Joan Bruna, Alberto Bietti
Source: arXiv:2605.12426
URL: https://arxiv.org/abs/2605.12426v1