Long Context Pre-Training with Lighthouse Attention

Problem
This paper addresses the limitations of training causal transformers with long context sequences, specifically the computational bottleneck caused by the quadratic time and memory complexity of scaled dot-product attention (SDPA). The authors propose a novel approach, Lighthouse Attention, to mitigate these issues during the training phase. This work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution is the Lighthouse Attention mechanism, which introduces a hierarchical attention algorithm that operates in a training-only mode. The method consists of three main components:

1. Hierarchical Pre- and Post-Processing: This step employs a subquadratic approach for adaptive compression and decompression of input sequences, significantly reducing the computational burden during training.
2. Symmetrical Compression Strategy: The method pools queries, keys, and values simultaneously while maintaining left-to-right causality, enhancing parallelism and efficiency in processing.
3. Two-Stage Training Approach: The model is primarily pre-trained using Lighthouse Attention, followed by a brief recovery phase where a full attention model is restored. This strategy allows for a more efficient training process, leveraging the benefits of the hierarchical attention during the majority of the training time.

Results
The authors conducted preliminary experiments on small-scale language model (LLM) pre-training, comparing Lighthouse Attention against traditional full attention training under matched settings. The results indicate a faster total training time and a lower final loss after the recovery phase, although specific numerical results and benchmarks are not detailed in the abstract. The effectiveness of the proposed method suggests a significant improvement in training efficiency for long-context models.

Limitations
The authors acknowledge that their approach is primarily focused on the training phase and may not be directly applicable to inference without modifications. Additionally, the experiments are conducted on a small scale, which may limit the generalizability of the results. The paper does not address potential impacts on model performance metrics beyond loss, nor does it explore the implications of the hierarchical attention mechanism on model interpretability or robustness.

Why it matters
The introduction of Lighthouse Attention has significant implications for the development of transformer models capable of handling longer context sequences more efficiently. By reducing the computational overhead associated with SDPA, this work paves the way for more scalable training of large language models, potentially enabling advancements in applications requiring extensive context, such as document summarization and long-form content generation. The proposed method could inspire further research into hierarchical attention mechanisms and their integration into existing architectures.

Authors: Bowen Peng, Subho Ghosh, Jeffrey Quesnelle
Source: arXiv cs.CL
URL: https://arxiv.org/abs/2605.06554v1