Aligned Training: A Parameter-Free Method to Improve Feature Quality and Stability of Sparse Autoencoders (SAE)

Problem
This paper addresses the limitations of Sparse Autoencoders (SAEs) in terms of feature activation and stability. Despite their utility in interpreting deep neural networks, SAEs often suffer from a significant number of inactive features and instability across training runs. Existing methods to enhance SAEs typically require additional data, resampling, or hyperparameter tuning, which can complicate their application. The authors propose a novel approach called “aligned training,” which is a parameter-free method aimed at improving the quality and stability of features in SAEs. This work is presented as a preprint and has not yet undergone peer review.

Method
The core contribution of this paper is the introduction of aligned training, which reparameterizes SAEs to enforce a geometric constraint between the encoder and decoder. Specifically, the method aims to ensure that the inner product of the encoder and decoder directions equals one for each feature, thereby addressing the degeneracy in SAE training. The authors define a metric called the “alignment score,” which quantifies feature quality based on this inner product. The training process does not introduce any additional hyperparameters, making it computationally efficient. The authors validate their approach across various models, dictionary sizes, and sparsity levels, demonstrating its versatility and effectiveness.

Results
The aligned training method yields significant improvements on the SAEBench benchmarks, achieving Pareto improvements in reconstruction quality, feature activation, and stability across different training seeds. While specific numerical results are not detailed in the abstract, the authors claim that their method outperforms existing SAE variants that require additional data or hyperparameter tuning. The improvements are characterized by a reduction in dead features and enhanced stability, indicating a robust enhancement in feature quality.

Limitations
The authors acknowledge that while aligned training improves feature quality and stability, it may not address all potential issues inherent to SAEs, such as the interpretability of the learned features in certain contexts. Additionally, the method’s performance across all possible architectures and datasets has not been exhaustively tested, which could limit its generalizability. The lack of hyperparameter tuning, while a strength, may also restrict the method’s adaptability to specific use cases where fine-tuning is beneficial.

Why it matters
The implications of this work are significant for the field of machine learning, particularly in the context of interpretability and feature extraction from deep neural networks. By providing a parameter-free method to enhance the performance of SAEs, the authors contribute to the ongoing discourse on improving model interpretability without incurring additional computational costs. The integration of aligned training with existing techniques in mechanical interpretability, such as Top/BatchTop-K architectures and p-Annealing, suggests that this approach could facilitate more effective and interpretable models in various applications, paving the way for future research in feature quality enhancement.

Authors: Michał Brzozowski, Neo Christopher Chung
Source: arXiv:2605.18629
URL: https://arxiv.org/abs/2605.18629v1