Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders

Problem
This preprint addresses the gap in post-training data engineering for large language models (LLMs), which traditionally relies on external signals while neglecting the rich intrinsic information encoded in model internals. The authors highlight that existing methods do not leverage the insights that can be derived from model internals, particularly in the context of reinforcement learning (RL) for data selection and optimization.

Method
The authors introduce SAERL, a novel data engineering framework that utilizes Sparse Autoencoders (SAE) to extract intrinsic properties of training data: diversity, difficulty, and quality. The framework employs the following techniques:

1. Diversity: SAE-space clustering is used to enhance batch diversity through moderate batch mixing, ensuring that training batches contain a varied set of examples.
2. Difficulty: A difficulty proxy is established to facilitate a curriculum learning approach, ordering data from easy to hard to optimize learning efficiency.
3. Quality: A quality probe is implemented for effective data filtering, allowing the model to focus on high-quality training examples.
   The authors report that SAERL achieves an average accuracy improvement of 3.00% over the baseline method, GRPO, while also reducing the number of training steps by 20% on the Qwen2.5-Math-1.5B benchmark. The framework is designed to be lightweight and reusable across different model families and scales.

Results
SAERL demonstrates significant performance improvements on the Qwen2.5-Math-1.5B benchmark, achieving a 3.00% increase in average accuracy compared to the vanilla GRPO baseline. Additionally, it reaches target accuracy with 20% fewer training steps, indicating enhanced training efficiency. The results are consistent across various model scales and RL algorithms, showcasing the robustness of the SAE-derived signals in guiding data engineering processes.

Limitations
The authors acknowledge that while SAERL effectively utilizes model internals, its reliance on SAE may introduce limitations in interpretability and generalization across all types of LLMs. They do not address potential computational overhead associated with the initial SAE training phase or the scalability of the framework in extremely large datasets. Furthermore, the framework’s performance in highly specialized domains or with non-standard data distributions remains untested.

Why it matters
The implications of this work are significant for the field of machine learning, particularly in enhancing the efficiency of LLM training through improved data engineering practices. By demonstrating that model internals can serve as a valuable source of signals, this research opens avenues for more informed data selection strategies, potentially leading to faster convergence and better generalization in LLMs. The framework’s lightweight nature suggests it could be integrated into existing training pipelines with minimal disruption, promoting broader adoption of intrinsic signal utilization in data engineering.

Authors: Yi Jing, Zao Dai, Jinwu Hu, Zijun Yao, Lei Hou, Juanzi Li, Xiaozhi Wang
Source: arXiv:2605.27354
URL: https://arxiv.org/abs/2605.27354v1