Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection

Problem
The paper addresses the gap in safety alignment for large language models (LLMs) during the pretraining phase, emphasizing that existing methods primarily focus on filtering or rewriting unsafe data. The authors argue that these approaches are insufficient, as LLMs can still generate unsafe behaviors by combining benign knowledge. This work is a preprint and has not undergone peer review.

Method
The authors propose a novel method called Safety Reflection Pretraining, which involves the regular insertion of short safety reflections into the pretraining corpus. This technique aims to instill self-monitoring capabilities directly into the language model during pretraining. The experiments utilize a 1.7 billion parameter model pretrained on the FineWeb-Edu dataset. The training process incorporates these safety reflections, which are designed to reinforce safe behavior patterns that can be further enhanced during post-training. The paper also introduces a synthetic environment, MedSafetyWorld, to evaluate the model’s ability to generalize unsafe behaviors from safe data, providing a controlled setting for rigorous testing.

Results
The results demonstrate that Safety Reflection Pretraining significantly improves safety classification accuracy compared to baseline models that rely solely on data filtering and rewriting. Specifically, the model exhibits a marked reduction in the success rates of inference-stage and finetuning attacks. In MedSafetyWorld, ablation studies reveal that models trained with Safety Reflection Pretraining outperform those trained with traditional safety data methods, effectively preventing the generalization of unsafe behaviors from safe data. The exact performance metrics are not disclosed in the abstract, but the qualitative improvements suggest a substantial enhancement in safety alignment.

Limitations
The authors acknowledge that while Safety Reflection Pretraining shows promise, it may not fully eliminate all unsafe behaviors, particularly in complex scenarios not covered by the synthetic environment. Additionally, the reliance on a specific dataset (FineWeb-Edu) may limit the generalizability of the findings. The paper does not address potential computational costs associated with the additional training complexity introduced by safety reflections.

Why it matters
This research has significant implications for the development of safer LLMs, suggesting that pretraining strategies should incorporate mechanisms for behavioral alignment rather than solely focusing on data safety. The findings advocate for a paradigm shift in how safety is approached in LLM training, emphasizing the need for proactive behavior shaping. This work contributes to the ongoing discourse on AI safety and aligns with broader efforts to enhance the reliability of AI systems, as published in arXiv cs.AI.