Invariant Gradient Alignment for Robust Reasoning Distillation

Problem
Large language models (LLMs) exhibit a significant vulnerability to shortcut learning, particularly when faced with out-of-distribution (OOD) inputs that differ semantically from their training data, despite sharing identical logical structures. This issue poses a challenge for knowledge distillation processes aimed at transferring reasoning capabilities from larger models to smaller ones. The authors address this gap by proposing a new training framework, Invariant Gradient Alignment (IGA), which is particularly relevant as the work is presented as a preprint and has not undergone peer review.

Method
IGA introduces three key innovations to align gradient updates across semantically diverse yet logically isomorphic examples. First, it defines Logical Isomer Sets, which are collections of problems that maintain identical logical structures across various semantic domains, such as mathematics, medicine, law, and science. Second, it employs a differentiable Continuous Gradient Conflict Mask that selectively suppresses parameter dimensions exhibiting high cross-domain gradient variance while retaining invariant gradient directions. Third, it incorporates a truncated Singular Value Decomposition (SVD) projection of the masked gradient back onto the low-rank manifold of Low-Rank Adaptation (LoRA), ensuring parameter efficiency. The theoretical framework asserts that IGA provides tighter OOD generalization bounds compared to Empirical Risk Minimization (ERM) and converges at the standard Stochastic Gradient Descent (SGD) rate under mild regularity conditions.

Results
Empirical evaluations demonstrate that IGA significantly outperforms eight baseline methods across four distinct benchmarks. Notably, IGA achieves accuracy improvements of up to 14.3 percentage points over ERM-Supervised Fine-Tuning (ERM-SFT). Additionally, it records a Logical Consistency Score of 0.031, compared to 0.142 for the baseline, indicating a fourfold enhancement in representational invariance. These results underscore the effectiveness of IGA in improving the robustness of knowledge distillation processes in LLMs.

Limitations
The authors acknowledge that while IGA shows promise, it may require extensive computational resources due to the complexity of the gradient masking and SVD projection processes. Additionally, the reliance on the definition of Logical Isomer Sets may limit the applicability of IGA to domains where such structures can be clearly identified. The paper does not address potential scalability issues when applied to larger datasets or more complex models beyond those tested.

Why it matters
The introduction of IGA has significant implications for enhancing the robustness of LLMs in real-world applications, particularly in scenarios where OOD inputs are prevalent. By improving the transfer of reasoning capabilities through knowledge distillation, IGA could lead to more reliable AI systems across various domains. This work contributes to the ongoing discourse on mitigating shortcut learning in LLMs and sets a foundation for future research in robust reasoning and generalization techniques, as published in arXiv.