The Double Dilemma in Multi-Task Radiology Report Generation: A Gradient Dynamics Analysis and Solution

Problem
This preprint addresses the limitations of existing multi-task learning (MTL) approaches in automatic radiology report generation (RRG), which primarily focus on architectural innovations while relying on coarse linear scalarization strategies. These strategies fail to effectively balance the hard constraints imposed by clinical supervision with the smoothness requirements necessary for coherent report generation. The authors identify this issue as a “Double Dilemma” characterized by gradient dynamics, specifically the drift term deviation and diffusion term decay, which hinder optimal performance in MTL settings.

Method
The authors propose a novel optimizer called Conflict-Averse Magnitude-Enhanced Gradient Descent (CAME-Grad), which is designed to be backbone-agnostic. CAME-Grad incorporates two main components: conflict-averse direction rectification and magnitude-enhanced energy injection. The former addresses the geometric validity of the optimization process, while the latter helps to avoid convergence to local optima. Additionally, an adaptive gradient fusion mechanism is introduced to dynamically balance the theoretical optimal direction with task-specific inductive biases. The training compute details are not disclosed, but the method is presented as a plug-and-play solution applicable to various RRG architectures.

Results
CAME-Grad was evaluated across eight distinct RRG methods on two benchmark datasets: MIMIC-CXR and IU X-Ray. The results indicate a significant improvement in clinical efficacy performance, with an average increase of 2.3% on MIMIC-CXR and 1.9% on IU X-Ray compared to baseline methods. These improvements are substantial, suggesting that the proposed optimizer effectively enhances the performance of existing RRG systems.

Limitations
The authors acknowledge that while CAME-Grad shows promise, it may not universally outperform all existing optimizers in every scenario, particularly in highly specialized tasks outside the scope of the evaluated benchmarks. Additionally, the paper does not provide extensive ablation studies to dissect the contributions of each component of CAME-Grad, which could limit understanding of its effectiveness in various contexts. The lack of detailed training compute specifications also raises questions about the practical deployment of the method in resource-constrained environments.

Why it matters
The findings of this research have significant implications for the field of medical AI, particularly in enhancing the reliability and quality of automated radiology report generation. By addressing the gradient dynamics issues inherent in multi-task learning, CAME-Grad offers a robust solution that can be integrated into existing systems to improve clinical outcomes. This work paves the way for further exploration of advanced optimization techniques in MTL settings, potentially leading to more effective AI applications in healthcare.

Authors: Erjian Zhang, Yatong Hao, Liejun Wang, Zhiqing Guo
Source: arXiv:2605.22635
URL: https://arxiv.org/abs/2605.22635v1