Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging

Problem
This paper addresses a significant gap in the Learning to Defer (L2D) paradigm, specifically in the context of hierarchical multi-label decision-making in medical imaging. Prior research has predominantly focused on flat label spaces, neglecting the complexities introduced by hierarchical structures where findings are organized according to clinical taxonomies. The authors highlight that treating deferral as an independent decision for each label can lead to incoherence, such as taxonomic contradictions and delegation violations. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors formalize a coherent hierarchical deferral framework under a Selective-Exclusion handoff contract. They characterize the Bayes-optimal coherent deferral rule and demonstrate that even nodewise Bayes L2D can result in action incoherence. To address this, they propose two main remedies:

1. Exact Coherent Projection: This method employs a dynamic programming decoder that operates over a coherent action set, ensuring that deferrals are consistent with the hierarchical structure.

2. Taxonomic Belief Propagation (TBP) with Recursive Policy Optimization (RPO): This approach integrates a contract-aware joint action model that is trained using the same recursive methodology applied during inference. This model aims to minimize incoherence while maximizing utility.

The training compute details are not explicitly disclosed, but the methods leverage recursive optimization techniques.

Results
The proposed methods were evaluated against naive binary-relevance L2D on both real-reader and controlled-expert medical imaging benchmarks. The results indicate that naive L2D exhibits significant incoherence, which is quantitatively assessed. The exact coherent projection method eliminates incoherence entirely, while the TBP+RPO approach reduces incoherence to near-zero levels while maintaining strong utility. Specific effect sizes and performance metrics are not detailed in the abstract, but the improvements are described as substantial compared to the baseline.

Limitations
The authors acknowledge that their methods may not generalize to all hierarchical structures and that the computational complexity of the dynamic programming approach could be a limiting factor in large-scale applications. Additionally, the reliance on a specific contract framework may restrict the applicability of their findings to other domains outside medical imaging. The paper does not address potential biases in the training data or the implications of model interpretability in clinical settings.

Why it matters
This work has significant implications for the deployment of AI systems in medical imaging, where accurate and coherent decision-making is critical. By formalizing a coherent approach to hierarchical multi-label learning, the authors provide a framework that can enhance the reliability of AI-assisted diagnoses and reduce the cognitive load on medical professionals. The methodologies introduced could pave the way for more sophisticated AI systems that can navigate complex clinical taxonomies, ultimately improving patient outcomes and operational efficiency in healthcare settings.

Authors: Joshua Strong, Pramit Saha, Emma Sun, Helen Higham, Alison Noble
Source: arXiv:2605.02734
URL: https://arxiv.org/abs/2605.02734v1