A Domain Incremental Continual Learning Benchmark for ICU Time Series Model Transportability

Problem
This paper addresses the gap in the literature regarding the transferability of machine learning models trained in one clinical setting to different hospitals, particularly smaller institutions with limited resources. The authors highlight the challenge of generalizing models trained on data from a single hospital to diverse patient populations across various regions in the United States. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors propose a novel benchmark for evaluating the transportability of machine learning models in the context of domain incremental learning. They frame the problem as one where a model must adapt to new data distributions while retaining knowledge from the original training domain. The benchmark assesses the model’s ability to learn from new domains while preserving critical features from the source domain. Two established domain incremental learning techniques are evaluated: (1) Data Replay, which involves storing and reusing examples from previous domains to fine-tune the model on new data, and (2) Elastic Weight Consolidation (EWC), a regularization method that constrains the model’s parameter updates to maintain performance on previously learned tasks. The authors do not disclose specific details regarding the architecture, loss functions, or computational resources used in their experiments.

Results
The benchmark results demonstrate that both Data Replay and EWC significantly improve model performance when transferring from one region to another compared to a baseline model that does not utilize these techniques. The authors report that models employing Data Replay achieve a performance increase of approximately 15% in accuracy on the benchmark tasks, while EWC shows a 10% improvement over the baseline. These results indicate that both methods effectively mitigate the challenges posed by domain shifts in clinical data.

Limitations
The authors acknowledge several limitations in their study. First, the benchmark is constrained to specific regions in the United States, which may not fully capture the diversity of patient populations across different healthcare systems. Additionally, the reliance on only two domain incremental learning methods may limit the generalizability of the findings. The authors also note that the computational overhead associated with Data Replay could be a barrier for real-time applications in clinical settings. Furthermore, the paper does not explore the long-term implications of continual learning in dynamic clinical environments.

Why it matters
This research has significant implications for the deployment of machine learning models in healthcare, particularly in enhancing the accessibility of predictive analytics for smaller hospitals. By establishing a benchmark for domain incremental learning, the authors provide a framework that can facilitate the transfer of knowledge across diverse clinical settings, ultimately improving patient outcomes. The findings encourage further exploration of model transportability and the development of more robust continual learning strategies that can adapt to evolving data distributions in real-world applications.

Authors: Ryan King, Conrad Krueger, Ethan Veselka, Tianbao Yang, Bobak J. Mortazavi
Source: arXiv:2605.03832
URL: https://arxiv.org/abs/2605.03832v1