ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents

Problem
Current benchmarks for evaluating AI in clinical settings are static and fail to capture the dynamic, interactive nature of medical decision-making. Existing interactive medical benchmarks compromise on critical aspects such as the sequential nature of decisions and the need for real-time information gathering. This paper addresses these gaps by presenting ClinEnv, an interactive benchmark designed to evaluate large language models (LLMs) as attending physicians in real inpatient scenarios. The work is a preprint and has not undergone peer review.

Method
ClinEnv employs a Longitudinal Inpatient Simulation paradigm, where each clinical case is structured into a sequence of decision stages. At each stage, the model interacts with four specialized agents to gather information before making decisions regarding medications, procedures, and diagnoses. The evaluation framework scores both the decisions made by the model and the quality of its information-gathering process. The authors utilize a deterministic ontology-grounded matching system to assess decision quality. The benchmark was tested across seven different models, with a focus on capturing the nuances of clinical management decisions and the iterative nature of information acquisition.

Results
The strongest model evaluated achieved a decision F1 score of 0.31, indicating limited effectiveness in clinical decision-making. Notably, the results revealed a stark contrast in performance between different types of decisions: the model was able to recover discharge diagnoses with an F1 score of 0.51, while management actions yielded a significantly lower score of 0.17. This disparity highlights the challenges faced by models in making complex management decisions. Additionally, the models exhibited a tendency to issue redundant queries as cases progressed, indicating inefficiencies in the information-gathering process.

Limitations
The authors acknowledge that ClinEnv’s scoring mechanism may not fully encapsulate the complexities of real-world clinical decision-making, particularly in terms of contextual factors that influence physician behavior. Furthermore, the benchmark’s reliance on a fixed set of specialized agents may limit its generalizability across diverse clinical scenarios. The paper does not address potential biases in the training data or the implications of model interpretability in clinical settings.

Why it matters
ClinEnv represents a significant advancement in the evaluation of AI systems in healthcare, providing a framework that emphasizes the importance of both decision quality and the process of information acquisition. This dual focus allows for a more comprehensive assessment of LLMs in clinical contexts, which is crucial for developing AI systems that can assist healthcare professionals effectively. The findings underscore the need for improved models that can navigate the complexities of clinical management, as highlighted in the paper. This work lays the groundwork for future research aimed at enhancing AI capabilities in real-world medical environments, as published in arXiv cs.AI.