Quantifying and Mitigating Premature Closure in Frontier LLMs

Problem
This preprint addresses the gap in understanding premature closure in large language models (LLMs), particularly in the context of medical diagnostics. Premature closure refers to the phenomenon where a model commits to a conclusion without sufficient evidence, which can lead to diagnostic errors. Despite its significance in clinical settings, this issue has not been thoroughly examined in LLMs. The authors aim to quantify the extent of premature closure in frontier LLMs and explore methods to mitigate it.

Method
The authors evaluated five state-of-the-art LLMs on both structured and open-ended medical tasks. They utilized two datasets: MedQA (n = 500) and AfriMed-QA (n = 490), where the correct answer was intentionally omitted to assess the models’ propensity to select incorrect answers. The models’ responses were analyzed for false-action rates, defined as the frequency of inappropriate commitments to answers when the correct choice was not available. Additionally, the authors conducted an open-ended evaluation using HealthBench (n = 861) and a set of 191 adversarial queries authored by physicians to further assess the models’ performance. To mitigate premature closure, the authors implemented safety-oriented prompting techniques and measured their effectiveness across the evaluated models.

Results
The findings revealed alarming false-action rates in the evaluated models. In the MedQA dataset, the false-action rates ranged from 55% to 81%, while in the AfriMed-QA dataset, they ranged from 53% to 82%. In the open-ended evaluation, models provided inappropriate answers in approximately 30% of the HealthBench questions and 78% of the physician-authored adversarial queries. Although safety-oriented prompting techniques reduced the incidence of premature closure, significant residual failure rates persisted, indicating that the models still struggle to recognize when to abstain from answering.

Limitations
The authors acknowledge that their study is limited by the datasets used, which may not fully represent the diversity of medical queries encountered in real-world scenarios. Additionally, the evaluation metrics primarily focus on false-action rates without considering the potential for false negatives, where models may fail to provide a necessary answer. The study also does not explore the underlying reasons for premature closure, such as model architecture or training data biases, which could provide deeper insights into the phenomenon.

Why it matters
This work has critical implications for the deployment of LLMs in clinical settings, where the consequences of premature closure can lead to significant diagnostic errors. By quantifying the extent of this issue, the authors highlight the urgent need for improved evaluation frameworks and training methodologies that emphasize uncertainty management in LLMs. The findings suggest that while safety-oriented prompting can mitigate some risks, further research is necessary to develop models that can effectively discern when to refrain from providing answers, thereby enhancing their reliability in high-stakes environments.

Authors: Rebecca Handler, Suhana Bedi, Nigam Shah
Source: arXiv:2605.15000
URL: https://arxiv.org/abs/2605.15000v1