A Clinician-Centered Pipeline for Annotation and Evaluation in Ultrasound AI Studies

Problem — The paper addresses the lack of integrated platforms for blinded model comparison and reproducible evaluation workflows in ultrasound AI studies. Existing medical image platforms primarily focus on dataset labeling, neglecting the clinician’s perspective in evaluating AI systems. This gap is particularly critical in ultrasound imaging, where quantitative metrics may not fully capture clinical usability. The work is presented as a preprint, indicating it has not yet undergone peer review.

Method — The authors propose a centralized pipeline that utilizes a server and lightweight browser interfaces, allowing clinicians to perform remote annotation, blinded ranking, and review without the need for local dataset downloads. The architecture supports multi-rater participation, enabling diverse clinician input, and aggregates results centrally. Automated statistical analysis is incorporated, generating metrics such as Spearman correlation, Kendall’s τ, and top-1 selection statistics. The pipeline was validated in a fetal ultrasound segmentation study involving six raters with varying levels of expertise (expert, generalist, and non-expert).

Results — The validation study demonstrated moderate to strong agreement among raters, with Spearman correlation coefficients indicating a range of 0.6 to 0.8 across different rater groups. The blinded evaluation revealed a preference for later active learning models, suggesting that the pipeline effectively captures clinician preferences and evaluations. The results indicate that the proposed pipeline can facilitate reproducible human-AI evaluation studies in ultrasound imaging, with statistical significance in the agreement metrics reported.

Limitations — The authors acknowledge that the study’s sample size is limited to six raters, which may affect the generalizability of the findings. Additionally, the reliance on subjective clinician input may introduce variability that is not accounted for in the automated statistical analyses. The pipeline’s performance in other imaging modalities or clinical contexts remains untested, which could limit its applicability. Furthermore, the system’s dependency on internet connectivity for remote access may pose challenges in low-resource settings.

Why it matters — This work has significant implications for the development and validation of AI systems in medical imaging, particularly in enhancing clinician engagement and ensuring that evaluations reflect real-world usability. By providing a structured and reproducible framework for annotation and evaluation, the pipeline can improve the reliability of AI assessments in clinical settings. This is crucial for fostering trust in AI applications among healthcare professionals and ensuring that AI tools meet clinical needs. The proposed pipeline is available on GitHub and further details can be found in the full paper available on arXiv.