When LLMs Analyze Scars: From Images to Clinically-Meaningful Features

Problem
Medical image classification, particularly in pathological scar assessment, faces significant challenges due to data scarcity stemming from high annotation costs, privacy issues, and the rarity of certain conditions. This paper addresses the gap in leveraging large language models (LLMs) for feature extraction in scenarios where labeled data is limited, particularly for differentiating between keloids and hypertrophic scars. The authors propose a preprint framework, ScaFE (Scar Feature Engineering), which repositions LLMs as feature engineers rather than end-to-end classifiers.

Method
The ScaFE framework utilizes LLMs to generate executable Python code that extracts clinically relevant features from high-dimensional medical images. The authors prompt the LLM with established scar assessment criteria, such as the Vancouver Scar Scale, to produce deterministic feature extraction algorithms. This approach allows for the transformation of image data into low-dimensional representations that are clinically interpretable. The method emphasizes data efficiency by decoupling knowledge acquisition from statistical learning, thus enabling robust performance with limited training samples. The authors do not disclose specific training compute or architecture details for the LLMs used.

Results
The experimental results demonstrate that ScaFE consistently outperforms traditional end-to-end deep learning baselines and LLMs used as black-box classifiers under limited data conditions. The paper reports significant improvements in classification accuracy, although specific numerical results and comparisons to named baselines are not detailed in the abstract. The authors emphasize the advantages of their method in terms of data efficiency, privacy preservation, and interpretability, suggesting a clear performance edge in clinical settings.

Limitations
The authors acknowledge that their approach relies on the quality of the prompts provided to the LLM, which may affect the feature extraction process. Additionally, the method’s reliance on existing clinical criteria may limit its applicability to novel or less well-defined conditions. The paper does not address potential biases in the LLMs or the generalizability of the extracted features across diverse patient populations. Furthermore, as a preprint, the findings have not undergone peer review, which may impact their validation.

Why it matters
The implications of this work are significant for the integration of LLMs into medical AI systems, particularly in enhancing data efficiency and interpretability in clinical applications. By transforming LLMs into feature extraction tools, the authors provide a promising direction for developing AI systems that can operate effectively in data-scarce environments while maintaining clinical relevance. This approach could pave the way for more robust and interpretable medical image analysis frameworks, as discussed in related literature on the integration of AI in healthcare, such as in arXiv cs.LG.