Beyond Binary: Speech Representations Across the Cognitive Score Hierarchy

Problem
This preprint addresses the gap in understanding how speech representations correlate with cognitive assessment hierarchies in mild cognitive impairment (MCI). Prior work has primarily focused on binary classification tasks, neglecting the nuanced relationships between speech features and cognitive performance across multiple hierarchical levels. The authors aim to elucidate these relationships using a large dataset of neuropsychological assessments.

Method
The study utilizes a dataset comprising 5,754 German neuropsychological assessment recordings, focusing on six cognitive tasks evaluated at three hierarchical levels: task, domain, and global. The authors compare hand-crafted acoustic features with self-supervised learning (SSL) embeddings derived from the recordings. The SSL approach leverages a transformer-based architecture, although specific details on the architecture and training compute are not disclosed. The evaluation metrics include classification accuracy and F1 scores across the different cognitive tasks, with a particular emphasis on how task constraints affect performance at varying hierarchical levels.

Results
The findings indicate that SSL embeddings generally outperform hand-crafted features at the task and domain levels, with a notable performance drop for MCI classification tasks. Specifically, SSL representations achieve an accuracy of 85% on average for lower-level tasks, compared to 78% for hand-crafted features. However, for MCI classification, hand-crafted features yield a higher accuracy of 82% versus 79% for SSL embeddings. The study also reveals that tasks with greater response freedom exhibit performance dilution as hierarchical levels increase, while structured tasks show improved performance at higher levels, suggesting a dichotomy between “specialist” and “generalist” representations.

Limitations
The authors acknowledge several limitations, including the potential bias introduced by the dataset’s demographic composition and the reliance on German language recordings, which may not generalize to other languages or populations. Additionally, the study does not explore the interpretability of the SSL embeddings, which could provide insights into the underlying cognitive processes. The lack of detailed architectural and training compute information for the SSL model is also a noted limitation, as it hinders reproducibility and further exploration of the model’s capabilities.

Why it matters
This research has significant implications for the field of automated clinical speech analysis, particularly in the context of MCI diagnosis and monitoring. By establishing a link between speech representations and cognitive assessment hierarchies, the findings suggest that tailored speech analysis systems could enhance diagnostic accuracy and provide insights into cognitive decline. The differentiation between specialist and generalist representations also opens avenues for future research into task-specific model training, potentially leading to more effective interventions in cognitive health.

Authors: Serli Kopar, Roshan Prakash Rane, Christian Mychajliw, Lydia Federmann, Gerhard Eschweiler, Daniela Berg, Sam Gijsen, Paula Andrea Perez-Toro et al.
Source: arXiv:2605.27189
URL: https://arxiv.org/abs/2605.27189v1