Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior

Problem
This preprint addresses a significant gap in the literature on computational cognitive modeling, specifically the reliance on behavioral data alone for model discovery. Traditional approaches often yield under-determined models, limiting their predictive power and interpretability. The authors propose incorporating Think Aloud (TA) traces as an additional data source to enhance the model discovery process, particularly in the context of risky decision-making.

Method
The core technical contribution of this work is the integration of Think Aloud data into the automated discovery of cognitive models. The authors employ a framework that combines behavioral trajectories with TA traces to constrain the model space during discovery. The models are evaluated based on their predictive performance on held-out data, with a focus on structural differences between models derived from behavioral data alone and those incorporating TA data. The study utilizes a dataset comprising both behavioral and TA data from participants engaged in risky decision-making tasks. The specific architecture and training compute details are not disclosed, but the methodology emphasizes the importance of process-level language data in reshaping cognitive model structures.

Results
The findings indicate that models discovered using Think Aloud traces exhibit significantly improved predictive performance compared to those derived solely from behavioral data. The authors report that 69.4% of participants had models that shifted from the Explicit comparator structure to the Integrated utility structure when TA data was included. This structural shift suggests that the incorporation of process-level data not only enhances model fit but also enables the identification of cognitive mechanisms that are otherwise obscured when relying solely on behavioral data. The exact performance metrics (e.g., accuracy, AUC) are not specified in the abstract, but the qualitative improvement in model structure is emphasized.

Limitations
The authors acknowledge that their approach may be limited by the quality and richness of the Think Aloud data, as well as the potential biases introduced by participants’ verbalizations. They do not address the scalability of their method to larger datasets or different cognitive tasks, nor do they discuss the computational overhead introduced by processing TA data. Additionally, the generalizability of the findings across diverse populations and decision-making contexts remains unexamined.

Why it matters
This work has significant implications for the field of cognitive modeling and decision-making research. By demonstrating that process-level language data can reshape the structure of cognitive models, the authors open avenues for more nuanced understanding of cognitive mechanisms. This approach could lead to the development of more accurate and interpretable models, enhancing the predictive capabilities of cognitive science applications. Furthermore, it encourages researchers to consider multimodal data sources in model discovery, potentially leading to richer insights into human cognition.

Authors: Hanbo Xie, Akshay K. Jagadish, Lan Pan, Robert C. Wilson
Source: arXiv:2605.05091
URL: https://arxiv.org/abs/2605.05091v1