Frozen Multimodal Embeddings for Personality and Cognitive Ability Assessment in Asynchronous Video Interviews

Problem
This work addresses the challenge of predicting psychological traits from asynchronous video interviews (AVIs), a task complicated by the limited availability of labeled datasets and the high-dimensional nature of the multimodal signals (visual, acoustic, and verbal) present in the responses. The authors present their approach as a solution for the ACM Multimedia AVI Challenge 2026, focusing on two specific tasks: predicting self-reported HEXACO personality traits and classifying cognitive ability levels. Notably, this paper is a preprint and has not undergone peer review.

Method
The authors propose a novel approach that leverages frozen multimodal encoders instead of fine-tuning large pretrained models. They utilize CLIP for extracting visual features, Whisper for acoustic features and transcripts, and RoBERTa, E5, and DeBERTaV3 for textual representations. The architecture employs low-capacity downstream models to process these embeddings. For Track 1, they implement a trait-specific regression model combined with a late-fusion strategy, which integrates predictions from individual traits. For Track 2, they utilize a compact subject-attribute baseline and a multimodal ensemble approach. The training process emphasizes small-sample representation learning, which is critical given the limited labeled data.

Results
In Track 1, the proposed model achieves an average validation Mean Squared Error (MSE) of 0.2696, significantly outperforming the official baseline of 0.3334, resulting in a 19.1% relative MSE reduction. The authors detail an ablation study showing a progression from a global model (MSE of 0.3189) to per-trait modeling (0.2871) and finally to per-trait late fusion (0.2696). For Track 2, the compact subject-attribute baseline reaches an accuracy of 0.5781, while the multimodal ensemble achieves 0.5313, both exceeding the official baseline of 0.4062. The authors suggest that the performance in Track 2 may reflect potential shortcuts in the validation dataset rather than robust cognitive inference.

Limitations
The authors acknowledge that their findings in Track 2 may be influenced by dataset shortcuts, indicating a need for careful validation to ensure that cognitive ability predictions are not merely artifacts of the data. They do not address potential biases in the multimodal embeddings or the implications of using frozen models in dynamic environments. Additionally, the reliance on low-capacity downstream models may limit the ability to capture complex interactions among modalities.

Why it matters
This research highlights the potential of using frozen multimodal embeddings for psychological assessment, particularly in scenarios with limited labeled data. The findings suggest that trait-specific modeling can enhance the accuracy of personality assessments, while caution is warranted in cognitive ability predictions due to possible dataset biases. This work contributes to the growing field of multimodal learning and psychological evaluation, as published in arXiv cs.AI.