Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation

Problem
This work addresses the gap in performance when spoken dialogue models, which typically leverage text-based large language model (LLM) backbones, are conditioned on speech inputs. The authors identify a temporal-granularity mismatch as a significant factor contributing to degraded reasoning capabilities. Specifically, they note that speech tokens are temporally redundant and longer than their text counterparts, which dilutes semantic density and hampers text-native reasoning dynamics. This study is presented as a preprint, indicating that it has not yet undergone peer review.

Method
The authors propose a novel approach to speech token design, framing it as a representation selection problem. They introduce a factorized Frame-Synchronous Quantization (FSQ) method and a lightweight non-autoregressive audio language model (LM) head, which allows for scaling capacity to nearly 300 bits/frame while maintaining efficient prediction. The study involves sweeping frame rates from 50 Hz down to 2.08 Hz while keeping the LLM backbone frozen and maintaining a fixed information rate. The authors also explore the alignment depth of intermediate-layer representations to optimize performance in speech question answering (QA) tasks.

Results
The experiments reveal that the optimal frame rate for speech QA is 4.17 Hz, where the alignment of intermediate-layer representations yields the best performance. The authors report significant improvements in reasoning capabilities compared to baseline models that do not utilize the proposed token design and frame rate adjustments. While specific numerical results are not detailed in the abstract, the consistent performance gains across various configurations suggest a robust enhancement over traditional methods.

Limitations
The authors acknowledge that their approach is contingent on the fixed information rate of the LLM backbone, which may limit the generalizability of their findings to other architectures or tasks. Additionally, the study does not explore the effects of varying the LLM backbone itself, which could yield further insights into the interplay between speech representation and reasoning performance. The reliance on a frozen LLM may also restrict the adaptability of the model to different domains or languages.

Why it matters
This research has significant implications for the development of more effective spoken dialogue systems, particularly in enhancing reasoning capabilities when processing speech inputs. By addressing the temporal-granularity mismatch and optimizing speech token design, the findings could lead to advancements in various applications, including virtual assistants and automated customer service systems. The insights gained from this study can inform future work on multimodal models that integrate speech and text more effectively, as published in arXiv.