To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

Problem

This work addresses a notable gap in the capability of fact-checking systems to verify claims presented in spoken formats. Traditional fact-checking methods primarily focus on text, leaving a significant challenge in adapting these systems to process and validate audio data. The authors highlight that existing models struggle with the modality shift from text to speech, which is critical for real-world applications where information is often conveyed verbally. This paper is a preprint and has not undergone peer review.

Method

The authors utilize a benchmark dataset named VeriSpeak, which comprises 3,879 spoken claims that include temporal, geographical, and relational facts, with a balanced distribution of true and false labels. The core technical contribution is the development of large audio language models (LALMs) that are augmented with retrieval mechanisms to enhance factual verification capabilities. The evaluation focuses on transferring the ability to verify facts from text to speech, leveraging textual evidence to support claim validation. A key innovation is the introduction of a thinking-tuned LALM, which is designed to improve the model's performance in this context.

Results

The model achieves an accuracy of 86.1% on the VeriSpeak benchmark. However, no baseline performance metrics are reported for comparison, making it difficult to contextualize this result against existing systems.

Limitations

The authors identify several limitations in their approach. Firstly, there is a significant text-speech modality gap, where LALMs exhibit reduced performance on spoken claims compared to their text-based counterparts. Additionally, the authors note that the gains from the retrieval mechanism alone are limited, as the models tend to conflate the retrieved evidence with the spoken claims, potentially leading to inaccuracies in verification.

Why it matters

This research has important implications for the development of more robust fact-checking systems that can operate in real-time audio environments, such as news broadcasts, podcasts, and live discussions. By bridging the gap between text and speech verification, this work paves the way for future advancements in automated fact-checking technologies, enhancing the reliability of information dissemination in spoken formats.