Evaluating Commercial AI Chatbots as News Intermediaries

Problem
This preprint addresses the lack of systematic evaluation of AI chatbots as news intermediaries, particularly in their ability to accurately handle emerging facts across diverse languages and regions. Prior studies have not comprehensively assessed the performance of these systems in real-world scenarios, especially given their proprietary search integrations and retrieval-synthesis pipelines. The authors aim to fill this gap by evaluating six prominent AI chatbots on their factual accuracy in relation to current news reporting.

Method
The study evaluates six AI chatbots: Gemini 3 Flash and Pro, Grok 4, Claude 4.5 Sonnet, GPT-5, and GPT-4o mini, over a 14-day period (February 9-22, 2026). The evaluation is based on 2,100 factual questions derived from same-day BBC News reporting across six regional services (US & Canada, Arabic, Afrique, Hindi, Russian, Turkish). The evaluation methodology includes both multiple-choice and free-response formats to assess the chatbots’ performance. The authors identify three primary failure patterns: regional bias in retrieval, reliance on correct source identification, and vulnerability to subtle false premises. The study highlights that retrieval failures account for over 70% of errors, indicating a critical dependency on the underlying retrieval infrastructure.

Results
The best-performing chatbots achieve over 90% accuracy in multiple-choice questions regarding events reported just hours prior. However, performance drops significantly in free-response evaluations, with a loss of 11-13% accuracy for the top systems and 16-17% across the cohort. Notably, the Hindi language queries yield the lowest accuracy at 79%, compared to 89-91% for other languages. The study also reveals that models exhibit a detection-accuracy paradox, where the best false-premise detector ranks second in adversarial accuracy, while a weaker detector ranks first. This indicates that premise detection and answer recovery are partially independent capabilities.

Limitations
The authors acknowledge several limitations, including the potential for regional inequity in performance, as evidenced by the lower accuracy on Hindi queries. They also note the reliance on retrieval mechanisms, which may not generalize across different contexts or datasets. Additionally, the study does not explore the long-term implications of these findings on user trust and the broader impact of AI chatbots on news consumption. An obvious limitation not flagged by the authors is the potential for biases in the training data of the chatbots, which could affect their performance across different languages and cultural contexts.

Why it matters
This research has significant implications for the deployment of AI chatbots in news dissemination. The findings highlight the critical need for improved retrieval mechanisms and the importance of addressing regional biases to ensure equitable access to accurate information. Furthermore, the study underscores the vulnerability of AI systems to misleading queries, which could have serious consequences in real-world applications. By identifying these gaps, the work paves the way for future research aimed at enhancing the reliability and fairness of AI chatbots as news intermediaries.

Authors: Mirac Suzgun, Emily Shen, Federico Bianchi, Alexander Spangher, Thomas Icard, Daniel E. Ho, Dan Jurafsky, James Zou
Source: arXiv:2605.22785
URL: https://arxiv.org/abs/2605.22785v1