Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals

Problem
This paper addresses the gap in effective uncertainty quantification methods for detecting hallucinations in Large Language Models (LLMs). Existing methods often rely on repeated sampling or external models, which can be computationally expensive and complex. The authors propose a novel approach that leverages internal model signals, specifically attention matrices, to provide a lightweight and efficient solution for hallucination detection. This work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution is a method that quantifies uncertainty by analyzing the Kullback-Leibler (KL) divergence between the distributions of attention heads and a uniform reference distribution. This approach allows for a single-pass evaluation of model outputs without the need for extensive computational resources. The authors utilize logistic regression as a probe to classify the correctness of answers based on the derived attention divergence features. The method is evaluated across various datasets, task types, and model families, demonstrating its versatility and robustness.

Results
The proposed method shows strong performance in predicting answer correctness, achieving competitive results against established uncertainty estimation techniques. While specific numerical results are not detailed in the abstract, the authors indicate that attention divergence is a highly predictive feature across multiple benchmarks. The findings suggest that the signal from attention dynamics is particularly concentrated in the middle layers of the model and is most relevant for factual tokens, such as named entities and numerical data. This indicates a clear relationship between attention behavior and model uncertainty.

Limitations
The authors acknowledge that their method may not capture all forms of hallucinations, particularly those that do not manifest through attention divergence. Additionally, the reliance on logistic regression may limit the complexity of relationships that can be modeled. The study does not explore the generalizability of the method across all LLM architectures or its performance in low-resource settings. Furthermore, the absence of a comprehensive comparison with all existing methods leaves room for further validation of the proposed approach.

Why it matters
This work has significant implications for the development of more interpretable and efficient uncertainty quantification methods in LLMs. By utilizing internal attention signals, the proposed method offers a promising avenue for enhancing the reliability of LLM outputs, particularly in applications where factual accuracy is critical. The findings could inform future research on model interpretability and robustness, potentially leading to improved designs for LLMs that better manage uncertainty and hallucinations. This approach may also inspire further exploration of attention mechanisms as diagnostic tools in other areas of machine learning.

Authors: Gijs van Dijk
Source: arXiv:2605.05025
URL: https://arxiv.org/abs/2605.05025v1