LLM-Check: Investigating Detection of Hallucinations in Large Language Models

Comparison with standard classifiers on hidden representations: We thank the reviewer for the suggestion. We first note that methods that rely upon supervised training of classifiers on internal model representations over samples with and without hallucinations, can add large computational overheads. For instance, Inference-Time Intervention (ITI) [2] relies upon training 1024 binary classifiers on the TruthQA datasets, and thus becomes prohibitively expensive as also noted by Chen et al. in the INSIDE paper.

• However, as suggested by the reviewer, we experimented with standard classifiers on hidden representations on the FAVA Annotation dataset. Namely, we create a train-test split of the dataset, and train classifiers on the hidden representations corresponding to layer 20 and layer 30 of a Llama-2-7B model. We use this split to ensure that we have about 100 testing samples balanced between classes (with and without hallucinations) to obtain reliable evaluations of the classifiers so trained.
• We observe that using layer 20, this method obtains AUROC, Accuracy and TPR@5%FPR of 55.76, 59.82 and 0.00 respectively, while using layer 30 we observe detection rates with AUROC, Accuracy and TPR@5%FPR of 56.52, 61.61 and 1.79 respectively. We note that despite requiring supervised training, this method performs significantly worse than the proposed Attention Score which achieves AUROC, Accuracy and TPR@5%FPR of 72.34, 67.96 and 14.97 respectively.
• We hypothesize that the generalization of these classifiers might not be adequate to achieve reliable detection performance across the diverse hallucination samples and types as present in the FAVA dataset as the corresponding hidden representations become similarly disparate.

MLP Contributions in Transformer Block: The MLP contribution is the standard form in the transformer block at layer $l$ is given by $M_l$ = $W^1_l$ $\rho ($ $W^2_l ($ $A_l$ + $H_{l-1}))$, as the residual block incorporates both the attention kernel of the same layer and hidden representation of the previous layer, as introduced by Vaswani et al. in the original transformers paper. We note that since the MLP is applied to each position separately and identically, it does not capture cross-token interactions, and thus we do not consider this component for detection itself.

Additional Comments: We also thank the reviewer for pointing out the typo in line 90, we will correct this error in the final version of the paper. In Line 137-138, “to determine the presence or absence of hallucination in the response $x$, in the context of $x_p$” refers to the detection of hallucinations in the output response $x$ when the input prompt is $x_p$.

We thank the reviewer for the detailed suggestions and constructive comments. We kindly ask if the reviewer would consider increasing their score if their concerns or questions have been addressed. We would be glad to engage further during the discussion period.

We thank the reviewer for their valuable feedback. We are encouraged that the reviewer appreciates the efficacy of the proposed method extensively demonstrated across diverse detection settings and datasets. We respond to the questions raised below:

Theoretical insights and comparisons with INSIDE

• In this work, we study the highly challenging task of hallucination detection within a single example - of the form $(x_p \oplus x)$ for a prompt $x_p$ and corresponding model response $x$ - without any training or inference time overheads. This sharply contrasts with INSIDE which performs population-level detection by computing the centered covariance matrix across multiple model responses to check self-consistency of the set so generated.
• Indeed, though the mean-log-determinant formula appears similar to that of INSIDE, in LLM-Check the interaction in latent space between the different token representations within the same sample response is utilized, as we posit that the LLM is in fact sensitive to the presence of non-factual, hallucinated information in responses which would otherwise be truthful. This arises from the probabilistic language-modeling learnt by the LLM, based on factual precepts encountered in its training and that this can then be efficiently leveraged to perform hallucination detection within a single model response itself.
• Theoretically, this is well-motivated as the eigenvalues and singular values capture the changes in hidden interactions and patterns to the extent of attention applied, which we know is different in hallucinated samples which contain non-truths compared to non-hallucinated sample sequences. We also remark that the Attention score can be found without explicitly computing the SVD or eigen-decompositions since it is lower-triangular, enabling speedups between 44x and 450x compared to existing baselines, while also simultaneously achieving significantly improved detection performance.
• Furthermore, the attention values represent softmax activations of the given attention head, and thus represent probability values arising from the cross-variation of the Keys and Values in the Attention Mechanism. Thus, the log-determinant correctly reduces to the trace of the log-values and characterizes the joint distribution of the predicted probabilities in the attention head, while the trace of the values themselves represents the sum of distinct probability values which is not theoretically sound, as these are not upper-bounded by 1.

Single-Response Analysis: As noted by the reviewer, in the work we do perform hallucination detection within a single prompt-output response. However, we strongly believe that this is an inherent advantage and not a limitation, as the single-response analysis can be extended to population-level analysis by statistically aggregating scores across multiple output responses. We crucially note that this is strictly unidirectional - in that population-level detection methods such as INSIDE cannot be easily applied for single-response analysis. Moreover, we believe that the single-response analysis is more relevant for real-time detection of hallucinations in practical real-world use-cases.

Robustness across Domains: In the paper, we do attempt to cover some extent of diversity in domains between the FAVA-annotation dataset, FAVA-train split, SelfCheckGPT Dataset and RAGTruth dataset. Indeed, the FAVA Annotation dataset itself spans four different data sources: Knowledge-intensive queries sampled from the Open Assistant dataset (Kopf et al. , 2023), Open QA prompts from the No Robots dataset (Rajani et al., 2023), Instruction-following and WebNLG (Gardent et al., 2017) datasets. In addition, the FAVA-train split consists of Wikipedia articles and Question Answering datasets (Kwiatkowski et al., 2019), while SelfCheckGPT Dataset consists of Annotations from the WikiBio dataset (Lebret et al., 2016). Lastly, the RAGTruth dataset consists of annotations from the CNN/Daily Mail dataset (See et al., 2017). We do however agree that this could be further expanded to more domains like medical datasets, but the annotation process specifically for hallucinations can be very expensive in such cases since it often requires a great deal of expertise to peruse and annotate carefully. However, we do believe that if specific domains such as a medical-data is considered, wherein a LLM specifically fine-tuned on such data is used, hallucination detection techniques such as LLM-Check will continue to be highly effective.

Real-world Feasibility: We wish to clarify that we do indeed believe that the proposed method is feasible in real-world scenarios, and relies upon the very mild assumption that at least one open-source LLM such as Llama is available to serve as a proxy to compute our detection scores. While this assumption may not always strictly hold in all scenarios, we do believe that this covers a very large fraction of practical use cases encountered in the real-world. Furthermore, the effective black-box performance indicates a considerable real-world advantage over methods such as INSIDE which is strictly white-box and requires multiple output responses at inference time.

Thank you for the detailed and comprehensive answer. I could agree with some remarks, especially about Single-Response Analysis as an advantage. Also, I would like to highlight the comparison provided with standard classifier looks convincing. However, I would also point out that some of remarks are not properly addressed possibly due to misunderstanding. I will clarify my concerns:

Theoretical motivation: In INSIDE this method was well motivated, exactly because EigenScore measured the interactions between different samples. The method works because similar samples would give co-aligned eigenvectors and low EigenScore. In this work, you measure the same score but for different tokens' embeddings, and I believe it is not evident why should they be more aligned in non-hallucinatory content. I think the method is efficient, but I miss the motivation or explanation, that would definitely strengthen your study. As a suggestion, you could try to inspect eigenvectors and eigenvalues for special examples of hallucinatory and non-hallucinatory content. Moreover, this metric seems to be related to linear intrinsic dimensionality of embeddings. The intrinsic dimensionality of embeddings was used, for example, in [1] or [2].

Robustness across Domain: when commenting about robustness to domain, I mean that if scores are robust to domain, the distribution of scores should be comparable for Wikipedia articles and Question Answering datasets from FAVA. Could you provide some results proving that scores are consistent across different distributions of texts?

I will increase my score, if you could discuss my concerns and provide some results.

[1] Tulchinskii, Eduard, et al. "Intrinsic dimension estimation for robust detection of ai-generated texts." Advances in Neural Information Processing Systems 36 (2024).

[2] Yin, Fan, Jayanth Srinivasa, and Kai-Wei Chang. "Characterizing truthfulness in large language model generations with local intrinsic dimension." arXiv preprint arXiv:2402.18048 (2024).

Comparing Eigenvalue Analysis of Internal LLM Representations and Output Token Uncertainty Quantification:

• We propose these two distinct lines of analysis towards hallucination detection in order to attempt to adequately capture the extremely heterogeneous and diversified forms of hallucination displayed by modern Large Language Models over different domains. Towards this, the Eigen-analysis of internal LLM representations helps highlight the consistent pattern of modifications to the hidden states and the model attention across different token representations in latent space when hallucinations are present in model responses as compared to truthful, grounded responses.

• On the other hand, the uncertainty quantification of the output tokens helps analyze hallucinations based on the likelihood assigned by the model on the actual tokens predicted at a specific point in the sequence generated auto-regressively. Indeed, especially considering that the LLM is trained using next-token prediction, we expect the probability distribution $p_f (\cdot |x)$ for a given token to be highly salient toward the relative choices available for completion, and in identifying non-factual components if present.

• Thus, we utilize these diversified scoring methods from different model components to potentially maximize the capture/detection of hallucinations amongst its various forms without incurring computational overheads at training or inference time. In practice, we generally observe that the Eigen-based analysis of Internal LLM representations achieves better detection performance over Output Token Uncertainty Quantification, as observed in Table-2 with the FAVA annotation dataset (without external references) and in Table-4 with the RAGTruth dataset (with external references).

• However, on the FAVA Train-data-split where hallucinations are inserted synthetically using GPT-3, the output-based uncertainty scoring methods are seen to be more effective at capturing the modified changes to the joint-distribution of the sequence level token prediction probabilities, as shown in Table-5. However, we do believe that this synthetic insertion of hallucinations is perhaps less likely to be encountered in real-world settings.

Combining Different Detection Methods: We thank the reviewer for the suggestion to try to combine these different methods towards a unified, holistic form of hallucination detection. We observe that this can be reduced to learning a classifier that takes the different scores as input and is trained to predict the absence/presence of hallucinations. We conducted preliminary experiments towards this on the FAVA Annotation dataset using Logistic Regression, but observed no appreciable gains over using the best performing individual method, the Attention Score. We seem to observe that learning an optimal combination of the different scores in a manner that it generalizes well across diverse truthful and hallucinatory inputs is a highly non-trivial task, especially considering the different forms of hallucination highlighted in the fine-grained taxonomy presented in the FAVA dataset.

We thank the reviewer for the detailed suggestions and constructive comments. We kindly ask if the reviewer would consider increasing their score if their concerns or questions have been addressed. We would be glad to engage further during the discussion period.

Comparing with other Uncertainty Metrics:

• We also thank the reviewer for the suggestion of sampling-based uncertainty metrics such as Predictive Entropy [1] and Semantic Entropy [2]. However, we observe that both these metrics quantify the total uncertainty over the distribution of all possible responses given a fixed prompt, in sharp contrast to the primary focus of this paper, which analyzes the hallucinatory behavior within a single fixed model response output on a given prompt. Thus, Predictive Entropy and Semantic Entropy perform population-level uncertainty estimation akin to INSIDE, and does not apply to the single-response case.
• Furthermore, Semantic Entropy is likely very costly to estimate for real-time detection, since multiple sequences have to first be generated, and then clustered based on bi-directional entailment again using a language model, and then finally estimating entropy over the different clusters. In a some manner, something similar Semantic Entropy based estimation for a single response is performed in “SelfCheckGPT with BERTScore” and “SelfCheckGPT with NLI” in their original paper, but these are seen to perform worse than their best scoring method “SelfCheckGPT with Prompt”. We note that LLM-Check performs better than SelfCheckGPT-Prompt as well (Tables-2,3).

However, we do remark that Predictive Entropy and Semantic entropy might possibly be useful to mitigate hallucinations in training phase rather than in detection, by utilizing fine-tuning and down-weighting excessive overall semantic entropy over all possible observed responses to a given fixed prompt.

[1] Uncertainty Estimation in Autoregressive Structured Prediction, Malinin and Gales, 2020

[2] Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation, Kuhn et al., 2023

Interpreting Table-4 and Black-Box results: In this setting, we utilized the Llama2-7B model for computing the various LLM-Check scores, and analyzed the detection performance of hallucinations present in texts generated from different models like Llama2-13B, GPT-4, Mistral-7B, and Llama2-7B from the RAGTruth dataset. Thus, the white-box setting is presented in the third column where the evaluation model and generation model are the same, and the remaining columns represent the black-box evaluations. Overall, we observe an interesting trend that the hallucinations in responses generated by larger models are easier to detect overall, though the frequency with which they hallucinate is lower. Thus as a result, the overall black-box results appear to be better than the white-box detection. In particular, when the model size is kept unchanged, the white-box Llama-2-7B detection results are very similar to the black-box detection of hallucinated responses from Mistral-7B. This also potentially indicates that models may be more sensitive to hallucination based outliers generated by other models due to their inherent differences in representation and training.

Implicit Modeling in Black-box setting: We certainly agree with the reviewer that in the black-box setting, implicit modeling assumptions are made with respect to the data distribution of the original and proxy model utilized (i.e. that they are trained on some similar data distribution), and that this may not always hold. However, we do observe that recent real-world LLMs such as from the Llama, GPT, and Claude family are trained on truly staggering amounts of data and have a significant degree of world-knowledge, as validated by their excellent performance on different benchmarks. Thus, on an extremely large variety of use-cases, these models do indeed have overlapping knowledge bases, to an extent where such transfer modeling is indeed effective, as seen with the black-box detection results in Table-4. Furthermore, we note that while methods such as INSIDE cannot inherently be used in the black-box setting, other methods such as SelfCheckGPT do indeed analyze the black-box setting too, since access to proprietary LLMs is expected to be fairly limited even moving forward.

We thank the reviewer for their valuable feedback. We are encouraged that the reviewer appreciates the efficacy of the proposed method across diverse detection settings and datasets, and the significant performance improvements achieved while requiring only a fraction of the computational cost (speedups upto 45x and 450x) as the proposed method does not utilize multiple model responses or extensive external databases. We respond to the questions raised below:

Insight on the Eigen-Analysis based Detection:

• In this work, we study the highly challenging task of hallucination detection within a single example - of the form $(x_p \oplus x)$ for a prompt $x_p$ and corresponding model response $x$ - without any training or inference time overheads. While LLMs do hallucinate, they still have a significantly appreciable degree of world-knowledge grounded on truthful facts encountered in the training stage, which is reflected in the fact that the hallucinations are absent in some of the autoregressively generated sample outputs when multiple model responses are considered for the same prompt. This also hints that one of the potential root-factors that induces hallucination could be from the nature of auto-regressive sampling for generation in LLMs, since tokens once generated and selected at a given point in the output sequence cannot be overwritten or corrected at a later stage, and the LLM simply attempts to maximize the likelihood of the overall response moving from that point on, though it potentially subtly sensitive to the reality that a non-factual error is already present in the response.

• Furthermore, this lack of self-consistency across multiple generated sample outputs forms the basis for consistency-based methods such as SelfCheckGPT and INSIDE. In contrast, in this work, we propose to directly analyze the variations in model characteristics between truthful and hallucinated examples by analyzing the rich semantic representations present in the hidden activations of LLMs, since we know that the LLM does generate the truthful version, albeit with potentially lower frequency. Thus, we posit that the model is in fact sensitive to the presence of non-factual, hallucinated information in responses which would otherwise be truthful, based on factual precepts encountered in its training and that this can then be efficiently leveraged to perform hallucination detection within a single model response itself. In particular, these differences would arise in the hidden latent-space representations and the pattern of attention maps across different token representations in hallucinated responses compared to non-hallucinated responses. To quantitatively capture these variations, we thus proposed to analyze the covariance-matrix for hidden representations, and the kernel similarity map of self-attention, since these form the foremost and paramount salient characteristics of the LLM itself.

• In addition, given that we require the method to be highly compute-efficient to enable real-time detection, we distill simple yet salient scalars - the mean log-determinant - from these variations in hidden representation and attention maps using eigen-analysis. Theoretically, this is well-motivated as the eigenvalues and singular values capture the interaction in latent space between the different token representations, which we know is different in hallucinated samples which contain non-truths compared to non-hallucinated sample sequences. We also remark that the Attention score can be found without explicitly computing the SVD or eigen-decompositions, enabling speedups between 44x and 450x compared to existing baselines, while also simultaneously achieving significantly improved detection performance.

Comparing with other Uncertainty Metrics:

• We thank the reviewer for the suggestion. We do first note that the Perplexity score is closely related to the overall negative log-likelihood, namely that the Perplexity is defined as the exponentiated length-normalized negative log-likelihood of a sequence. However, we did measure the detection performance of standard negative log-likelihood as suggested by the reviewer, and we observe AUROC, Accuracy and TPR@5%FPR metrics of 52.29, 54.19 and 2.99 respectively on the FAVA Annotation Dataset, which are worse than those obtained with Perplexity (AUROC, Accuracy and TPR@5%FPR of 53.22, 58.68 and 3.59 respectively).

Layer Selection: We thank the reviewer for the valuable comment and suggestion. We first wished to clarify that the method is computationally efficient even when all model layers are used for computing Hidden or Attention scores. Indeed, the runtime analysis presented in Figure-2, includes the empirical average runtime for computing Attention scores and Hidden scores from all 32 layers of Llama-2-7b. That is, the Attention score computation for all 32 layers takes 0.22 seconds per example, while the Hidden score computation for all 32 layers requires 2.72 seconds per example when averaged over the FAVA Annotation dataset. Since the overhead to compute scores for all layers was so small, we expect that they can be utilized for real-time analysis.

• But we certainly agree with the suggestion that rapid methods for selecting layers would be beneficial in practice. We first observe that the performance obtained with the Hidden score is extremely stable across layers, and thus it is relatively easy to choose, though we recommend a middle-level layer such as layer 20 for a 32 layer, 7-billion parameter model such as Llama-2. On the other hand, we do indeed observe a larger degree of oscillations across layers with the Attention score. Here, we perform an experiment to potentially rapidly select layers, by plotting the results obtained using few samples, and subsequently check if the overall performance on the dataset can be estimated using this for each layer. We present these evaluations in Figure-2 of the rebuttal PDF. We do indeed observe a fair degree of agreement between results obtained with 5, 20 and 50 pairs as compared to the full dataset.
• In general, we do observe that the layers between 19 and 23 achieve close-to-optimal performance for the Attention score computed on 32 layer, 7-billion parameter models such as Llama-2-7b and Vicuna-7b, and the similar Llama-3-8b model for white-box detection across different datasets. We hypothesize that for the white-box setting, while the very early layers are involved in feature extraction, and the last layers are involved more towards making an optimal next-token prediction, the layers after the midpoint of the network are quite suitable for hallucination detection. In the black-box setting, this is much more difficult since it is non-trivial to map representations of intermediate layers between different LLMs, especially when the original LLM has many more layers, as with GPT-4 or Llama-2-70B.

We thank the reviewer for the support for acceptance and greatly appreciate the suggestions and constructive comments. We kindly ask if the reviewer would consider increasing their score if their concerns or questions have been addressed. We would be glad to engage further during the discussion period.

We thank the reviewer for their valuable feedback. We are encouraged that the reviewer appreciates the compute-efficiency of the proposed method, and its practicality and versatility towards real-time hallucination detection within a single model-response across diverse settings. We respond to the questions raised below:

Comparing Eigenvalue Analysis of Internal LLM Representations and Output Token Uncertainty Quantification:

• We propose these two distinct lines of analysis towards hallucination detection in order to attempt to adequately capture the extremely heterogeneous and diversified forms of hallucination displayed by modern Large Language Models over different domains. Towards this, the Eigen-analysis of internal LLM representations helps highlight the consistent pattern of modifications to the hidden states and the model attention across different token representations in latent space when hallucinations are present in model responses as compared to truthful, grounded responses.

• On the other hand, the uncertainty quantification of the output tokens helps analyze hallucinations based on the likelihood assigned by the model on the actual tokens predicted at a specific point in the sequence generated auto-regressively. Indeed, especially considering that the LLM is trained using next-token prediction, we expect the probability distribution $p_f (\cdot |x)$ for a given token to be highly salient toward the relative choices available for completion, and in identifying non-factual components if present.

• Thus, we utilize these diversified scoring methods from different model components to potentially maximize the capture/detection of hallucinations amongst its various forms without incurring computational overheads at training or inference time. In practice, we generally observe that the Eigen-based analysis of Internal LLM representations achieves better detection performance over Output Token Uncertainty Quantification, as observed in Table-2 with the FAVA annotation dataset (without external references) and in Table-4 with the RAGTruth dataset (with external references).

• However, on the FAVA Train-data-split where hallucinations are inserted synthetically using GPT-3, the output-based uncertainty scoring methods are seen to be more effective at capturing the modified changes to the joint-distribution of the sequence level token prediction probabilities, as shown in Table-5. However, we do believe that this synthetic insertion of hallucinations is perhaps less likely to be encountered in real-world settings.

Combining Different Detection Methods: We thank the reviewer for the suggestion to try to combine these different methods towards a unified form of hallucination detection. We observe that this can be reduced to learning a classifier that takes the different scores as input and is trained to predict the absence/presence of hallucinations. We conducted preliminary experiments towards this on the FAVA Annotation dataset using Logistic Regression, but observed no appreciable gains over using the best performing individual method, the Attention Score. We seem to observe that learning an optimal combination of the different scores in a manner that it generalizes well across diverse truthful and hallucinatory inputs is a highly non-trivial task, especially considering the different forms of hallucination highlighted in the fine-grained taxonomy presented in the FAVA dataset.

We thank the reviewer for their valuable feedback. We are glad that the reviewer found the proposed method to be novel, and is effective towards hallucination detection in various settings, while being extremely efficient computationally. We respond to the questions raised below:

The paper lacks a figure illustrating the overall pipeline, which makes it hard to grasp the main idea at first glance.

• We sincerely thank the reviewer for the suggestion, and as suggested we include a schematic of the eigen-analysis detection methods as Figure-1 in the rebuttal PDF. We shall certainly incorporate the figure into the final version of the paper.

Do you have any preliminary thoughts or plans on how the detection method could be extended to also mitigate hallucinations?

• We thank the reviewer for this question. We anticipate that LLM-Check could be directly incorporated towards providing additional automated feedback in the fine-tuning stage of LLMs with Reinforcement Learning, wherein output sample generations that are detected to be hallucinatory in nature can be down-weighted appropriately. Additionally, the detection methods can assist in flagging samples in a highly efficient manner towards a customized human-feedback loop with RLHF, wherein annotators can introduce an orthogonal ranking which reflects the desired extent of factuality for the sample generations so considered.

What does population-level mean in hallucination detection?

• As we note in Section 3.2, the prior method INSIDE performs hallucination detection by generating multiple stochastically sampled responses $x_1 , x_2 , . . . x_K$ for a given prompt $x_p$, and then computes eigen-decomposition of the covariance matrix of hidden activations of these samples. This is used to then statistically infer the presence of hallucinations within this generated sample set $x_1 , x_2 , . . . x_K$, based on a possible lack of self-consistency at a “population level” between samples within this specific set. This is in contrast to single-response detection that infers the presence/absence of hallucinations in a given fixed (single) output response $x$ for a given prompt $x_p$, as performed in our method, LLM-Check.

What are the differences between the black-box settings and white-box settings? What do the hidden score, attention score indicate?

• For a given prompt $x_p$, we consider hallucination detection in an output response $x$ that is generated by a given LLM $f$. If the original LLM $f$ is available and accessible to compute the scores proposed with LLM-Check, the setting is considered to be “white-box”. If the original LLM $f$ that generated the response $x$ for prompt $x_p$ is no longer available or inaccessible, we utilize an auxiliary substitute LLM $\hat{f}$ (such as open-source LLMs like Llama-2) to compute scores with internal activations and attention kernel maps using teacher-forcing, and this setting is considered to be “black-box”. The hidden score and attention score are real-valued scalar scoring metrics that can be then thresholded to determine the presence/absence of hallucinations in a given output response $x$, using the original LLM $f$ itself in the white-box setting, and an auxiliary substitute LLM $\hat{f}$ in the black-box setting. The hidden scores and attention scores are derived from the mean log-determinant of the covariance-matrix for hidden representations, and the kernel similarity map of self-attention of the LLM. Theoretically, this mean log determinant is given by the average logarithm of the corresponding singular values and eigenvalues which capture the interaction in latent space between the different token representations, which we know is different in hallucinated samples containing non-truths compared to non-hallucinated sample sequences.

We sincerely thank the reviewer for the support for acceptance and greatly appreciate the suggestions and constructive comments. We kindly ask if the reviewer would consider increasing their score if their concerns or questions have been addressed. We would be glad to engage further during the discussion period.