INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection

Comment We thank Reviewer 5aJw (R4) for their constructive suggestions. According to the suggestion, we have provided a comprehensive Case Studies in the Appendix H (Page16-21) of our revised paper. Here is a point-by-point response.

---

W1 Regarding the outliers in the activations, it would be nice to see if these outliers really lead to overconfident predictions. For example, is there any correlation between the number of outliers and accuracy of the answer? At least, it could be informative to see a few generated examples corresponding to many outliers or no outliers.

Ans Thanks for your suggestion. The outliers features increase the likelihood of generating overconfident and self-consistent generations. For example, the top 50 tokens that contain the most outliers have an average confidence of 0.98, while the bottom 50 tokens that contain the least outliers have an average confidence of 0.54. According to your suggestion, we provide several examples that contain many/few extreme features in the Appendix H.2 (Page19). The results show that when there are many extreme features, the model tends to generate consistency hallucination outputs across multiple generations. Instead, when there are few extreme features, the model generates diverse hallucination outputs which can be spotted by different hallucination detection metrics.

---

W2 For Table 2, it would be interesting to see if the quality of answers is increased after using clipping (as confident mistakes are excluded).

Ans It is worth noting that the feature clipping approach is proposed to prevent the model from generating overconfident outputs for uncertain questions, but not to improve the quality of answers. To demonstrate the effectiveness of the feature clipping strategy, we show several cases before and after feature clipping in the Appendix H.3 (Page20)

---

W3 In general, it would be nice to see some qualitative examples of hallucinations that are detected by the proposed method, but not detected by the baselines.

Ans Thanks for your suggestion. We have provided comprehensive cases in Appendix H.1 (Page16), including good cases and failure cases. Please see the details in our revised paper.

---

W4 typos：centralized matrix -> centering matrix；"prior works that rectifying", "nli-roterta-large".

Ans Thanks，the typos have been revised. We will re-polish our paper carefully.

Q1 For the correctness Measure, I agree that the evaluation of answers is challenging, but it's better to have an exact match score so that we can compare it with other existing works

Ans According to the suggestion, we have provided the evaluation results by employing Exact Match as the correctness measure. The results in Table 3 show similar conclusion with using ROUGE and sentence similarity as the correctness measure, which demonstrates that our proposal consistently outperforms the comparison baselines under different evaluation metrics.

Please see the Appendix E (Page14) in our revised paper for more details.

Table3: Performance evaluation with Exact Match as correctness measure.

---

Q2 (1) As feature clipping is claimed as a contribution of this work, could you provide some ablation studies for it? (2) I am curious why it needs a dynamic memory bank. Does that mean the distribution of activation values changes a lot across samples?

Ans (1) We have already provided the ablation study in our original submission in (Page7 - Effectiveness of Feature Clipping), which demonstrates the effectiveness of the introduced feature clipping techinque. Besides, we also add several samples to show the hallucination detection results before and after feature clipping in the Appendix H3 (page18).

(2) We illustrate the distributions of neuron activation from four selected tokens in Fig.6 (Appendix F, Page15), which shows that the distribution changes a lot across samples. Therefore, it is risky to determine the clipping threshold with only the current input sample (EigenScore-C). A feasible solution is to pre-compute the optimal threshold based on a batch of input samples (EigenScore-P). Besides, another solution is to dynamically record the activation values and determine the threshold during the inference process (EigenScore-MB). We have tried both solutions and the experimental results are presented in Table 4. The results show that determining the thresholds with a memory bank works slightly better. We attribute this variability to potential differences in the activation distributions across various datasets.

Table4: Ablation study of determining the clipping threshold with different technique. AUC$_s$ is reported.

Please see the Appendix F (Page15) in our revised paper for more details.

---

Reference

[1] TruthfulQA: Measuring How Models Mimic Human Falsehoods

[2] Training a helpful and harmless assistant with reinforcement learning from human feedback

Comment We thank Reviewer 9Srq (R3) for the careful reviews and insightful suggestions, which really helped us improve our paper.

W1 (1) The experiment setting could be improved. I think TruthfulQA is closer to the topic of hallucination compared to the benchmarks used in the paper. (2) It's better to include recent LLMs, such as LLaMA2 and Falcon.

Ans (1) According to your suggestion, we have compared our proposal with several competitve approaches in the TruthfulQA dataset. The results are presented in Table 1. For the ITI [1], which provides a strong hallucination detector that trained on TruthfulQA, we report the best performance in their paper. As can be seen, our proposal consistently outperforms the baseline methods and achieves comparable performance as ITI when we utilize 50 in-distribution prompts.

Table 1: Performance comparison of different methods on TruthfulQA dataset. Accuracy is reported.

It's worth nothing that the ITI [1] relies on training 1024 binary classifiers in TruthQA datasets, and they report the best performance (83.3) in the validation set. Therefore, their best performance is better than our method which has not been trained on TruthfulQA. However, training on the validation set also limits the generalization of their method on other domains [1].

As TruthfulQA is a very challenging dataset for LLMs, zero-shot inference results in poor performance. Therefore, we follow previous work [2] to utilize different number of in-distribution prompts during inference time. The results show that the performance could be significantly improved when we increase the number of in-distribution prompts, which also explains why ITI performs good.

Please see the Appendix A (Page13) of our revised paper for the detailed results and analysis.

(2) In the main experiments, we evaluate the performance of different methods in LLaMA-7B, LLaMA-13B and OPT-6.7B. To demonstrate the robustness of our method across different models, we also provide the performance comparison in the recent LLaMA2-7B and Falcon-7B. The results in Table 2 reveal that our proposal consistently exhibits superior performance compared to the competitive methods across different LLMs.

Table2: Performance evaluation on LLaMA2-7B and Falcon-7B. AUC$_s$/AUC$_r$ score is reported.

For more information, please see the Appendix C (Page13) of our revised paper.

---

W2 The covariance of LLM internal states may contain distracting information (such as the diversity of expressions, styles, and terminology), it's hard to guarantee the generalization ability of the proposed method.

Ans The diversity of natural language expressions has long presented a challenge in evaluating neural language generation and detecting hallucinations. Our proposal, of course, cannot completely solve the problem. However, compared to existing uncertainty or consistency metrics, our proposal takes a step further in solving this problem: (1) It captures the semantic divergence (entropy) in the dense embedding space, which is expected to retain highly-concentrated semantic information. (2) The proposed EigenScore, which is determined by the LogDet of the sentence embeddings' Covariance, is more intuitive and effective compared to existing metrics that explores sentence-wise similarity

Besides, As presented in our experimental setup, we set the temperature t=0.5 through the experiments, which appropriately suppressed the diversity of LLMs' generations. In Fig. 4(a) (of our submission), we have presented the performance sensitivity to temperature t, which shows that our EigenScore significantly outperforms baseline methods when t<1， and when t>1 the performance decreases rapidly.

---

Reference

[1] TruthfulQA: Measuring How Models Mimic Human Falsehoods

[2] Training a helpful and harmless assistant with reinforcement learning from human feedback

W4 Beyond hallucination detection, is that possible to mitigate hallucination through the proposed framework eventually?

Ans In most studies, the hallucination detection and mitigatition are two independent modules [3]. Most hallucination mitigatition methods can be categorized into prompting-based, external knowledge based and training/model-editing based methods [9]. In this study, we mainly focus on the hallucination detection. However, the hallucination content can be appropriately suppressed by the well-established prompting-based or external knowledge based approaches after identifying the hallucination content with EigenScore. For exapmle, in [7], the authors use a simple prompt template to correct the detected self-contradiction hallucination.

In future work, we will delve into how to effectively mitigate hallucinations with our EigenScore.

---

Q1 How about the proposed method compared with this work [1] on the TruthfulQA dataset? Both work relies on the internal state of the LLM models.

Ans Considering that replicating the ITI [1] would be prohibitively expensive, we test the performance of our proposal and comparison baselines in the TruthfulQA dataset. The results are presented in Table 2. For the ITI, we report the best performance in their paper. As can be seen, our proposal consistently outperforms the baseline methods and achieves comparable performance as ITI when we utilize 50 in-distribution prompts.

Table 2: Performance comparison of different methods on TruthfulQA dataset. Accuracy is reported.

It's worth nothing that the ITI [2] relies on training 1024 binary classifiers in TruthQA datasets. Therefore, their best performance is better than our method which has not been trained on TruthfulQA. However, training on the validation set also limits the generalization of their method on other domains [2].

As TruthfulQA is a very challenging dataset for LLMs, zero-shot inference results in poor performance. Therefore, we follow previous work [10] to utilize different number of in-distribution prompts during inference time. The results show that the performance could be significantly improved when we increase the number of prompts, which also explains why ITI performs good.

For more information about the experiments, please see the Appendix A (Page13) of our revised paper.

---

Q2 How about the effect of sentence embeddings on the final performance?

Ans We have performed this ablation study in Page8 - How EigenScore Performs with Different Sentence Embeddings. The ablation results presented in Fig.3 (b) shows that using the sentence embedding in the shallow and final layers yields significantly inferior performance compared to using sentence embedding in the layers close to the middle. Besides, another interesting observation is that utilizing the embedding of the last token as the sentence embedding achieves superior performance compared to simply averaging the token embeddings, which suggests that the last token of the middle layers retain more information about the truthfulness of generations.

---

Q3 Can you provide the computational cost estimation of the whole framework and which type of resources you are using?

Ans Please refer to our response to W2. We have provided a detailed Computational Efficiency Analysis in Appendix D (Page14) of our revised paper. All experiments are performed on the NVIDIA-A100 GPU.

---

Reference

[1] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model.

[2] SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models.

[3] Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation.

[4] Shifting attention to relevance: Towards the uncertainty estimation of large language models.

[5] Survey of Hallucination in Natural Language Generation.

[6] On Hallucination and Predictive Uncertainty in Conditional Language Generation.

[7] Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation.

[8] Representation engineering: A top-down approach to ai transparency.

[9] Insights into Classifying and Mitigating LLMs' Hallucinations

[10] Training a helpful and harmless assistant with reinforcement learning from human feedback.

Comment We thank Reviewer Kmk8 (R2) for the careful reviews and insightful suggestions, which helped us improve our paper.

W1 One shortcoming of the proposed method is the applicability to blackbox LLMs. The method relies on the internal state of the model, e.g., LLaMA and OPT, but the authors may not discuss the limitation well.

Ans The reliance on white-box LLMs is a limitation of our work, and thanks for your suggestion, we have discussed the limitations of our work in the revised paper (Appendix G, Page 16). In addition, black-box based [1,8] and white-box based [2,3] methods are equally important for different application scenarios for LLMs. Compared to the methods designed for black-box methods, exploring the internal information for hallucination detection has its own advantages: (1) The internal states retain highly-concentrated semantic information; (2) Exploring the sentence embedding in the internal states is much more efficient compared to extracting the sentence embedding with an additional large model [2,3].

---

W2 The computational cost is another concern since essentially this method is a sampling-based approach. How to further reduce the cost for real production is still unsolved.

Ans One limitation of our work is that it requires additional computational overhead，which is also the limitation for all consistency-based [2] or prompting-based methods [7]. We have discussed this limitation in our revised paper (Appendix G, Page16).

Besides, we also compare and analysis the computational overhead of our proposal with the base LLM and other compared baselines. The average inference time (second per question) is computed on one NVIDIA-A100 GPU and shown in Table 1. As observed, our EigenScore is about 10 times more efficient than the methods that rely on another large model to measure the self-consistency (such as SelfCheckGPT [2]), and shares the similar computational overhead with the LN-Entropy and Lexical Similarity. It is worth noting that the inference overhead required to generate multiple results is not linearly proportional to the time of generating a single output, owing to the sampling and decoding strategy of the autoregressive LLM model.

Table 1: Inference cost comparison of different methods in LLaMA-7B and LLaMA-13B.

Compared to the computational overhead of generating multiple outputs, the cost of feature clipping and EigenScore computation is negligible (0.06s). For more details about the Computational Efficiency Analysis, please see the Appendix D (Page14) of our revised paper.

---

W3 The proposed method is mainly inspired by uncertainty estimation but how to bridge the connection with hallucination detection is still unclear, even though the results look promising.

Ans LLMs' knowledge hallucinations primarily arise from uncertainties regarding the source information or uncertainties inherent in the decoding process [5]. Therefore, exploring the uncertainty [5,6] or self-consistency [2,7] of LLMs' generations is one of the most promising direction for effective hallucination detection. According to [7], the self-contradictory hallucinations is one of the most important form of hallucination, and 67% inconsistent generation indicates hallucination. Therefore, we suggest that our proposal exploring semantic consistency with LLMs' internal information is potentially a promising approach for hallucination detection.

---

Reference

[1] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model.

[2] SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models.

[3] Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation.

[4] Shifting attention to relevance: Towards the uncertainty estimation of large language models.

[5] Survey of Hallucination in Natural Language Generation.

[6] On Hallucination and Predictive Uncertainty in Conditional Language Generation.

[7] Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation.

[8] Representation engineering: A top-down approach to ai transparency.

[9] Insights into Classifying and Mitigating LLMs' Hallucinations

[10] Training a helpful and harmless assistant with reinforcement learning from human feedback.

Comment We thank the Reviewer VqqH (R1) for the insightful feedbacks.

W1 The related work comparison is not comprehensive. There is no related work comparison between the internal-state detection approach. It might be better if the authors can compare their approach with the internal state-based hallucination detection approaches, e.g., [1].

Ans Thanks for your suggestion. Considering that replicating the ITI [2] would be prohibitively expensive, we test the performance of our proposal and the baselines in the TruthfulQA dataset [1]. The results are presented in Table 1. For the ITI, we report the best performance in their paper. As can be seen, our proposal consistently outperforms the baseline methods and achieves comparable performance as ITI when we utilize 50 in-distribution prompts.

It's worth nothing that the ITI [2] relies on training 1024 binary classifiers in TruthfulQA datasets, and they report the best performance (83.3) in the validation set. Therefore, their best performance is better than our method which has not been trained on TruthfulQA. However, training on the validation set also limits the generalization of their method on other domains [2].

Table 1: Performance comparison of different methods on TruthfulQA dataset. Accuracy is reported.

As TruthfulQA is a very challenging dataset for LLMs, zero-shot inference results in poor performance. Therefore, we follow previous work [6] to utilize different number of in-distribution prompts during inference time. The results show that the performance could be significantly improved when we increase the number of prompts, which also explains why ITI performs good.

Besides, to demonstrate the effectiveness of our proposal, we also compare our proposal with several competitive methods [3,4,5] that explore self-consistency for hallucination detection. The results in Table 2 demonstrate that our EigenScore consistently outperforms the the competitors by a large margin.

Table 2: Performance comparison of EigenScore and and several competitive methods on CoQA dataset. AUC$_s$/AUC$_r$ score is reported.

For more information about the experiments, please see the Appendix A/B (Page13) of our revised paper.

---

W2 The llm models involve multiple transformation blocks; which block internal state do the authors investigate? It would make the paper more impactful if the authors could present the ablation on transformation blocks.

Ans We have demonstrated the performance of exporing EigenScore in different layers (transformer blocks) in Fig. 3(b). And the detailed discussions are illustrated on Page8 in our submission. The results show that using the sentence embedding in the shallow and final layers yields significantly inferior performance compared to using sentence embedding in the layers close to the middle. Besides, another interesting observation is that utilizing the embedding of the last token as the sentence embedding achieves superior performance compared to simply averaging the token embeddings, which suggests that the last token of the middle layers retain more information about the truthfulness of generations.

---

Reference

[1] TruthfulQA: Measuring How Models Mimic Human Falsehoods

[2] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

[3] Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models

[4] Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

[5] Shifting Attention to Relevance: Towards the Uncertainty Estimation of Large Language Models

[6] Training a helpful and harmless assistant with reinforcement learning from human feedback

Dear Reviewers:

Thank you for your valuable and constructive feedbacks, which have inspired further improvements to our paper. As a gentle reminder, it has been more than 2 days since we submitted our detailed responses and revised paper. As the rebuttal deadline is approaching, we would greatly appreciate knowing if our response adequately addressed your concerns. We are looking forward to your response and are happy to answer any future questions.

Best,

Authors of #5370