HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection

We thank you for recognizing our work as interesting and for studying an important challenge. We appreciate the reviewer's comments and suggestions, which we address below:

A1. The effect of adding more unlabeled data

Thank you for the suggestion! Our ablation on the number of unlabeled data for hallucination detection in Appendix Table 8 indeed shows that scaling up the unlabeled data is beneficial. To further verify this, as suggested, on the TruthfulQA dataset, we have explored (1) adding LLM-generated sentences for a specific concept from the WikiBio dataset [1] into the unlabeled data and (2) adding LLM generations for a different TriviaQA dataset into the unlabeled data. The hallucination detection results with different numbers of added samples are shown as follows (the model is LLaMA-2-7b and the test dataset is TruthfulQA):

We observe a similar trend as in Table 8 of our paper and will be happy to add more discussion on this in the revised version.

A2. Discussion on robustness comparison

Thank you for pointing this out! Firstly, we concur with the reviewer's opinion that our approach can be affected by the distribution shift between the unlabeled data and the test data, which we also discuss in the Limitation section of our paper.

Although we did not explicitly claim the better robustness of our approach compared to other alternatives, we are happy to clarify the method robustness and its relationship with dataset similarity. Specifically, the four datasets used in our paper have the following domain differences:

• TruthfulQA: This dataset includes questions from various domains such as health, law, fiction, conspiracies, etc. It is designed to measure the truthfulness of language models by including questions that some humans might answer falsely due to misconceptions.
• TriviaQA: This dataset includes question-answer pairs from trivia enthusiasts. The questions cover a wide range of topics and are paired with evidence documents from sources like Wikipedia and web search results.
• CoQA: This dataset includes passages from seven diverse domains: children’s stories, literature, middle and high school English exams, news, Wikipedia, Reddit, and science.
• TyDiQA: This dataset consists of questions related to Wikipedia articles.

Based on this and Figure 3(a) of our paper, we have made the following observations:

The knowledge scope matters for method transferability: We find that a hallucination subspace calculated within a more general knowledge scope can transfer well to data with a smaller knowledge scope, but not vice versa. For example, the subspace extracted from TruthfulQA (where the knowledge scope is more restricted compared to other datasets) demonstrates less generalization ability to other datasets. The detection AUROC drops from 76.42% to 67.81% when the TruthfulQA subspace is used for hallucination detection on the CoQA test set, while the CoQA subspace achieves an AUROC of 77.36% on the TruthfulQA test set (only a 1.28% decrease). Additionally, our approach works well when the unlabeled data and the test data have a similar knowledge scope. For instance, the TriviaQA and CoQA datasets have similar knowledge scopes (Wikipedia and other web knowledge), allowing the subspace learned from one dataset to generalize to the other, as verified by our experiments.

We believe this explains why, in certain cases, our approach is robust when transferring to other data distributions. We would be happy to add this discussion to our revised paper.

A3. Explanation on the hyperparameter

Great point. As explained in line 203 of our paper, we determine the threshold $T$ on a separate validation set consisting of 100 LLM generations. For the LLaMA-2-7b model with the TruthfulQA dataset, we provide further ablation results of $T$ on the test set as follows. Here, "max" and "min" denote the maximal and minimal membership scores of the unlabeled data.

The final reported result (78.64%) in our main Table 1 is selected based on the AUROC on the validation set.

A4. Visualization

Another great point. As suggested, we provide visualization results here on the embeddings of the truthful and hallucinated LLM generations for the TyDiQA-GP dataset with the LLaMA-2-7b model. These embeddings are almost linearly separable and align well with empirical observations in the literature [2].

[1] Manakul et al., SELFCHECKGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models, EMNLP 2023

[2] Zou et al., Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023.

We are deeply encouraged that you recognize our method to be novel, welcome, and beneficial to the research community.

Your summary and comments are insightful and spot-on :)

A1. Clarification on the problem setup

You raise a great point! We agree with you that the truthfulness of an LLM generation should be dependent on the given user prompt. We plan to revise it as follows: define a new variable $\widehat{\mathbf{x}}$ that denotes the concatenation of the user prompt and the LLM generation, and the distribution P_unlabeled, P_true, P_hal are thus defined over the new variable. Moreover, that will also address your confusion on the assumption that hallucinated data is generated with probability $\pi$ independent of the input. Thank you for catching this issue!

A2. Clarification on the thresholds

Thank you for the question! We are happy to clarify their differences. $T$ is the threshold when determining the membership of the unlabeled data (true or false) using the membership estimation score (Equation 7 of our paper), which can be chosen on a small amount of validation data as discussed in Section 4.1 of the paper. $\lambda$ is the threshold on the probabilistic outputs from the hallucination classifier during test time to determine whether a testing LLM generation is hallucinated or not.

A3. Mention distribution shift in the conclusion

Certainly! We will be happy to discuss this future work in the conclusion section. Possible extension ideas can include firstly setting up new experimental environments where the unlabeled data and the test data belong to different domains (such as between daily dialogues and medical question-answer pairs), and then developing a distributionally robust algorithm for training. Another interesting idea to explore might be out-of-sample extension for SVD [1] that specifically deals with the reconstruction and projection precision for samples that are not in the training set.

[1] Bengio et al., Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering, NIPS 2003

G({x_\text{prompt}, x}) = \begin{cases}
    1, &\text{if }  {(x_\text{prompt}, x)} \sim P_\text{true} \\\\
    0,         &otherwise
\end{cases}

P_{\text{unlabeled}} = (1-\pi) P_{\text{true}} + \pi P_{\text{hal}},

We are glad to see that the reviewer recognized the strengths of our work from various perspectives. We thank the reviewer for the thorough comments and suggestions. We are happy to clarify as follows:

A1. Clarification on the BLEURT metric

Thank you for pointing this out! BLEURT [1] is designed to evaluate the quality of text by comparing it to reference texts, similar to traditional metrics like BLEU. However, BLEURT leverages pretrained Transformer models, such as BERT, to capture deeper semantic nuances and contextual meanings. This makes it particularly effective for detecting subtle discrepancies between generated outputs and reference ground truths, which is critical in identifying hallucinations that may not be immediately obvious through surface-level evaluation.

In addition, hallucinations can vary widely in form, from subtle factual inaccuracies to more blatant falsehoods. BLEURT's embedding-based approach allows it to handle this variability better than traditional n-gram-based metrics, which might miss nuanced errors. This robustness ensures that the evaluation can effectively capture both major and minor hallucinations, providing a more comprehensive assessment of the model's output quality.

Finally, BLEURT has been shown to correlate well with human judgments in various natural language generation tasks [1, 2], which makes it a reliable proxy for assessing the factual correctness and coherence of LLM outputs, which is essential in hallucination detection. Additionally, we show that the effectiveness of our algorithm is robust under a different similarity measure, ROUGE, in Appendix D, which is based on substring matching.

A2. Effectiveness of our approach on additional tasks

Thank you for the suggestion! We evaluate our approach on two additional tasks, which are (1) text continuation and (2) text summarization tasks.

For text continuation, following [1], we use LLM-generated articles for a specific concept from the WikiBio dataset. We evaluate under the sentence-level hallucination detection task and split the entire 1,908 sentences in a 3:1 ratio for unlabeled generations and test data. (The other implementation details are the same as in our original submission.)

For text summarization, we sample 1,000 entries from the HaluEval [3] dataset (summarization track) and split them in a 3:1 ratio for unlabeled generations and test data. We prompt the LLM with "[document] \n Please summarize the above article concisely. A:" and record the generations while keeping the other implementation details the same as the text continuation task.

The comparison on LLaMA-2-7b with three representative baselines is shown below. We found that the advantage of leveraging unlabeled LLM generations for hallucination detection on Wikipedia articles still holds.

A3. Details of Table 3

Absolutely! As in the first row of Table 3, we use 4 datasets (TruthfulQA, TriviaQA, CoQA, and TyDiQA-GP) to ablate on the effect of where the embedding is extracted for hallucination detection. The results are the detection AUROCs based on different locations in a typical transformer block to extract the embeddings, i.e., the output of each transformer block, the output of the self-attention module, and the output of the MLP feedforward layer. From Table 3, we observe that the LLaMA model tends to encode the hallucination information mostly in the output of the transformer block, while the most effective location for OPT models is the output of the feedforward layer.

[1] Sellam et al., BLEURT: Learning Robust Metrics for Text Generation, ACL 2020.

[2] Bubeck et al., Sparks of Artificial General Intelligence: Early experiments with GPT-4, arXiv preprint, 2303.12712.

[3] Li et al., HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models, EMNLP 2023.

We thank the reviewer for the comments and suggestions. We are encouraged that you recognize our approach to be clear and with interesting experiments and thorough ablations. We address your questions below:

A1. Effectiveness of our approach on additional tasks

Thank you for the suggestion! We evaluate our approach on two additional tasks, which are (1) text continuation and (2) text summarization tasks.

For text continuation, following [1], we use LLM-generated articles for a specific concept from the WikiBio dataset. We evaluate under the sentence-level hallucination detection task and split the entire 1,908 sentences in a 3:1 ratio for unlabeled generations and test data. (The other implementation details are the same as in our original submission.)

For text summarization, we sample 1,000 entries from the HaluEval [2] dataset (summarization track) and split them in a 3:1 ratio for unlabeled generations and test data. We prompt the LLM with "[document] \n Please summarize the above article concisely. A:" and record the generations while keeping the other implementation details the same as the text continuation task.

The comparison on LLaMA-2-7b with three representative baselines is shown below. We found that the advantage of leveraging unlabeled LLM generations for hallucination detection still holds.

A2. Discussion on access to internals of proprietary, cloud hosted LLMs

You raise a great point. We concur with your opinion that access to the internal representations is not easy for proprietary, cloud-hosted LLMs. We provide our understanding on this as follows:

• Firstly, we believe that hallucination detection is still an ongoing research topic, and the research efforts on exploring the internal representations of open-sourced models are equally, if not more, important than the research assuming black-box access to LLMs. Such access to model internals is beneficial for transparency and debugging, which can help us understand where, when, and how the hallucinated generations occur. By investigating specific layers or components where the model's reasoning deviates, researchers and developers can debug and refine the model more effectively, leading to improvements in both performance and safety.
• Secondly, internal representations capture the nuanced, multi-layered information that LLMs process as they generate responses compared to the textual outputs. By analyzing these representations, we gain access to a more detailed understanding of the model's internal decision-making process. This granularity allows us to identify potential hallucination subspaces, leading to better hallucination detection performance compared to existing black-box approaches, such as SelfCKGPT [1], Self-evaluation [3], etc., which are already compared in our submission (Table 1).
• Finally, we may consider some strategies to mitigate the concerns about the proprietary nature of cloud-hosted LLMs, such as experimenting with proxy models that closely approximate the behavior of the proprietary LLMs. This can help study and detect hallucinations without infringing on intellectual property rights.

Thank you for bringing this up with us! We will include the discussion in the Limitation section of our submission.

A3. Discussion on subspace primary vectors and the layer-wise variations

You raise a great point!

Firstly, subspace primary vectors derived through SVD often encapsulate the dominant patterns and variations within the internal representations of a model [4]. These vectors can highlight the primary modes of variance in the unlabeled data, which are not purely random but instead capture significant structural features of the model’s processing. In our case, it could be the hallucination information. Even though these vectors could, in theory, capture various features, they are particularly informative for detecting hallucinations because hallucination and truthfulness patterns are among these primary modes of variation in the unlabeled data. This phenomenon can be verified by the empirically observed separability in both our submission (Figure 7 in Appendix) and literature [5]. In addition, we provide visualization results here on the embeddings of the truthful and hallucinated LLM generations for the TyDiQA-GP dataset on the LLaMA-2-7b model, which are almost linearly separable. We believe that this can help illustrate the effectiveness of projection against the top singular vectors.

Moreover, for the variation with respect to the layers, we select the layer based on a small number of validation data as described in Section 4.1 of our submission.

A4. Incorporating training objective on certain subspaces

Another great point! We do anticipate the possibility of explicitly regularizing the training of LLMs for better hallucination detection. One straightforward idea is to add an objective that fine-tunes the LLMs (or their feature subspace) to clearly distinguish the true vs. false based on the representation space and the separated unlabeled data at each training step, and then keep training for a few epochs. This could be a promising future work.

[1] Manakul et al., SELFCHECKGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models, EMNLP 2023

[2] Li et al., HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models, EMNLP 2023.

[3] Kadavath et al., Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.

[4] https://en.wikipedia.org/wiki/Principal_component_analysis

[5] Zou et al., Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023