KnowHalu: Multi-Form Knowledge Enhanced Hallucination Detection

We are deeply grateful to the reviewer for their insightful and thorough feedback, and we appreciate the recognition of our work's significance! The suggestions and comments made for our work have significantly helped to improve its quality.

Q1: Open source the code and provide detailed reproducible instructions.

Thank you for recognizing our work! We guarantee that we will open source the code and provide detailed instructions for reproducing the results shown in the paper as clearly as possible once it is accepted.

Q2: While some latency comparisons are provided, a more thorough analysis of the computational overhead compared to simpler approaches would be valuable, especially for real-time applications.

Thank you for your insightful suggestion and for highlighting this important aspect. We fully recognize the critical role that latency plays in real-time applications and understand the necessity of balancing computational efficiency with detection accuracy.

Our approach is designed to prioritize robust detection, particularly in scenarios where minimizing hallucinations is paramount. In such cases, the slight increase in computational overhead is justified by the enhanced reliability our method provides. However, we also acknowledge that for applications requiring rapid initial detections with lower computational demands, simpler approaches may indeed be more suitable.

To address your feedback, we will include a more detailed analysis of the computational overhead in our revision. Thank you again for your valuable comment, which will help us improve the applicability of our work.

Q3: I would like the authors to test on a wider dataset with more sources of hallucinations or real sources of hallucinations (1000/500 given the limited size of the test set).

Thank you for your valuable suggestion. In our current work, for the QA task, we utilize 1,000 correct answers and 1,000 hallucinated answers. Similarly, for the summarization task, we have 500 accurate summaries and 500 hallucinated summaries. Our primary focus has been on knowledge-enhanced hallucination detection. To address your concern, we explored an additional popular WikiQA dataset [1] to achieve a broader evaluation. We selected questions with correct answers and complemented them with randomly sampled incorrect answers for the same questions within the dataset. This approach ensured that each question was paired with one correct and one incorrect answer, resulting in a final set of 857 questions with both correct and incorrect answers. For example, a sample QA pair from WikiQA is:

Q: How are glacier caves formed? A (correct): A glacier cave is a cave formed within the ice of a glacier. A (hallucinated): Glacier caves are often called ice caves, but this term is properly used to describe bedrock caves that contain year-round ice.

Here are the final results: With the Starling-7B model:

With GPT-3.5:

We directly adopted the aggregation hyperparameters used in the HaluEval dataset, which may lead to suboptimal performance for KnowHalu (Aggregation) due to distribution shifts. Nevertheless, as demonstrated with GPT-3.5, our method still provides an 18.55% improvement over the baselines.

Thank you again for your insightful feedback. We believe these additional evaluations strengthen our work and address your concerns effectively.

[1] Yang, Y., Yih, W. T., & Meek, C. (2015, September). Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 2013-2018).

Q5: Regarding the experimental results presented in Table 2, it is noteworthy that GPT-3.5 does not exhibit superior performance compared to other models. Isn't that somewhat counterintuitive?

Thank you for your insightful question and for pointing out the observed performance of GPT-3.5 in Table 2. We apologize for any confusion caused. There are two possible reasons why GPT-3.5 does not exhibit superior performance compared to other models like Starling-7B:

1. Competitive Performance: First, GPT-3.5 does not necessarily have to outperform all other models. As shown on the Arena Leaderboard, the arena score of GPT-3.5-Turbo-1106 is 1068, which is actually even lower than the arena score of the model Starling-LM-7B-alpha, which is 1088. This is consistent with the performance comparisons presented in our paper. Besides, in Appendix C, we also show that newer versions of GPT-3.5 also exhibit a decline in performance, which is also consistent with [1].

2. Handling of Requests and Query Behavior: As shown in Appendix D of our paper, GPT-3.5 occasionally rejects certain requests, especially those identified as containing inappropriate or potentially invasive content. While this behavior was not observed for other models, leading to a degradation in its overall performance. Additionally, GPT-3.5 tends to exhibit a more `lazy' approach during the step-wise querying process, which can further impact its effectiveness compared to other models. As a result, the average inconclusive rate of GPT-3.5 is approximately 5% higher than other models.

We appreciate your question and will incorporate additional clarifications in our paper to better explain why GPT-3.5 does not outperform other models. Your feedback helps us improve the clarity and comprehensiveness of our work.

[1] Chen, L., Zaharia, M., & Zou, J. (2023). How is ChatGPT's behavior changing over time?. arXiv preprint arXiv:2307.09009.

Q6: The overall framework appears to combine various manually designed prompting strategies with established fact-checking techniques. For example, the approach involves decomposing the question into smaller components and verifying each part through retrieval on a one-by-one basis. From your view, what's the difference between the current hallucination detection task and the original fact-checking task?

Thanks for the insightful question! We will articulate more clearly the motivations behind our methodology and the specific contributions, as outlined below:

1. Two-Phase Hallucination Detection:

We first introduce a two-phase detection strategy for both off-target and fabrication hallucinations. This distinction is crucial because off-target types, such as broad answers, are considered hallucinations despite being factually correct, and thus cannot be detected through standard fabrication detection methods. A separate detection stage is thus essential for these hallucinations before proceeding to factual checking. We will further expand on this motivation in Section 3.1 of our revision.

2. Off-target Hallucination:

Our framework first categorizes specific off-target hallucinations and treats their detection as an extraction task, significantly improving detection accuracy. For instance, incorporating this specialized detection led to an approximate 7~8% increase in accuracy across all three models tested, as demonstrated in Table 6. This underscores the importance of identifying and accurately addressing these subtle but critical types of hallucinations.

3. Fact-Checking Enhancements:

   • Diverse Knowledge Forms: We show that the form of knowledge (structured vs. unstructured) significantly influences the detection accuracy. For example, the Starling-7B model benefits more from non-structured knowledge, with a 3.70% improvement, while GPT-3.5 gains a 5.55% increase with structured knowledge. This underlines the importance of matching knowledge forms with a model’s inherent reasoning capabilities for enhanced fact-checking precision.

   • Query Formulation in Knowledge Retrieval: The formulation of queries (specific vs. general) during the knowledge retrieval stage has a significant impact on performance. This strategic approach to query formulation, aimed at fetching relevant and accurate knowledge for fact-checking, can lead to a further 3.50% improvement in results.

Together, these methodological innovations and strategic enhancements result in a notable 15~20% improvement in QA task performance. Our forthcoming revision will further elucidate these points, ensuring a coherent understanding of the motivations and insights our paper offers. We will make these discussions and contributions more clear in our revision!

We are deeply grateful to the reviewer for their thorough and insightful feedback. Their expertise and dedicated time have significantly contributed to improving the quality of this work!

Q1: The author claims in the abstract that they introduce a new category of hallucinations. However, it is noteworthy that prior research, such as HaluEval, has already categorized model responses into "unverifiable", "non-factual" and "irrelevant" classifications.

Thank you for your insightful comment. We apologize for any confusion regarding the categorization of hallucinations in prior work such as HaluEval.

In HaluEval, the authors indeed utilize "non-factual" and "irrelevant" categories to generate hallucinated data. However, during the evaluation, tasks—such as the QA task—they still simply employs a binary classification setting, i.e., just only determining whether an answer is a hallucination or not, and there is also no class for "unverifiable" category (while we have one category called “INCONCLUSIVE” for it).

In contrast, our work introduces a more nuanced categorization of hallucinations by refining these existing classes and incorporating additional categories. This finer-grained classification enables more accurate detection of different types of hallucinations. Specifically, our approach achieves a 15-20% improvement in detection accuracy over the original HaluEval framework. Furthermore, in scenarios where recall for hallucination is more important, we could just simply map all “unverifiable” cases to hallucinations, which would be aligned with the same binary classification setting, the improvement would then be 20-25% across all models over the original HaluEval on QA task.

To clarify, our abstract refers to the first introduction of these new categories to enhance hallucination detection precision. While we acknowledge that prior research like HaluEval has already identified broad categories, our contribution lies in making this categorization more detailed and developing specialized detection strategies for them. We appreciate the reviewer’s recognition of the importance of finer categorization for improved hallucination detection and will ensure that this distinction is more clearly articulated in the revised abstract. Thank you again for your valuable feedback!

Q2: Additionally, I am interested to know whether there is statistical data in Table 1 regarding the percentage of different types of questions present in actual LLM responses.

Thank you for the insightful observation! Here is the statistical data for hallucinated data in QA:

As we can see, aside from pure fabrication (which can be detected by fact-checking), there are still a lot of other types of hallucinations, i.e., off-target hallucination cases.

Dear Reviewer,

We want to follow up on our discussion here. In our work, we aim to provide a unified hallucination detection framework based on two stages, five steps including various forms of knowledge and query formulations. We also offer a clearer motivation and definition for 'off-target' hallucinations, including a mathematical formulation following your suggestion and Reviewer NS6F, which has been acknowledged by other reviewers. In short, an off-target hallucination is a type of response generated by a language model where the content may be factually correct but fails to directly address the specific question or prompt.

We have also carefully improved our work following your feedback and will extend our efforts to include more tasks, such as the dialogue task you proposed. Your suggestions and comments would be invaluable to us. Please let us know if you have further questions or suggestions, and we would be eager to discuss and further improve our work to help build a safer AI community.

If our revisions meet your expectations, might you consider raising the score? We greatly appreciate your effort and expertise in helping to polish our work!

Q3: Furthermore, it does not consider how data missingness or redundancy might affect the outcomes. It would also be valuable to explore whether data conflicts could influence the model's robustness.

Thank you for your insightful suggestion! We apologize for any confusion caused. Indeed, we have carefully considered the impact of data missingness and data conflicts in our analysis, as detailed on line 304 of the paper. Unlike the baseline models, which do not account for these scenarios, our approach introduces an additional judgment category called “INCONCLUSIVE.” This category represents the situations where the retrieved knowledge is insufficient to determine the correctness of a response.

To illustrate, here are the inconclusive rates for both the Starling-7B and GPT-3.5 models in the QA task:

Inconclusive Rates with Starling-7B Model:

Inconclusive Rates with GPT-3.5 Model:

As a result, the incorporation of inconclusive examples might initially seem to undermine the performance of our methods compared to baselines that only support binary classification (hallucinated or not). Since in scenarios where recall for hallucination is more important, we could just simply map all INCONCLUSIVE cases to hallucinations. This alignment with the same binary classification setting as the baselines could directly increase the average accuracy by an additional 5~10%.

We will ensure that this is more clearly explained in our revision and will include the detailed statistics for the INCONCLUSIVE category. Thank you for bringing this to our attention!

Q4: In Tables 4 and 5, does KnowHalu perform better in unstructured scenarios than in structured ones? However, doesn’t Table 3 suggest the opposite conclusion?

We apologize for any confusion regarding the performance differences observed in our tables. The primary factor is the model used: GPT-3.5 demonstrates superior performance in structured scenarios, while the open-source Starling-7b model excels in unstructured scenarios. Specifically, Tables 4 and 5 showcase KnowHalu's performance with Starling-7b, highlighting better results in unstructured contexts. Similarly, Table 3 presents results indicating that the Starling-7b model outperforms in unstructured scenarios, whereas the GPT model exhibits stronger performance in structured scenarios. Therefore, there is no contradiction here; the key difference lies in the models used. We will make this observation clearer in our revision and thank you for your valuable feedback!

Many thanks to the reviewer for the thoughtful and detailed feedback. The expertise and time invested in this work have been instrumental in enhancing its quality.

Q1: The analysis lacks a thorough examination of the RAG data sources. For instance, it does not address the specific impact of the knowledge base (KB) as a data source on the results.

Thank you for this insightful question! In our study, the selection of RAG data sources is carefully tailored to each specific task to ensure optimal performance. For the QA task, we utilize the Wikipedia database as our knowledge base because the questions in the dataset (i.e., HotpotQA[1]) are derived from Wikipedia articles. This alignment ensures that the KB is both relevant and comprehensive for effectively addressing the QA task. Similarly, for the summarization task, the knowledge base consists exclusively of the original documents that need to be summarized, providing direct and pertinent information necessary for generating accurate summaries. In our experiments, we consistently use the same knowledge base for both our proposed method and all baseline models.

Regarding your suggestion to explore the impact of varying knowledge bases, we fully acknowledge its value and consider it an important direction for future research. However, for QA tasks, Wikipedia Corpus currently stands as the only large-scale, high-quality open-source dataset available in academia. Utilizing Wikipedia dumps has also been a widely accepted standard in the RAG domain for QA tasks and is employed by leading methods and benchmarks such as WikiChat[2], FlagEmbedding[3], DPR[4], and KILT[5], etc.

So, while exploring different knowledge bases could provide additional insights, relying on Wikipedia as our sole knowledge source aligns with current best practices and ensures the reliability and comparability of our results. We appreciate your recommendation and will consider it for future studies to further enhance our analysis!

[1] Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 2369-2380).

[2] Semnani, S., Yao, V., Zhang, H., & Lam, M. (2023, December). WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 2387-2413).

[3] Zhang, P., Xiao, S., Liu, Z., Dou, Z., & Nie, J. Y. (2023). Retrieve anything to augment large language models. arXiv preprint arXiv:2310.07554.

[4] Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., ... & Yih, W. T. (2020, November). Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 6769-6781).

[5] Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., ... & Riedel, S. (2021, June). KILT: a Benchmark for Knowledge Intensive Language Tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2523-2544).

Q2: Section 3.2 provides a rather lengthy description of the methods. Streamlining this section would help improve overall clarity.

Thank you for your valuable feedback! We aimed to provide a detailed description of our methods in Section 3.2 to ensure clarity for readers who may be less familiar with the area. We understand that this level of detail might be lengthy for people quite familiar with the subject. In our revision, we will streamline this section to better balance detail and clarity. Thanks again for your helpful suggestion!