Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation

We would like to thank reviewer a7zQ for providing insightful feedback. We hope that our rebuttal has addressed all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.

Why does your mitigation even work at all?

Our mitigation approach does not correct hallucinations. Instead, it removes contradictory information that already exists in the two sentences. Our removal can be achieved by comparing the two sentences and does not rely on external knowledge. For instance, in Figure 3, the two sentences provide conflicting information about the birth date for William T. Freeman. This inconsistency can be removed without knowing Freeman’s actual birth date, by dropping the information about the birth date entirely from the sentence. It is rare but not impossible that LLMs introduce a new hallucination when generating the revised sentence. Since our mitigation algorithm is iterative, these new hallucinations are resolved in subsequent iterations.

Our removal approach offers a novel perspective on mitigating hallucinations compared to the traditional correction approach. The motivation is to prevent LMs from generating mistakes when they are uncertain about specific facts. It aligns well with machine learning approaches that incorporate a reject or abstention option [2].

How do the annotators manually verify contradictory information using Wikipedia?

The annotators read relevant Wikipedia articles and perform verification through a combination of human understanding and text searches. To enhance coverage, they are asked to examine various articles: (i) articles of all entities in the contradictory sentences; (ii) articles of relevant entities mentioned in (i); (iii) If non-English articles provided more information than English counterparts, the annotators used Google Translate to review non-English versions translated to English. To improve annotation accuracy, each piece of contradictory information must be examined by two annotators independently. Then, the two annotators discuss to resolve inconsistent cases and reach final decisions.

We believe that the above protocol has resulted in high-quality verification annotations. We will incorporate this discussion into the paper.

UPDATE: As suggested by reviewer 3teD, we additionally ask the annotators to use web search and examine broader online text. However, since Wikipedia already has a high coverage, the unverifiability numbers do not change significantly and our claims still hold.

Do you consider the information extraction system as external grounded knowledge?

No. First, the information extraction system CompactIE does not retrieve any additional, external knowledge at any point. It only sees the sentences generated by gLM as input and outputs fact triples that are guaranteed to be spans of the input sentences. For example, from the sentence “X does Y”, it would extract the triple (“X”, “does”, “Y”). Second, CompactIE is based on a small BERT model with 110M parameters [3]. In contrast, external grounded knowledge typically refers to input-dependent text retrieved from considerably larger knowledge bases, such as Wikipedia. We will update the paper to clarify this.

For your work, “entity” seems a better term than “topic”.

We agree and will update the paper to replace “topic” with “entity”.

Can you provide experiments on PopQA [1], an existing entity-centric QA dataset?

Yes. We have provided this experiment in the global response. The results show that our method can accurately detect a large number of self-contradictions, for both vanilla and retrieval-augmented generation.

How is mitigation performance influenced by the value of n?

The initial submission includes an ablation study on this. Please refer to the paragraph “Ablation Study on Mitigation” in Appendix B. Our ablation study shows that each iteration progressively removes self-contradictions, while preserving fluency and informativeness. The mitigation converges quickly after three iterations, which is why we choose $n=3$.

The value of n seems to fluctuate based on the size of |x|.

We believe that there is a misunderstanding. The value of n is fixed at $n=3$ throughout all experiments presented in the paper and we do not intend to suggest at any point that n should be adapted based on |x|.

[1] When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. ACL 2023

[2] Machine Learning with a Reject Option: A survey. arXiv:2107.11277

[3] CompactIE: compact facts in open information extraction. NAACL 2022

We would like to thank reviewer gQAk for providing insightful feedback. We hope that our rebuttal has addressed all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.

Is aLM robust to another text generation task?

Yes. We have added an evaluation on question answering, with the PopQA benchmark [1] suggested by reviewer a7zQ. The results show that our method can accurately detect a large number of self-contradictions, for both vanilla and retrieval-augmented generation. More details can be found in the global response.

Can you provide evaluation on different temperature values, for both aLM and gLM?

Yes. We have provided this evaluation in the global response. The results show that across various temperatures (0, 0.25, 0.5, 0.75 and 1.0), self-contradictions are consistently prevalent and our method for detecting self-contradictions is robust.

Is the last highlight in paragraph Revise of Section 5 wrong?

We have re-checked it and are confident that it is correct. According to https://wiki.factsider.com/william-t-freeman/, Freeman’s birth country is indeed the USA. If the reviewer still disagrees, we kindly request the reason.

[1] When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. ACL 2023

We would like to thank reviewer 3teD for providing insightful feedback. We hope that our rebuttal has addressed all of the reviewer’s questions, are happy to provide more details, and look forward to the reviewer’s response to our rebuttal.

Limitation and significance of self-contradiction.

We agree with the reviewer that our work does not solve all types of hallucinations. In fact, we have explicitly stated this limitation in the initial submission (at the end of Section 7). However, we still believe our contribution is significant, because self-contradiction occurs frequently and our approach complements existing methods that retrieve external resources. More on the latter is provided in subsequent answers and the global response.

Can you extend your evaluation to another existing dataset?

Yes. We have added another evaluation on PopQA [1], a question-answering benchmark suggested by reviewer a7zQ. We have also performed human annotation for this experiment. The results show that our method can precisely detect a large number of self-contradictions, for both vanilla and retrieval-augmented generation. More details can be found in the global response and will be discussed in subsequent answers.

How comprehensive is your human-labeled evaluation dataset?

While having added more evaluation data, we would like to emphasize that our current human-labeled dataset is already comprehensive. It covers 4 different LLMs, meaning that we perform separate annotations for each LLM. To address sampling variance, we generate and annotate 3 descriptions for each entity. Furthermore, to improve quality, each annotation label results from a joint effort of 2 annotators.

Can you provide more analysis on the relationship between aLM and gLM?

Yes. The following table presents the detection results with all four combinations of ChatGPT and GPT-4 as gLM and aLM. We can observe (i) gLM and aLM should be different to achieve optimal F1 scores: ChatGPT-GTP-4 outperforms ChatGPT-ChatGPT, and GPT-4-ChatGPT outperforms GPT-4-GPT-4; (ii) Compared to ChatGPT, GPT-4 generates content that is more nuanced and more difficult to classify: in terms of F1 score, ChatGPT-ChatGPT outperforms GPT-4-ChatGPT, and ChatGPT-GPT-4 outperforms GPT-4-GPT-4.

How does your conservative mitigation approach position in ML literature?

Our approach for removing contradictory information offers a novel perspective on mitigating hallucinations compared to the traditional correction approach. The motivation is to prevent LMs from generating mistakes when they are uncertain about specific facts. It aligns well with machine learning approaches that incorporate a reject or abstention option [9].

If one of the answers is correct, it would be better to keep it instead of removing it.

We acknowledge the value of keeping the original answers, and indeed, we do have such a feature in our tool at https://iclr9113.com/. Users can easily access the original answers by clicking on the revised answer.

What is your argument on the relationship between your method and other methods that retrieve external resources?

The reviewer may have a misunderstanding of our paper, thinking that we advocate for an exclusive choice between our approach and methods that retrieve external resources. To clarify, our paper actually only argues that our approach complements retrieval-based methods. In other words, while these two approaches can be used alone, they can also co-exist and their combination can yield improved performance. For more details, please search for the word “complement” in our initial submission or examine the paragraph “Self-contradiction vs. Knowledge Retrieval”.

We fully appreciate and agree with the reviewer on the value of retrieval-based methods. However, it is equally important to recognize their limitations, which have been extensively discussed [1, 2, 3, 4]. First, retrieval can be infeasible or prohibitively expensive in certain domains. Second, the retrieved information can be insufficient, inaccurate, or misleading, due to various factors such as the quality of the knowledge base or the retriever’s imprecision. As shown in our paper, even a resource as comprehensive as Wikipedia falls short in verifying a large portion of self-contradictions. Therefore, our approach serves as a valuable complementary solution to address these shortcomings.

We have also strengthened the above argument experimentally with the PopQA benchmark [1], which can be found in the global response and covers self-contradiction detection on top of retrieval augmented model output.

We thank the reviewers again for their constructive feedback and eagerly anticipate their response to our rebuttal.

We have performed additional evaluation of Llama2-70B-Chat on the PopQA benchmark and have updated the corresponding part of the Global Response. Moreover, we have incorporated the entire rebuttal into our paper and submitted a rebuttal revision of the PDF, making the following changes:

• Our paper has seamlessly integrated our PopQA results, including a discussion on the instantiation of our approach to different text generation tasks such as question answering (Section 5) and the PopQA experiments (Section 6.2 and Appendix C). This integration has significantly improved the generality and practical usefulness of our work.
• Appendix B now includes the other two suggested experiments, specifically different choices of temperature values and an analysis of the relationship between aLM and gLM. These additions demonstrate the robustness of our approach.
• We have added or revised text to address all other review feedback, thereby enhancing the overall clarity, detail and coherence of the paper.
• Several minor adjustments have been made to improve the writing quality.