Chain-of-Verification Reduces Hallucination in Large Language Models

Thanks for the review and the positive comments on our approach’s ease of implementation, on the breadth of our experimentation, and the clarity of our draft!

We have responded to your concerns below.

Inference overhead

Please check the common response. We also provide further comparative analysis in section 8 in our appendix.

Few-shot examples

We have now added the few-shot prompts used in our supplementary material for reproducibility. We agree that changing the few-shot prompts can lead to a variance in the final output. In our comparative analysis, we use the same prompts to generate the baseline response for all our baseline approaches.

Questions:

1. Few-shot prompts:
   1. Our few-shot prompts for the list-based QA benchmarks were randomly chosen from the train set. We however keep these prompts constant across our variants.
   2. For the biographies dataset, we choose some of the most viewed people's pages on Wikipedia and craft our prompts based on their biographies. We find few-shot examples that allow us to show the model our desired verification responses. We have added the few-shot prompts used to the supplementary material.
2. We perform an analysis of the inference overhead. Please see the common response and Section 8 in our appendix.

Thanks for the review and for finding our draft well-written. We are grateful for your positive comments on the value and efficacy of our approach. We have addressed your concerns and answered your questions below

Concerns:

1. We have now added three new state-of-the-art baselines to the paper. Our work deals with hallucinations specifically for longform generations. We have thus updated Table 3 with baselines using SelfCheckGPT (NLI and LLM) and ChatProtect - three state-of-the-art papers that deal with hallucinations in longform generations. We find our approach, CoVe, outperforms both SelfCheckGPT (with NLI and LLM) and ChatProtect. Please see our common response for more details.

2. We add our few-shot prompts in the supplementary material.

3. Our approach is intended for mitigating hallucinations in longform generations, a highly relevant problem. As LLMs are used more and more as replacements to search engines, it is crucial that we show CoVe reduces hallucinations across 4 generation benchmarks. We are optimistic that CoVe style verifications can be extended to generative models on real applications, which is beyond the scope of this paper.

Questions

1. Like several reasoning tasks, it is hard to measure the performance of the intermediate parts of the chain owing to the absence of labels, although clearly you can measure the end-to-end performance, or hand annotate some data. For the Wikidata task we manually measure the accuracy of the verification questions and we find that the Llama 65B is around 70% accurate at answering factoid questions related to the professions and birth-places of people. We describe this in the paper under Shortform verification questions are more accurately answered than longform queries.
2. That’s an interesting approach to try. For some tasks, it is not likely to work as you will not know which questions to ask, e.g. consider list-based questions (e.g. “who are some politicians born in NYC“), the only question you can really ask at the start is the list-based question itself.
3. We had added our few-shot prompts in the supplementary material
4. We perform a comparative analysis of the inference overhead encountered. Please check the common response and Section 8 in our appendix.

We thank you for your review and are appreciative of your positive comments on the applicability of our approach and our experimentation!

We have addressed your concerns below

Technical contribution

The paper “Measuring and Narrowing the Compositionality Gap in Language Models” (which is cited) as the title suggests concentrates on compositional questions with multiple hops such as “Who won the Master’s Tournament the year Justin Bieber was born?” – and breaking into subquestions. It does not focus on hallucinations (does not mention that word at all). It uses a method called self-ask to build this subquestion decomposition. In our work questions are used for a different purpose: to verify hallucinations, rather than decompose a multi-hop question, so this is different. Further, we develop joint, 2-step, factored and factor+revise approaches and compare these and show how they are important – none of which is related to this paper.

Wikidata + Wiki-Category datasets

We would like to clarify that we do not create the Wiki-Category benchmark ourselves but use a subset of an existing benchmark: Quest (https://aclanthology.org/2023.acl-long.784.pdf). Wikidata uses real data from Wikipedia, but the queries are synthetically constructed. We also report results on the MultiSpanQA task which does use real queries from humans (from Google). Overall, list-form answers are very natural in assistant-based applications. List-generation questions can be excellent ways to understand LLM hallucinations and rapidly test approaches. List generations mimic longform generations with the advantage that they can be automatically, efficiently and perfectly evaluated, while in other tasks evaluation is much more tricky. We agree that questions of the form “Name some politicians born in Boston” appear deceptively trivial but large-scale commercially deployed LLM systems fail in producing correct answers. We have provided several qualitative examples of this in the supplementary material.

Set of tasks used

Hallucinations mitigation in longform generations has very recently started receiving significant attention and a lot of benchmarks don’t yet exist for this problem. We chose these four tasks because (i) they are known to produce hallucinations for state-of-the-art models (as shown in experiments), both open source and commercial products; and (ii) ways of evaluating these tasks exist. In particular, the FactScore metric was recently built for the biography task, and hence several other papers, other than ours, are biographies as well (see new baselines). We agree that it would be interesting to see how CoVe would perform on further benchmark tasks as they are developed. Scientific, legal and medical domains are great applications and will warrant deep analysis beyond a methods paper such as this one. Hence, this is exciting future work beyond this paper.

Number of facts vs. number of hallucinations

Any method that reduces hallucinations will reduce the number of facts. As we note in our research questions, we are interested in whether CoVe can selectively remove fictitious facts with a minimal impact on the number of truthful facts. The FactScore metric is normalized by number of facts. So a shorter generation does not automatically mean fewer hallucinations. Note that we also built a baseline whereby the number of facts is reduced by chopping off the ends of passages to match the same number of facts as our best model and we see CoVe significantly outperforms this baseline. We describe this in our passage: CoVe outperforms existing hallucination mitigation baselines.

Questions

Our work along with several prior works, eg SelfCheckGPT, ChatProtect have shown that it is possible to mitigate hallucinations without retrieval. We agree that it is impossible to guarantee veracity without access to a verified source of information. We would argue that even retrieval as shown by several Retrieval Augmented Generation (RAG) style approaches still cannot guarantee this without assuming that everything on the internet is true. We also would like to note that errors can still creep into the RAG pipeline for example the PerplexityAI result from our paper which uses a retrieval system.

Thank you for your feedback and we are happy you found our draft clear and easy to follow. We agree that hallucinations are an important and interesting problem and we are glad you found our approach well-motivated.

We have responded to your comments below

Absence of necessary baselines:

We have added three new state-of-the-art baselines to the paper. Our work deals with hallucinations specifically for longform generations. We have thus updated Table 3 with baselines using SelfCheckGPT (NLI and LLM) and ChatProtect - three state-of-the-art papers that deal with hallucinations in longform generations. We find our approach, CoVe, outperforms both SelfCheckGPT (with NLI and LLM) and ChatProtect.

Difficulty of direct comparison

Llama65B is not an instruction-tuned model and hence produces gibberish (without stopping) when prompted zero-shot. This is expected behavior for a base language model, hence we do not report it (we will clarify this in the paper). When prompted zero-shot with “Let’s think step-by-step”, the model generated erroneous steps, for example, to check information on the web instead of generating the required answer, and did not produce answers. We have updated our appendix on the unsuccessful experiments of getting Llama 65B to generate the desired response when prompted zero-shot. In any case, these serve as baselines to CoVe rather than being interesting in themselves to compare. The CoVe experiments are also few-shot using Llama65B, hence the baseline few-shot Llama65B comparison. Llama 2 (which should be superior) is there as a baseline just to show that zero-shot instruction-tuned model baselines do not actually fare better.

Inconsistency across Tables

The Factor+revise method adds an additional step where the LLM can check for inconsistencies between the generated content and the answered verification question.

Consider the example

      “Trump received a BS in Business from the University of Pennsylvania” 
       And the verification question and output:

      What degree did Donald Trump graduate with from the University of Pennsylvania in 1968? 
      Answer: A BS in Economics. 

The Factor+revise method adds a separate step that has the LLM check if these two outputs are consistent, inconsistent or partially consistent and then recomposes the output based on the consistent parts.

For the set QA tasks checking for inconsistencies and removing an answer from the list is more trivial, as it is just an entity match.

Example:

     Name some politicians born in New York, NY
    A: Hillary Clinton

    Where was Hillary Clinton born?
    A: Chicago (Chicago != New York)

CoVe - 2 step is a variant of CoVe-Factored where the Verification questions are evaluated in one step (thus reducing the need to prompt the LLM several times for each verification question). This is feasible to use on tasks such as set QA but as the number of verification questions grows answering them all can be difficult in the fixed context size. We hence don’t experiment with this approach on the harder benchmarks.

More qualitative examples

Figure 1 and 3 show full examples of CoVe. There is also analysis in section 9 of how and why ChatGPT fails on a number of tasks, and why verification questions can work. Finally, we have now added the prompts used as part of our supplementary material