A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Actively Validating Low-Confidence Generation

We thank and sincerely appreciate the detailed comments by the reviewer.

1) A Detailed Discussion on Latency:

We note that our approach results in improvements in the form of reduced hallucinations and thus makes the model more reliable (in addition to all the other advantages specified in B.4); however, it comes at the expense of increased inference cost. We completely agree with the reviewer’s point that though high latency may not be a problem for all use cases, it would be important to have a detailed discussion on it. We have briefly discussed this point in Appendix B.5.1. Here, we provide a detailed discussion on it for all the steps with suggestions on their lower-cost alternatives. We will also include it in the revision.

Identifying Important Concepts: Firstly, we note the importance of this step because validating the correctness of the entire sentence at once is infeasible as a sentence can contain multiple different facets all of which can not be validated at once. In contrast, individually validating correctness corresponding to the concepts provides opportunities for accurately detecting incorrectness. Thus, if we skip this step and directly proceed to the validation step for the entire sentence then it will have limitations. For example, sentences like “​Steven Threet i​s best known for his time at the University of Michigan, where he was a three-year starter and led the Wolverines to a Big Ten Championship in 2008.” contain multiple facets that need to be validated separately because a single web search may not return all the information that is required to validate the entire correctness.

This step incurs the cost of inference in which the input is the instruction (provided in Table 3) and the sentence. We mention the benefits of “instructing the model” technique in Section 2.1.1.

We discuss other lower-cost alternatives for this step below: A simple yet efficient method is to leverage a relatively smaller LLM for this step. This is feasible because identifying the concepts is an “easy” step and even smaller LLMs are typically very effective in this. Moreover, even a more smaller model such as T5 can also be finetuned for this particular task which can considerably reduce the cost. Smaller models have low inference cost (both in terms of FLOPs and latency). Furthermore, the other techniques already discussed in the paper, namely Entity Extraction and keyword extraction are other lower-cost alternatives. Specifically, the KBIR model is built on top of RoBERTa architecture which is even more efficient. In summary, smaller models (smaller LLMs or task-specific finetuned models) can be utilized for this task to make it more efficient.

Calculating Model’s Uncertainty: This is not a resource intensive task as it just requires calculating the score from the logit values.

Creating Validation Question: Similar to the first step, creating a validation query for a concept is also a task at which even smaller models (that even have only a few million parameters) do quite well. A lot of existing research on question generation uses the T5 models. Creating a validation question using an LLM requires taking the instruction (filled with the concept) (Table 3) and the sentence as input.

Another cost-effective alternative for this step is to simply mask out the selected concept from the sentence and use it as the validation query for the web search. Though, it requires some heuristics to create an appropriate validation query (such as selecting only a window of tokens on both sides of the concept after masking as the validation query, this would be required because using the entire sentence would have many different facets, and web search may not return relevant results). This would definitely make it much more efficient but it will lose effectiveness in creating “high-quality” queries pertinent to the concept and thus may not result in slight degradation in the validation procedure.

Answering Validation Question & Mitigation: These steps are more costly than the others because they also take the retrieved knowledge as input. We note that these are crucial steps of the method. They can be made more efficient (though it will compromise the effectiveness) by combining them into a single step, i.e., validation and mitigation can be done using a single instructional prompt. However, we note that this is a relatively difficult task as compared to the previous steps and thus decomposing it into two individual steps provides better results. Thus, making this step more efficient will have tradeoffs with the performance.

Overall, these steps can be made more efficient (in terms of both computation cost and latency) using smaller LLMs or external task-specific tools. In contrast, the methodology highlighted in red in Figure 1 uses the same model for all the steps. Furthermore, we note that in resource-constrained applications, the suggested efficient alternatives can be utilized.

Thanks for your reply. Please find our responses below:

1) Experiments on Latency: During the rebuttal period, we have conducted an empirical analysis of the latency where we compare the latency of all the steps of the methodology. Figure 9 in the revised manuscript shows the comparison of latency of various steps (at a sentence level). We note that the latency of the mitigation step is low as it is only conditionally called for some sentences. We show the average mitigation latency for sentences on which it is called in the Mitigation$^*$ bar.
We conduct this study for 10 topics (i.e., 50 sentences) for the GPT-3.5 (text-davinci-003) model.

Comparison of Overall Latency with the Generation: The overall latency of the method is $2.58$ times that of the regular generation ($5354.20$ against $2071.69$).

Why the latency of the generation step is high? This is because for the later sentences, it also takes the context in the input.

Why the latency of validation is high? This is because validation procedure includes three steps (validation question creation, retrieval, and answering validation question). Furthermore, validation could be required for multiple concepts.

What does Mitigation* represent? Note that the mitigation step is only conditionally executed for some sentences. We show the average mitigation latency for sentences on which it is called in the Mitigation$^*$ bar.

5) Justification of selecting the instruction technique for the concept identification step: We studied randomly sampled 50 sentences. The instruction technique identified 155 concepts in total. It missed only 2 concepts (that too these missed concepts can only be loosely regarded as important in the context of the sentence). Beyond this, please let us know what additional experiment we can perform for this step.

6) Is this statement is solely based on the five examples shown in Table 14? No, this is an overall statement for all the instances; in Table 14, we have only shown some supporting examples.

7) Regarding Labeling Hallucinations: Yes, such an example will be labeled as a hallucination as this is an incorrect statement. Also, this can also be regarded as an example of propagation of hallucination.

10) Effectiveness of Retrieval alone: We conducted this study for the multihopQA and FalsePremiseQA tasks only; in these settings, it shows a fair and clearer comparison of the efficacy of the retrieval alone method with our active intervention approach.

We thank the reviewer for the detailed feedback and provide our responses below:

1) Discussion on Related Work: We thank the reviewer for sharing these related arxiv papers. We have included all these papers in the revised Related and Concurrent Work Section (Appendix A) and discussed the key differences. We would also like to respectfully mention that although there are key differences in our work, all these papers should be viewed as concurrent works to our paper as our work has been on arxiv within merely 1-2 months of these other arxiv papers.

2) Examples when the original response is completely hallucinated:

We first provide examples where the complete response is hallucinated and our approach is able to address that hallucination. Then, we discuss it quantitatively.

For the topic “Ron Meagher”, the model generated “Ron Meagher is a renowned entrepreneur and business leader who has made a name for himself in the business world.” which is completely hallucinated and this hallucination also propagates in the next sentences: “He is the founder and CEO of Meagher Enterprises, a successful business consulting firm that specializes in helping companies grow and succeed.”, “Ron Meagher was born in New York City and grew up in a family of entrepreneurs.”

After applying our method the result is “ Ron Meagher is an American musician best known as the bassist of the rock band The Beau Brummels. He was discovered by Autumn Records in 1964 when guitarist-songwriter Ron Elliott was putting the band together….” which is correct.

Similarly, for the topic “Twila Shively”, the model generated “Twila Shively is a renowned American artist and sculptor who has been creating art for over four decades." which is completely hallucinated. In this case also, the hallucination propagates in the generation.

After applying our method the result is “Twila Shively was an American competitive baseball player who played from 1945 through 1950 in the All-American Girls Professional Baseball League. She was born in Decatur, Illinois on March 20, 1922 and passed away on November 25, 1999….” which is correct.

These examples demonstrate the efficacy of our approach is addressing hallucinations even when the complete sentence is made up. Quantitavitly, such examples fall in the “Not Related Category” in our paper and our approach addressed hallucination in 76.36% of such cases.

3) Actively Running our Method vs Running it at the end:

We briefly mentioned this point in Appendix B.3.3.

Our detection and mitigation technique can also be applied in a ''posthoc'' manner after complete response generation. However, it has several limitations which are addressed by our ''active'' approach.

The ``active'' approach prevents the propagation of hallucinations in the subsequently generated sentences, i.e., if hallucination is detected in the initially generated sentences then it would be mitigated and course correction would be done for the subsequently generated sentences. However, the ''post-hoc'' approach does not provide such an opportunity of course correction. In other words, in the ''active'' approach, the model sees the mitigated / corrected sentences while generating the subsequent sentences; thus, its output will be more correct, coherent, and fluent. In contrast, in the ''posthoc'' approach, the generated sentences are based on the initially generated previous sentences and thus the mitigated sentence will not be able to influence the generation of subsequent sentences; thus, the output would not be as coherent and fluent as the active approach.

Also, applying it in a post-hoc manner will fix the sentences individually thus, redundant information could be present in multiple sentences hampering the quality of the response.

For example,

For the topic “Twila Shively”, the model generated “Twila Shively is a renowned American artist and sculptor who has been creating art for over four decades. She is best known for her large-scale sculptures, which often feature abstract shapes and forms. …" which is completely hallucinated.

After applying our approach in a post-hoc manner gives “ Twila Shively was an American competitive baseball player who played from 1945 through 1950 in the All-American Girls Professional Baseball League. Twila Shively is known for playing baseball. …”

In contrast, active approach results in “Twila Shively was an American competitive baseball player who played from 1945 through 1950 in the All-American Girls Professional Baseball League. She was born in Decatur, Illinois on March 20, 1922 and passed away on November 25, 1999. Twila began playing softball at the age of eight and quickly moved up in the softball ranks in Chicago. …"

Thus, the active approach results in an output of much higher quality and doesn’t suffer from issues such are incoherence, consistency, repetition, etc.

We have included these examples in Appendix B.3.3.

Thank you for the reply!

I want to further understand the following questions:

Actively Running our Method vs Running it at the end

The examples are convincing. Do you have quantitive results to demonstrate the difference between the two approaches?

Comparison of Fact checking keyphrases of different confidences

If I understand correctly, Figure 5 does not tell the end-to-end effect of fact-checking based on confidence. In other words, though there's a correlation between hallucination and confidence, it is unclear if this component affects your system's final performance.

How does the model know the context while generating a sentence

For keyphrase detection / question generation / etc, is the context being used?

We thank the reviewer for the encouraging feedback. Please find our responses to your questions below:

Discussion on the effectiveness of the method beyond the first five generated sentences:

Thanks for this thoughtful question. Our study on the article generation task is conducted on the first five generated sentences. After applying our method, the correctness at sentence number level (averaged over all the inputs) is as follows (Sentence 1: 90.0%, Sentence 2: 82.67%, Sentence 3: 86.67%, Sentence 4: 82.67%, Sentence 5: 85.34%). These values are indeed close and do not considerably reduce as the sentence number increases. With this result, we show that our method of active detection and mitigation successfully mitigates the hallucination throughout the generation (not restricted to any specific sentence number). Furthermore, it shows that the ability to address hallucinations does not considerably diminish as the sentence number increases. Thus, even increasing the number of sentences is not expected to considerably impact the improvement that our method would bring. We will also include this additional result with the corresponding discussion in the revised version.

Will it be more useful to distribute these checkpoints across the generated text?

Our interpretation of this question is "why the unit of generation is a sentence and not multiple sentences."

We select a unit as a sentence over multiple sentences and (also over just a few words instead of a sentence) because of the following reasons.

Why not multiple sentences?

In autoregressive generation, the generation depends on the context including the model’s previously generated text. Thus, if we consider multiple sentences as a unit in our approach (let’s say 3 sentences) and if one of the initial sentences is hallucinated (and thus replaced with the corrected sentence), the subsequent sentences (i.e., the remaining sentences of the unit) may not stand relevant (as they were based on a sentence that has been replaced) and it may make the generation incoherent. Furthermore, the propagation of hallucination is another negative contributor as the next sentences may carry forward the hallucination of the previous incorrect sentences. Thus, the subsequent sentences in the unit would need to be regenerated. This implies that using multiple sentences as a unit may not return that benefit (that too at the extra cost of generating multiple sentences at once).

Similarly, we justify why we do not use a few words as a unit instead of a sentence.

We note that using a few words (i.e., a window of text) may not have sufficient information to test the correctness of the concepts in the generation. For instance, if the window is of the following words: “Rick Mahler won three gold medals and 2 silver medals at the”, it doesn’t have sufficient information to validate the correctness of the individual concepts. On the other hand, a sentence typically provides richer context to validate the correctness of the concepts of the sentence.

Because of the above two reasons, we use a sentence as the unit in our method.

We thank the reviewer for the comments. Please see our responses to your comments below.

1) Distinguishing our work from the paper “Language Models (Mostly) Know What They Know”: In our revised manuscript, we will include a citation to this paper and provide more discussion in the expanded related work (Appendix A). While we acknowledge that this paper is related to our research, we respectfully disagree with the reviewer’s comment that not citing Kadavath’s work is a major weakness. The focus and scope of Kadavath’s work are different from ours.

The paper “Language Models (Mostly) Know What They Know” has the following major findings:

• larger models are well-calibrated on multiple-choice and true/false questions
• Calibration improves with model size, and it also improves when we pass from zero-shot to few-shot evaluation;
• Explored self-evaluation, i.e., whether the output generated by the model is True or False. Section 4.1 (page in this paper) shows an example of Basic Self-Evaluation.

In contrast, our paper proposed an end-to-end methodology to detect and mitigate the hallucinations in the output of the LLMs. Specifically, we start by identifying the candidates of potential hallucination leveraging the model’s logit output values, check their correctness through a validation procedure (by retrieving the pertinent knowledge), mitigate the detected hallucinations via prompting, and then continue with the generation process. We show that this active intervention also helps in preventing the propagation of hallucination in the model’s autoregressively generated output.

We note that the focus and scope of our work are different from Kadavath’s work as the only common string is the use of logit values (which is just a component in our end-to-end methodology of detecting and mitigating the hallucinations in the output of the LLMs). We explain the differences in the use of logit values in more detail below:

We calculate the “concept-level” uncertainty of the model in “open-ended” long text generation using our techniques (presented in section 2.1.2 in our paper), namely, Average of Probabilities, Normalized Product of Probabilities, and Minimum of Probabilities unlike Kadavath’s work which focuses on "response-level" and evaluates calibration on multiple choice, true/false questions, and tasks like TriviaQA which are non-trivially different from open-ended article generation. Also, the “Self-Inquiry” technique in our work (which is although not a part of our main methodology as we use web search as the preferred technique for the retrieval step) is at the concept level and automatically creates validation inquiry questions based on the sentence and the concept (Section 2.1.3). In contrast, Kadavath’s paper used the following template: “Is the proposed answer:(A) True (B) False. The proposed answer is:” which is not suitable for open-ended long generation which typically has multiple facets in a single sentence.

Furthermore, we additionally use our proposed validation procedure (Section 2.1.3-2.1.5) to check the correctness of the uncertain concepts rather than just relying on the uncertainty values to detect hallucinations.

In summary, the objective, the focus, and the scope of our work are different from Kadavath’s work. The way we use the logit output values, the way we identify the concepts, the way we calculate the score corresponding to the identified concepts and the way we validate the correctness of the concept are different from Kadavath’s work.

2) Clarification on web search procedure for annotation: We note that the line “In some cases, multiple web searches were required to check the correctness of different facets of a sentence” in our paper in Section 3 is written in the context of the collection of human annotations for the hallucination task (and this is not a part of the methodology). As specified in Section 3, the human annotation for this task requires validating the correctness of the generation. Since the generation is for a variety of topics of different domains and would be beyond the common knowledge of a typical human, thus, we use web search to gather the relevant information to check the correctness of the generation. Multiple web searches were required in some cases because a generation can contain multiple facets of information all of which can not be validated in a single web search.

For example, sentences like “​Steven Threet i​s best known for his time at the University of Michigan, where he was a three-year starter and led the Wolverines to a Big Ten Championship in 2008.”, "Rick Mahler was a Major League Baseball pitcher who played for the Atlanta Braves, Cincinnati Reds, and St. Louis Cardinals from 1979 to 1994.” contains multiple facets that need to be validated separately because a single web search may not return all the information that is necessary to validate the correctness of all the facets of such sentences.

In the proposed active detection and mitigation method, we use the standard web search procedure (as detailed in 2.1.4), i.e, the validation query created in the previous step is used as the search query. We use Bing search API for retrieving the pertinent knowledge. Using a query to obtain the search results is a widely used standard procedure.

3) Further Explanation on the concept identification step: Firstly, we underline the importance of the concept identification step. It is crucial because validating the correctness of the entire sentence at once is infeasible as a sentence often contains multiple different facets all of which can not be validated at once. In contrast, individually validating correctness corresponding to the concepts provides opportunities for accurately detecting incorrectness.

We define “important” concepts as those that can be potential hallucinations in the generation and thus need to be validated. We provide the motivation behind this step in Section 2.1.1.

We note that the large language models are considerably good at understanding such instructions and the success of a variety of prompting-oriented methods (including chain of thought, Least to Most, etc) is a testament to that. Though this task is considerably straight-forward for LLMs (even for those of relatively smaller sizes), in cases where a specific LLM under study is not capable of following the instructions, we have provided alternative techniques for this step, namely Entity Extraction and Keyword Extraction which do not rely on the LLM and instead use an external tool for this step.

Please refer to Table 4 and Table 5 in the Appendix where we provide examples of the identified concepts in this step.

4) Performance of models in following different instructions presented in Table 3: Yes, we have indeed conducted evaluations of the efficacy of the instructions. Specifically, for the concept identification step, we studied randomly sampled 50 sentences. The instruction technique identified 155 concepts in total. It missed only 2 concepts (that too these missed concepts can only be loosely regarded as important in the context of the sentence). We have only provided a qualitative study for this step in the Appendix and will include this in the revision. Furthermore, the efficacy of the validation and mitigation instructions is presented in Table 1 and 2, respectively.

We note that the overall efficacy of these techniques (and how well they serve their purpose) is evaluated by the overall improvement in reducing the hallucinations.

We also note that the LLM can be prompted in a different way also to achieve the same objective; however, the purpose of this work is to show that the complex task of addressing hallucinations in an end-to-end manner can be decomposed into simpler steps that can be solved via instructing the model.

Dear reviewer pdV3, given the authors detailed response, please provide your updated feedback. Even if your score/decision does not change, it is critical to engage in the discussion. Thank you for your important contribution to this conference.