Response to Reviewer 6ngE

We thank all the reviewers for the insightful comments and the recognition of our work

"Novel Idea" (Reviewers #1, #3, #4)
"Promising Results"(Reviewers #2, $3, #4)
"Interesting"(Reviewers #1, #4)
"Easy to understand"(Reviewers #1, #4). Now, we carefully answer your question as follows.

W1: Generalization issue.

W1-A1: Our proposed Hallucination-Induced Optimization (HIO) strategy significantly enhances the generalizability of hallucination mitigation. Specifically, HIO induces multiple potential hallucinations (as claimed in Sec.4.2) for the current token using a beam search mechanism based on their predictive logits. Given that these potential hallucinations can be any word within a 32,000-word dictionary in the LLM, HIO theoretically has the capacity to generalize to unseen objects, effectively mitigating hallucinations. Futhermoe, to validate the generalizability of our method, we collected samples containing out-of-distribution object categories and evaluated the model's performance on these unseen classes. The test sets, Unseen-N and Unseen-P, consist of 426 images crawled from the Internet and 69 images sourced from MSCOCO, A-OKVQA, and GQA, with no overlap with the model's training set. These test sets include a total of 10 unseen categories. The evaluation results, shown in Table 1, demonstrate that our method outperforms the baseline model (i.e., Regular), highlighting its superior generalizability to open-vocabulary objects.

Table 1: Generalization experiments of our method on two unseen datasets

W2: Authors are suggested to include some results comparing these methods. Whether an amateur model can help an RLHF model seems an interesting question and may improve this paper.

[A] Aligning Large Multimodal Models with Factually Augmented RLHF.

W2-A2: Following your suggestion, we have included experimental comparisons with method [A] under the POPE benchmark. As shown in Table 2, our method consistently outperforms LLaVA-RLHF7B across three settings in POPE, demonstrating the effectiveness of our approach in mitigating hallucinations. Furthermore, we believe our method can help RLHF models generate multiple hallucinated answers with minimal effort, thereby augmenting method [A] with high-quality instruction-tuning data for fine-grained RLHF alignment. We plan to explore this in future work.

Table 2:Comparative experiments between our method and the RLHF method on POPE

Q1:Authors are suggested to include the other two as well for more comprehensive evaluations.

Q1-A3: In response to your suggestion, we conducted additional experiments on the MSCOCO, GQA, and A-OKVQA datasets for a more comprehensive evaluation. The results, presented in Table 3, show that our method consistently outperforms VCD across all three setups, demonstrating its effectiveness in mitigating hallucinations in LVLMs.

Table 3: Results of our method on MSCOCO, A-OKVQA, and GQA

Q2: Typos. In line 135.

Q2-A4: Thank you for highlighting the issue. We have addressed it in our revised manuscript.

W1: The description of experimental details is weak.

W1-A1: Based on your suggestions, we have revised the descriptions of the experiments in Tab.1, Tab.2, and Tab.5 as follows: Table 1 presents the experimental results on the POPE dataset across random, popular, and adversarial settings. Our method consistently outperforms state-of-the-art decoding methods across three settings, demonstrating its effectiveness in mitigating object hallucinations in diverse scenarios. Tables 2 and 5: We assess our model's performance in open-ended caption generation by evaluating 500 images from COCO val2017 and val2014 in Tab.2 and Tab.5. We observed that our method significantly reduces object hallucinations in generated captions with lower CHAIRS and CHAIRI scores, while enhancing caption detail, as indicated by higher Recall scores. It demonstrates that our method can achieve an effective balance between accuracy and detail in open-ended caption generation by widening the gap between hallucinated and correct tokens.

W2: Lack results of the three models.

W2-A2: We have included the results for the LLava, MiniGPT-4, and InstructBLIP models on the MSCOCO, A-OKVQA, and GQA datasets in Tab.1.

Table 1:Results of the three models on MSCOCO, A-OKVQA, and GQA

W3: Lack detailed implementation.

W3-A3: We provide an detailed description of our implementation as follows: Implementation Description: We evaluate our HIO by incorporating it into three LVLMs: LLaVA 1.5, InstructBLIP, and MiniGPT-4. For decoding, we use Llama-7B and Vicuna-7B as the linguistic decoder for LLaVA and InstructBLIP/MiniGPT-4. To ensure a fair and rigorous comparison, we adhere to the configurations and guidelines from the original works and codebases of the compared models. The training is conducted on 4x RTX 3090 GPUs for LLaVA 1.5, 8x V100 GPUs for MiniGPT-4, and 4x A6000 GPUs for InstructBLIP. Each training session lasts approximately 2-4 hours. Hyperparameters including alpha and beta are set to 1.0 and 0.1, in accordance with the VCD model's specifications.

W4: Details betray a rather hasty preparation of the article.

W4-A4: We have revised all your concern based on your suggestions.

W5: Error in Figure 2.

W5-A5: We have fixed this issue in our revised manscript.

W6: Inappropriate caption for Table 4.

W6-A6: Based on your suggestion, we have revised the caption for Table 4. Table.4: Ablation study with different components of our model on CHAIR-COCO.

W7: Inappropriate bolding for Tables 1 and 3.

W7-A7: We have reviewed the formatting and make sure that the bolding is applied consistently and appropriately.

W8: Conclusion section is missing.

W8-A8: Due to the length of the initial submission, the conclusion was placed in the supplementary materials. In the revised manuscript, we have included the conclusion in the main text.

Q1: Utilization of hallucinatory information is unclear.

Q1-A9: The following discussion has been incorporated into our revised manuscript: We utilize hallucinatory information—multiple potential hallucinated tokens—to sharpen the contrast between hallucinated and correct tokens, thereby enhancing the effectiveness of contrastive decoding. The rationale is that while the single-target DPO mechanism addresses a specific hallucinated token, it may inadvertently trigger other potential hallucinations. For example, as shown in the lower half of Fig. 2, single-target DPO can correct the hallucination of "Dogs" but may introduce "Tracks" as a new hallucination with the second-highest confidence. To strengthen the contrastive decoding process, we generate multiple potential hallucinated tokens and ensure that the model considers them simultaneously. We then apply the HIO mechanism for adversarial training between the correct token and the generated hallucinations, creating an "evil" model that favors these hallucinated tokens. This approach significantly improves the effectiveness of contrastive decoding, as the contrastive logits of the correct token surpass those of all hallucinated tokens, a fact theoretically validated in Sec. 5.

Q2: Efficiency and effectiveness.

Q2-A10: Unfortunately, achieving a trade-off between efficiency and effectiveness remains challenging. As noted in [1], using pretrained models with lower capacities (e.g., BLIP2-7B) can significantly reduce training costs but results in decreased effectiveness in mitigating hallucinations. [1] Contrastive Decoding: Open-ended Text Generation as Optimization

Q3: Revise the writing details and experimental descriptions.

Q3-A11: We have revised writing details and enriched experimental descriptions in Tab.1, Tab.2 and Tab.5 of Sec.6.

W1: Novelty: motivation is similar to [1]. [1] Alleviating Hallucinations of Large Language Models through Induced Hallucinations

W1-A1: We indeed adopt the same concept of induced hallucination from ICD[1], but we also identified two key issues of it, i.e., less effective and poor generalizability. To address these, we propose our Hallucination-Induced Optimization including three conponents, i.e.,CBTM, AMTH and ACI.

1.Less Effectiveness：

(1)Sentence-Level vs. Token-Level Hallucination Induction: The Supervised Fine-Tuning (SFT) strategy operates at the sentence level, which limits its ability to precisely induce specific hallucinated tokens at token-level, as discussed in Lines 203-207 of Section 4.3. This limitation impairs ICD's ability to effectively amplify the distinction between hallucinated tokens and correct tokens, thus reducing the effectiveness of Contrastive Decoding for hallucination mitigation. To address this issue, we propose a DPO-based hallucination mitigation method, the CBTM, which precisely induces token-level hallucinations for the desired tokens.

(2) Predefined vs. Self-Generated Hallucinations: ICD [1] relies on hallucination samples generated by ChatGPT, rather than the model's own hallucinations as proposed in our Amplification of Multiple Targeted Hallucinations (AMTH) method (Section 4.2). Since these externally-generated hallucinations may not accurately reflect the model's actual hallucination states, they can lead to less effective optimization or even result in overfitting. To address this issue, AMTH induces the model's own hallucinations, providing a more precise representation of the model's internal states. As a result, our method more effectively targets and reduces the model's hallucinations. To compare the effectiveness of our Hallucination-Induced Optimization (HIO) with ICD [1], we conduct experiments to assess their performance in mitigating open-ended and discriminative hallucinations, as detailed in Tables 1 (CHAIR) and 2 (POPE). To implement ICD based on LLaVA, we first fine-tune an Evil-LLaVA-7B model using hallucinated answers generated by ChatGPT. We then apply this fine-tuned model to perform contrastive decoding against the original LLaVA-7B model. As shown in Tables 1 and 2, our method achieves significant improvements over ICD, demonstrating its superior capability in reducing hallucinations. This enhancement is attributed to HIO's token-level induction, which effectively utilizes hallucinations generated by the model itself.

Table 1: Comparison with ICD on CHAIR

Table 2: Comparison with ICD on POPE

2.Poor Generability： The model tends to produce numerours different hallucinations for each visual semantic scene. However, ICD only considers only one of them annotated by ChatGPT, ignoring other possible hallucinations. As illustrated in the lower part of Figure 2, where 'People,' 'TV,' and 'Clock' are all hallucinations. For ICD, it can only induce one hallucinated token, such as 'Dog', while ignoring the other potentional hallucinations, such as 'Clock'. Finally, ICD overfis on the annotated hallucinations given by ChatGPT, failing to address most potential hallucinations, and further reducing the model's ability to generalize to unseen hallucinations. In contrast, our proposed Amplification of Multiple Targeted Hallucinations (AMTH) generates multiple potential hallucinations using a beam search mechanism based on their predictive logits, as described in Section 4.2. Since these potential hallucinations can be any word from the 32,000-word dictionary of the LLM, AMTH demonstrates improved generalizability to unseen objects, thereby more effectively mitigating hallucinations. To demonstrate the generalization capabilities of our method, we collected two OOD test set as Unseen-N and Unseen-P that the model had not encountered during training for evaluation. Unseen-N contains 426 samples gathered from the web, while Unseen-P includes 69 samples collected from MSCOCO, A-OKVQA, and GQA. This resulted in a total amounts of 495 samples with 10 unseen object classes. From Tab.3, we observe that our method significantly outperforms ICD[1] across Accuracy, Precision and F1-Score on unseen catagories by a large margin, thereby proving more powerful generalization capability of our proposed HIO method in comparision to ICD.

Table 3: Comparison with ICD on unseen datasets.

W2: This paper is not complete.

W2-A2: Due to space constraints, the conclusion section was initially included in the supplementary materials. We have incorporated it into the main manuscript in our revised submission. Moreover, we have enriched the ablation study to analyze the generalization capability of our proposed components to unseen categories, as detailed in Tab.4. Finally, we have integrated the ablation study into the experimental results section, rather than presenting it separately.

Table 4: Ablation study on the generalization of each component on unseen datasets

Response to Reviewer DdM1

We thank all the reviewers for the insightful comments and the recognition of our work
"Novel Idea" (Reviewers #1, #3, #4)
"Promising Results"(Reviewers #2, $3, #4)
"Interesting"(Reviewers #1, #4)
"Easy to understand"(Reviewers #1, #4). Now, we carefully answer your question as follows.

W1: The writing is sometimes confusing and makes it difficult to understand details. For example, Sec 4.1 and 4.2 use Eqn 17 in Sec 5 as the motivation of how they design the current method and refer to it frequently. However, Eqn 17 is not explained and introduced previously, making it very difficult for the reader to understand why you design this method like this. It would be much better to explain the high level idea of eqn 17 first.

W1-A1:Thanks for your suggestion, we have added the descriptions of Eqn.17 in line 141 as below: ... to induce more potential hallucinations for effective contrast decoding, we propose to amplify multiple hallucination tokens, building on the theoretical foundation presented in Eqn. 17 of Section 4.2. This theory demonstrates that effective contrastive decoding requires a consistent difference between the logits of potential hallucinated tokens and the correct token. And Section 4.3 introduces additional constraints ... .

W2: Besides, the figure 1 itself is also confusing. The picture provided is vague and even a human finds it difficult to identify the people and output the true answer.

W2-A2: As per your suggestion, we have clarified the image and enhance its quality to ensure that the details are more discernible.

W3: When reading the Sec 5, it is also very confusing. How do you define the ideal logits for contras decoding? Based on my understanding, it should not equal to the hallucinatory token's logits. However, at the later part line 243, you are referring to it as the hallucinatory tokens. Besides, how do you interpret the second line of Eqn 17? I interpret J as the average logit difference between the hallucinatory token and the true token. However, I find it difficult to interpret the left part of the second line of Eq 17. What is the logic behind line 242? The whole explanation does not make sense to me.

W3-A3:
(a) The ideal logits for effective Contrastive Decoding, as we defined, require that the logits of all hallucinated tokens should be lower than those of the correct token. The rationale behind this is that traditional DPO-based Contrastive Decoding methods, while reducing hallucinations on the preferred hallucinated token, often generate new hallucinated tokens, thereby diminishing the effectiveness of contrastive decoding. Therefore, we assert that the model should simultaneously reduce the logits of multiple potential hallucinated tokens so that they all remain lower than the logits of the correct token, thereby significantly enhancing the effectiveness of Contrastive Decoding in mitigating hallucinations.

(b) Eqn.17 illustrates that hallucinations can be effectively eliminated through contrastive decoding if the difference between the logits of the hallucinatory token and the correct token in the "evil" LVLM's output (Left part of Eqn.17) exceeds that in the original LVLM output (J in Eqn.17). For example, as depicted in the lower part of Figure 2, where "Dogs" is a hallucination and "Benches" is the correct label, the hallucination of "Dogs" is removed when the difference between the logits for "Dogs" and "Benches" in the "evil" LVLM output surpasses the difference in the original LVLM output. When this condition is met for all potential hallucinations, all hallucinations are effectively eliminated. A more detailed explanation will be provided in line 242 of the revised manuscript.

W4: For Eqn 14, is it too strong to require the minimal logit of correct tokens to be larger than the maximum logit of hallucinatory tokens? For greedy docoding, if one of the correct tokens after contrast is larger than all the hallucinatory tokens, then it will be output?

W4-A4:
(a) Eqn.14 represents a theoretical upper bound, which guides us in enhancing the effectiveness of Contrast Decoding method for hallucination elimination by ensuring that the logits of all hallucinated words are lower than those of the correct words. We further refine this upper bound into a more practical optimization objective, as detailed in Eq.17. Specifically, the induced Evil model must simultaneously maintain lower logits of multiple hallucinated words than that the correct token. To achieve this, we propose the Hallucination-Induced Optimization (HIO) method, which significantly improves the effectiveness of hallucination elimination in contrast decoding.
(b) If the logit of the correct token after contrast is greater than all hallucinatory tokens, it would be selected during greedy decoding.

Q1: Line 127 should be "probability" instead of "probably"

Q1-A5: Thank you for highlighting this issue, we have corrected it in our revised manscript.