Analyzing and Mitigating Object Hallucination in Large Vision-Language Models

Thank you for your valuable feedback to help us improve our paper. We have revised our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.

Q1: The study and the proposed method are limited to caption hallucination problems, and seem not generalized to other settings like VQA.

A1: To strengthen the generalization of LURE, we conduct additional experiments on other popular benchmarks containing VQA (Visual Question Answering) questions, specifically on MME [Fu et al., 2023] and POPE [Li et al., 2023]. Since LURE is a post-hoc rectification method, during testing, we incorporated the captions rectified by LURE as context in the prompts for reference to run these VQA evaluations. The results in Tables R1 and R2 consistently demonstrate that incorporating LURE significantly reduces hallucination in VQA problems on MME and POPE (see detailed analysis in Appendix B.5). In the future, we aim to extend LURE to more complex VQA settings.

Table R1: POPE results of LLaVa on MSCOCO, A-OKVQA, and GQA.

Table R2: Hallucinatory performance of MME before and after the correction by LURE. Since we found that TN (True Negatives) and FN (False Negatives) are both zero in the MME dataset, the values of accuracy and recall are the same.

Q2: Both the training data of the hallucination revisor and the testing data are from COCO datasets. Whether the proposed method can be generalized to new datasets with object labels needs to be validated.

A2: We conduct additional analyses to assess the performance of LURE on two newly introduced datasets: ImageNet [Deng et al., 2009] and CC (Conceptual Captions) [Piyush et al., 2018]. Currently, the CHAIR metric can only be applied to the COCO dataset, which limits its usability beyond that dataset. To overcome this limitation, we manually annotate ImageNet and CC datasets to investigate object hallucination. Specifically, we randomly select 200 images from each dataset to be annotated. We evaluate the presence of hallucination in the generated captions through manual evaluation, using a scale where 0 indicated no hallucination and 1 indicated the presence of hallucination. The results presented in Table R3 demonstrate the performance improvements achieved by LURE across different datasets, thereby reinforcing our claims regarding LURE's effectiveness in reducing object hallucination in generated descriptions. These findings have been incorporated into our updated paper and can be found in Appendix B.5.

Table R3: Results (human evaluation) on additional datasets - ImageNet and CC. We assessed hallucination in the generated captions through manual evaluation, employing a scale where 0 indicates the absence of hallucination, and 1 indicates its presence. The average hallucination ratio (%) is reported in this table.

Q3 In the analysis of Co-occurrence, can the authors please explain what is the meaning and why $\hat f_2=\langle \phi_1(s_{<i},x),\hat\beta_1\rangle+\langle\phi_2(s_{<i},x),\hat\beta_2\rangle$?

A3: Thank you for your comment. We would like to note that $\langle\phi_1(s_{<i},x),\hat\beta_1\rangle+\langle\phi_2(s_{<i},x),\hat\beta_2\rangle=\langle (\phi_1(s_{<i},x), \phi_2(s_{<i},x)), (\hat\beta_1, \hat\beta_2)\rangle$. We use the enriched feature vector $(\phi_1(s_{<i},x), \phi_2(s_{<i},x))$ for the second object prediction to model the sequential prediction manner in auto-regressive inferences. The $\hat\beta_1$ in $\hat f_1$ and $\hat f_2$ are not necessarily equal. They are equal in our proof by coincidence as the solutions (to the linear discriminant analysis rule) of $\hat\beta_1$’s for $\hat f_1$ and $\hat f_2$ happen to be equal under our model assumptions.

Q4 The reviewer understands how the proposed methods related to the three observations on the object hallucinations. However, the reviewer doesn't see a clear connection between the theoretical analysis and the proposed methods. Can the authors explain this point?

A4: LURE can be conceptualized as an augmentation of the original LVLM, achieved by introducing a revisor as the last layer within the LVLM framework. Here, the input image is also used as an input for this revisor layer. Subsequently, we engage in a fine-tuning process specifically targeting the revisor component. During the fine-tuning phase, our objective is to address the issue of hallucinations that may arise within the LVLM. This is achieved by fine-tuning the revisor using carefully curated training data that is aware of the co-occurrence and uncertainty issue. In our theoretical analysis, we demonstrate that incorporating such data can significantly enhance prediction accuracy. For example, consider the “reconsidering uncertain objects” step of LURE. This step involves considering more data points with large uncertainty in the curated dataset. According to Theorem 2.2, this adjustment is proven to have a positive impact on prediction accuracy. We have elaborated on these connections between the theoretical analysis and the proposed method further in Section 2.4 of the revised paper.

Thanks a lot to the authors for the detailed response and the extra experiments. In the results in Table R1, the reviewer is not sure whether the following understanding is correct: The input to the LLaVA baseline is image embeddings, and the input to the LLaVa (LURE) includes the caption corrected by LURE and the image.

Dear Reviewer ZMPq,

Thank you very much for your response.

In the experiments presented in Table R1, during the inference process, for LLaVa (Original), the input to LLaVa 13B consists of the original question from POPE and the corresponding image. For LLaVa (LURE), the input during inference comprises the original question from POPE, the image, and the caption that has been modified by LURE.

Q3: Why were co-occurrence, uncertainty, and object positions identified as the three key factors for object hallucinations? Were other factors investigated?

A3: The selection of these three factors is based on their broad applicability, as they can apply to various types of images. Apart from these factors, we did explore other factors that may contribute to hallucinations. For instance, our investigations revealed that LVLMs are susceptible to hallucinations when confronted with (1) sketch images or sketch objects and (2) composite images. In the case of composite images, combining multiple semantically similar images into a grid can trigger hallucinations in LVLMs. However, it's important to note that these findings are more specific and are primarily associated with certain types of images. Consequently, we have not included them to design LURE.

Thank you for your constructive comments and suggestions. We have revised our paper according to your comments. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.

Q1: The proposed method helps improve performance on object hallucinations. However, there is a concern that it may harm performance on other metrics like creativity and completeness of captions. It seems to replace detailed words with coarse words as shown in Fig 8. 2.It is unclear if the removed objects are truly hallucinated or if it wrongly removes some non-hallucinated objects. A new metric to quantify this would be helpful.

A1: To delve deeper into the issue of creativity and completeness in captions, we conduct experiments to analyze the variations in the diversity of descriptions generated by different models before and after applying LURE. Our primary focus was on assessing changes in description length and the reduction in the proportion of correctly identified objects, as well as the decrease in hallucinated objects when LURE was introduced. We have presented the findings in Table R1.

The results of our study reveal that the incorporation of LURE leads to a significant reduction in hallucinated objects, averaging around 56%, while only slightly affecting the number of correctly identified objects, with an average decrease of approximately 1.6%. This noteworthy outcome can be attributed to the fact that LURE encourages the model to reevaluate these potentially hallucinatory objects, either removing or replacing them. This approach substantially enhances performance and reduces hallucination. Moreover, this advantage is evident in the average length of descriptions before and after applying LURE. It is clear that LURE has only a minor impact on the length of descriptions, indicating its effectiveness in preserving the utility and diversity of the responses.

In summary, the use of LURE achieves a balance between response accuracy and utility in LVLMs. We have included these newly obtained results in Appendix C.2 of the updated paper.

Table R1: Analysis of correctness and usefulness before and after applying LURE.

Q2: Do the authors think image captioning metrics are good metrics for LVLMs? The BLEU scores seem low compared to image captioning models. Some important metrics like METEOR, ROUGE, CIDER, and SPICE are missing in Table 10.

A2: We conduct additional experiments to evaluate performance using METEOR, CIDER, and SPICE, while ROUGE has already been evaluated and is presented in Table 10 of Appendix B.3. The results are detailed in Table R2, demonstrating improvements across various metrics when LURE is applied. We’ve included these results in Table 11 of Appendix B.3 in the updated paper.

However, it is worth noting that while LURE can enhance image captioning metrics in most scenarios, these metrics may not provide a sufficiently accurate evaluation of hallucinations in LVLMs. This limitation arises from the metrics' emphasis on measuring the similarity between generated captions and groundtruth. In the case of fine-grained and detailed captions, these metrics may not accurately reflect the level of object accuracy in the descriptions. This issue occurs because increased object accuracy in fine-grained content may not substantially impact caption similarity.

Table R2: Performance with additional metrics.

Thanks to the rebuttal of the reviewers. I like the paper as it is inspirational on reducing hallucinations in VLLM.

I will keep my score.

Q4: Lack of comparing LURE's performance on the fine-grained caption and concise caption. Intuitively, the problem of hallucination would be more common in fine-grained captions.

A4: In our initial experiments, we employed detailed and fine-grained captions, obtained by prompting "Generate a short caption of this image". To further explore the effectiveness of LURE with concise captions, we conduct additional experiments, the results of which are presented in Table R3. The concise descriptions are generated using the prompt 'Generate a short caption of the image.' Our findings indicate that LURE remains effective in reducing object hallucinations even with shorter captions, thus reinforcing its capability in mitigating such issues. We’ve included these results in Appendix C.3 of the updated paper.

Table R3: Performance of LURE on concise captions generated by four best-performing LVLMs.

Q5: The effect of object position on object hallucination is not clear. I am still confused why hallucination occurs in the latter part of the descriptions. Is it possible to fundamentally reduce LVLM hallucination from this perspective?

A5: One potential reason for object hallucination occurring later in the generated descriptions is highly related to the 'snowballing effect' [Zhang et al., 2023]. This phenomenon occurs when object hallucinations in the early part of generated descriptions lead to a cascade of accumulated hallucinations in the latter portions.

In LURE, we actively encourage the hallucination revisor to ``reconsider” objects in the latter portion of the descriptions, with the aim of mitigating hallucinations. The results presented in Table 15 (updated paper) provide evidence that LURE is effective at reducing hallucinations in the latter part of the generated descriptions.

In future research, to further resolve this issue, an interesting direction to explore would be the combination of multi-modal retrieval techniques with sentence-by-sentence hallucination rectification. This approach could offer a promising direction for addressing the 'snowball' issue. Such research would be considered as part of our future work.

Reference

[Zhang et al., 2023] Zhang M, Press O, Merrill W, et al. How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534, 2023.

[Fu et al., 2023] Fu, Chaoyou, et al. "MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models." arXiv preprint arXiv:2306.13394 (2023).

[Li et al., 2023] Li, Yifan, et al. "Evaluating object hallucination in large vision-language models." arXiv preprint arXiv:2305.10355 (2023).

Q3: LURE is designed as a post-hoc solution aimed at addressing object hallucination; however, it doesn't directly confront the underlying causes of the issue. A more direct challenge would be formulating strategies for guiding the LVLM to produce answers with reduced hallucination tendencies.

A3: LURE, while being a post-hoc solution, proves to be both promising and effective in mitigating object hallucinations in LVLMs. Its key advantage as a post-hoc rectification method lies in its compatibility—it can seamlessly integrate with multiple LVLM architectures as a plug-and-play module, significantly enhancing response quality while minimizing hallucination. Importantly, it achieves these improvements without requiring additional fine-tuning or updates of pre-trained LVLMs, and it doesn't necessitate access to the private data corpus used in the pre-training of commercial large models.

Furthermore, to strengthen the effectiveness of LURE, we conduct additional experiments on other widely recognized benchmarks, specifically MME [Fu et al., 2023] and POPE [Li et al., 2023] (detailed experimental setups can be found in Appendix B.5). The results, presented in Tables R1 and R2, consistently demonstrate that the incorporation of LURE significantly reduces hallucination in both the POPE and MME datasets, further reinforcing its effectiveness. Detailed results and in-depth analysis are available in Appendix B.5 of the updated paper.

Nevertheless, it's equally important to devise strategies that guide LVLMs to produce answers with reduced hallucination tendencies. This direction holds great promise, and we intend to investigate this problem in the future.

Table R1: POPE results of LLaVa on MSCOCO, A-OKVQA, and GQA.

Table R2: Hallucinatory performance of MME before and after the correction by LURE. Since we found that TN (True Negatives) and FN (False Negatives) are both zero in the MME dataset, the values of accuracy and recall are the same.

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we’ve revised our paper according to your suggestions. We would appreciate it if you could let us know whether your concerns are addressed by our response.

Q1: The dataset's composition is not entirely clear, especially concerning the proportion of hallucinated content within it. Secondly, during the training of the hallucination revisor, they used 5,000 image-text pairs from the LLaVA-150k dataset randomly. However, the study does not provide adequate experimental backing to validate the adequacy of this sample size for their objectives. It would be beneficial if the authors could offer a more comprehensive description of their dataset, and ideally make it open-sourced. Such an act could serve as an added contribution to their work. Furthermore, the LURE methodology, as presented, comes across as somewhat straightforward, lacking a distinctive innovative edge.

A1: Our data construction process involves the creation of examples composed of three key elements: an image, a groundtruth description, and a hallucinatory description. The groundtruth descriptions are sourced from the LLaVA-150k dataset, while the hallucinatory descriptions are generated using an analysis based on three factors derived from real descriptions. This construction process necessitates the use of GPT's API and manual screening. Due to budget considerations, we have currently assembled a dataset comprising 5,000 training data samples. Our experimental results indicate that this dataset size is sufficient for training an effective hallucination revisor.

Furthermore, the groundtruth descriptions in our dataset underwent meticulous manual screening, resulting in very few instances of hallucination. To assess the quality of the groundtruth descriptions, we also calculated the CHAIR metric for both the groundtruth and hallucinatory descriptions in the dataset. It's worth noting that CHAIR, designed to detect 80 specific objects, may not provide entirely accurate results in this context. Nevertheless, the results on both metrics are as follows:

• Groundtruth descriptions: CHAIRs: 3.4, CHAIRi: 0.4
• Hallucinatory descriptions: CHAIRs: 41.7, CHAIRi: 7.0

We have also made the dataset publicly available. You can access the dataset via the following anonymous link: https://anonymous.4open.science/r/hallucination5k-2A20/hallucination5k_train.jsonl

Q2: Furthermore, the LURE methodology, as presented, comes across as somewhat straightforward, lacking a distinctive innovative edge.

A2: We agree that LURE is a seemingly simple method, but its effectiveness in rectifying object hallucinations in LVLMs is demonstrated by our empirical results. The design of LURE, however, is not as straightforward. It stems from our comprehensive and rigorous statistical analysis of the reasons behind LVLM hallucinations.

Q2: Lack of results of other popular benchmarks. This paper only reports the performance of the COCO 2014 test dataset, which is small and may be biased. There is no result about the performance on other popular benchmarks for LVLMs, such as MMBench, MME, POPE, SEED, MM-Vet, etc. Will the performances be better or worse on these benchmarks?

A2: We conduct additional experiments for LURE on other popular benchmarks, specifically MME and POPE, as they are the most suitable datasets for evaluating hallucinations (refer to the comprehensive experimental setups in Appendix B.5). The results are presented in Tables R1 and R2, where we observe a noteworthy reduction in hallucination with the introduction of LURE in both the POPE and MME datasets. These results not only underscore the effectiveness of LURE but also provide additional support for the conclusions drawn in our original paper. We have included these new findings and an in-depth analysis in Appendix B.5 of the updated paper.

Table R1: POPE results of LLaVa on MSCOCO, A-OKVQA, and GQA.

Table R2: Hallucinatory performance of MME before and after the correction by LURE. Since we found that TN (True Negatives) and FN (False Negatives) are both zero in the MME dataset, the values of accuracy and recall are the same.

Thank you for your valuable feedback to help us improve our paper. We have revised our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.

Q1: Comprehensive analysis of object position and hallucination, including subquestion “the definition of positioning score is not intuitive. Have the authors analyzed the position score distribution under different description lengths?” and “The results of Figure 1(c). Is this quantity of images (just 200) sufficient to consolidate the distribution statistics? And even if a sufficient number of samples are provided in a specific domain, can this conclusion be generalized to the distribution of other datasets or benchmarks?”

A1: To gain a deeper understanding of the impact of object position on hallucinations, we extend our analysis beyond the existing evaluation presented in Figure 1(c). This extended analysis encompasses the following evaluations:

• Evaluation with More Examples: In the first phase of our evaluation, we re-assess the distribution of hallucinatory objects concerning their positions using a larger dataset comprising 5,000 examples from the COCO dataset. The results are detailed in Figure 5(a) within Appendix C.1.1 of our updated paper.

• Evaluation on short descriptions: Similarly, in the second phase, we evaluate the distribution of hallucinatory objects concerning their positions within short descriptions generated by models such as OFA, BLIP2, etc., using the same 5,000 data points as in the first evaluation. These findings are illustrated in Figure 5(b) of Appendix C.1.1 in our updated paper.

• Evaluation on other datasets: In the third phase, we explore the connection between the distribution of hallucinatory objects and their positions in datasets such as CC dataset. For this evaluation, descriptions are manually annotated to identify hallucinated objects, and the results are reported in Figure 5(c) of Appendix C.1.1.

Across all evaluations, our findings consistently indicate that high-density areas of hallucinatory objects predominantly appear towards the end of the sequence, regardless of the length of the descriptions. This further reinforces our original conclusions. Furthermore, it is worth noting that generating shorter descriptions does not yield lower position hallucination. Therefore, simply generating multiple short descriptions and combining them may not necessarily lead to higher-quality descriptions.