Visual Evidence Prompting Mitigates Hallucinations in Multimodal Large Language Models

We sincerely thank the reviewer for taking the time to review. We appreciate that you find our approach a new idea, well-organized and easy to follow. According to your valuable comments, we provide detailed feedback.

Q1: Only providing object labels:

The authors are recommended to provide the performance comparisons on object and relation hallucinations where only object labels in the input image are provided. If the performance is also good, please make explanations on the necessity of the proposed method.

Ans for Q1:

Thanks for the inspiring and interesting comments!

We conduct validation experiments using the detr-resnet-101 model to provide object labels as evidence for relation hallucination.

The results show that providing object labels as evidence also has some improvement although not as effective as relation label. We suppose it is because object labels themselves contain crucial object information from the image, which leads to mitigating relation hallucination. This result not only validates the necessity of relation labels but also further verifies that our approach is orthogonal to the specific task. We have added the above discussions in the revision (Appendix A.4).

Q2: Open-vocabulary objects and few-shot relations:

The authors are recommended to provide the hallucinations mitigations on open vocabulary objects and few-shot relations.

Ans for Q2:

Thanks for your insightful question. We conduct experiments in response to your valuable advice. The following discussions are added in the revision (Appendix A.5).

a) open vocabulary objects: We construct a new out-of-domain object hallucination dataset using the test sets from Object365 [A]. This dataset is divided into two parts. One part includes 80 objects that appear in COCO, while the other portion consists of objects that do not appear in COCO. The performance of these two parts are shown in the table below. It can be observed that there is a consistent improvement in performance for both in-domain and out-of-domain object categories.

b) few-shot relation: We chose the bottom-10 tail relations as defined in [C] of VG to construct a medium-sized relation hallucination dataset with 1006 samples. We used OpenPSG as the SGG model and conducted experiments on Qwen-VL-Chat and mPLUG-Owl. The experiment results are shown in the table below, and it can be seen that our framework still achieves significant improvements in few-shot relations.

[A] Objects365: A Large-scale, High-quality Dataset for Object Detection
[B] Simple Open-Vocabulary Object Detection with Vision Transformers
[C] Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation

Q3: Figure 4 explanation:

make explanations on the decline of 'at' or slight improvement on 'under', displayed in Figure 4.

Ans for Q3:

Thanks for your careful review. We conducted statistical analysis on the samples of these two relations. It is observed that "at" is naturally ambiguous with "on", "next to", and "in", and "under" is ambiguous with "below" and "around". We will carefully review the collected data to minize the confusions.

Q4: Analysis of Table 9 in Appendix:

make necessary analysis for situations where the performance of RelTR on SGG is signiìcantly higher than motifs, while the performance of hallucination mitigations on LVLMs is equal or even opposite, displayed as in Table 9.

Ans for Q4:

Thank you for your meticulous review. In the Table 9, different scene graph generation models (RelTR, MOTIFS and OpenPSG) have comparable improvements on mPLUG-Owl and Qwen-VL-Chat. For example, RelTR achieves 5.92% and MOTIFS achieves 5.8% improvement on mPLUG-Owl. RelTR achieves 11.35% and MOTIFS achieves 12.55% improvement on Qwen-VL-Chat. The gains brought by different scene graph generation models to LVLM are within a stable range (saturated).

Dear reviewer bJ8t,

We sincerely apologize for inconveniencing you, but we have no other choice at the end of the discussion period. We sincerely hope you can understand.

Thank you for taking the time to review our work. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work. Your support or feedback are very important to us. We greatly appreciate your constructive comments and efforts.

Best regards

Authors of #3272

Dear reviewer bJ8t,

We sincerely apologize for inconveniencing you again, but we really have no other choice as the discussion period is drawing to a close (only 1 days left). We sincerely hope you can understand.

Thank you for taking the time to review our work. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work. Your support or feedback are very important to us. We greatly appreciate your constructive comments and efforts.

Sorry again for inconveniencing you.

Best regards

Authors of #3272

Dear reviewer bJ8t,

Deep apologies for the repeated interruption. Sorry so much!

We are truly honored that you have reviewed our paper and provided valuable constructive suggestions. Sincerely thanks for these constructive suggestions which greatly aid in our paper’s refinement, we have diligently addressed every concern and question you raised during the initial review. We genuinely hope our responses have resolved your concerns and provided satisfactory explanations. We sincerely appreciate your dedication and valuable time.

To further improve our work, we sincerely hope for a valuable opportunity to discuss with you. It would be greatly appreciated if we could have a chance to hear your feedback. Your feedback is highly valuable to our paper and greatly contributes to the entire community.

Very sorry for inconveniencing you again. Hope you have a good day!

Best regards and many thanks,

Authors of #3272

We sincerely thank the reviewer for taking the time to review. We appreciate that you find our approach simple, experimental improvement, contributions of dataset and benchmark and the in-depth analysis. According to your valuable comments, we provide detailed feedback.

Q1: Overlap between objects:

Have the authors explored questions regarding how overlapped between objects in questions and objects in evaluation datasets' questions?
1.1 What is the current stats on how overlapping between objects in questions and objects in evaluation datasets' questions?
1.2 What if visual evidence prompt contains objects that are not in the question? what if the prompt contains objects that are not in the questions exclusively?
1.3 What if the object names are switched to the synonyms to the objects appear in question?

Ans for Q1:

Thanks for the inspiring and interesting comments!

a) We calculate the current stats of the overlap using detr-resnet-101. In the 3,000 visual evidence prompts, there are 298 prompts that contains object that are not in the question (Type A), and 1,202 prompts that contain objects that are not in the questions exclusively (Type B).

b) Following Figure 3 in the draft, we calculate the stats of samples which were initally answered corrently/wrongly and answer correctly/wrongly after provided with Type A/B prompts. In the 298 Type A prompts, 110 of which (36.9%) allievates the hallucination of LVLM with detr-resnet-101 on Qwen-VL-Chat. In the 1,202 Type B prompts, 563 of which (46.8%) allievates the hallucination of LVLM.

c) With the help of ChatGPT, we manually change the object appear in question to its synonyms respectively. The evaluation of object hallucination slightly decreases from 87.70% to 86.53% on Qwen-VL-Chat and from 78.38% to 71.54% on mPLUG-Owl, but there is still a non-trival improvement over the baseline especially on mPLUG-Owl, the results are shown in the table below.

In summary, with the LVLM's intrinsic robust ability to understand the output visual evidence of small models, visual evidence prompting is also robust in terms of the specific words used for objects in the given problem. We have add the above discussions in the revision (Appendix A.3).

Q2: Scale of Image:

does it matter to change the scale of the image? for example, "cup <0, 284, 133, 424>" to "<0 /width, 284 / height, 133 / width, 424 / height>". or to "<0 /width * 2, 284 / height * 2, 133 / width * 2, 424 / height *2>". I wonder to what extend LVLMs will diëerentiate object relations

Ans for Q2:

Thanks for your careful review. In Qwen-VL-Chat, for any given bounding box, a normalization process is applied (within the range [0, 1000)). For example, x_input = x_pixel / width * 1000, y_input = y_pixel / height * 1000. So the scale of the image does not differentiate any object relations. We clarify this setting in Sec. 4.1.3 in the revision.

Dear reviewer YNjN,

We sincerely apologize for inconveniencing you, but we have no other choice at the end of the discussion period. We sincerely hope you can understand.

Thank you for taking the time to review our work. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work. Your support or feedback are very important to us. We greatly appreciate your constructive comments and efforts.

Best regards

Authors of #3272

Dear reviewer YNjN,

We sincerely apologize for inconveniencing you again, but we really have no other choice as the discussion period is drawing to a close (only 1 days left). We sincerely hope you can understand.

Thank you for taking the time to review our work. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work. Your support or feedback are very important to us. We greatly appreciate your constructive comments and efforts.

Sorry again for inconveniencing you.

Best regards

Authors of #3272

We sincerely thank the reviewer for taking the time to review. We appreciate that you find our approach motivation good, experimental effectiveness and well-written. According to your valuable comments, we provide detailed feedback.

Q1: Out-of-domain data:

The authors utilize small detection and scene graph models as evidence, potentially introducing domain-speciìc knowledge that skews the results.
a) It's crucial to clarify the differences in training data between these small models and the evaluation data.
b) Additionally, the paper should address whether the proposed method remains effective when applied to out-of-domain models.

Ans for Q1:

a) Thanks for pointing out this potentially confusing point. There is no overlap between the training data and the evaluation data. We are using the open-source small models trained on the COCO or Visual Genome training set. The test set we have constructed is derived from the COCO validation set or Visual Genome test set. We have clarified this in the revision in Sec. 4.1.1.

b) Thanks for your insightful question. We have incorporated this discussion into our revision in Appendix A.1. Specifcally, we collect 2,540 additional samples from open-world datasets and scenarios (Object365 [A] and OpenImage [B]) to further evaluate the genaliation ability of our method quantitatively.

The above results present the comparison with baseline results for the evaluation on out-of-domain datasets. After incorporating visual evidence prompting, all models enables more precise discernment of object or relation presence within the image.

Besides Figure 5 in the draft that showing some out-of-domain cases, we also follow CLIP and select 2 samples from 10 open-world datasets (without groundtruth) for qualititive analysis, including CLEVER and Caltech 101. These 20 cases are in the revised Appendix F.1.

The contribution of our method lies in combining small and large models, utilizing the domain-specific knowledge of small models to complement the large models. In practical applications, it is possible to customize different small models to tailor different domain knowledge. The Figure 3 in the draft verified that if the small models struggle to effciently capture visual evidence, incorrect evidence has limited impact on cases that are already correct.

[A] Objects365: A Large-scale, High-quality Dataset for Object Detection
[B] The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale

Q2: Boosting and bagging methods:

The authors should extend their comparisons to include boosting and bagging methods, especially considering the paper primarily demonstrates performance improvements for yes/no VQA questions. This would provide a more robust validation of the proposed method.

Ans for Q2:

Thanks for your insightful comments. In response to the suggestion, we conduct the experiments of utilizing the conventional bagging methods [A] between LVLM and VQA models which does a plurality vote when predicting. Combining the Qwen-VL-Chat and N2NMNs, the evaluation accuracy of object and relation hallucination slightly increases from 81.23% to 83.02% and from 63.62% to 67.91%. This result further validates the effectiveness of our framework.

[A] Bagging predictors

Dear reviewer w1s8,

We sincerely apologize for inconveniencing you, but we have no other choice at the end of the discussion period. We sincerely hope you can understand.

Thank you for taking the time to review our work. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work. Your support or feedback are very important to us. We greatly appreciate your constructive comments and efforts.

Best regards

Authors of #3272

We sincerely thank the reviewer for taking the time to review. We appreciate that you find our approach intriguing, the motivation clear, and the presentation good. According to your valuable comments, we provide detailed feedback.

Q1: Efficiency:

What degree of efficiency degradation can be expected with the introduction of new mini models for object detection and scene graph generation into the original method?

Ans for Q1: Thanks for your constructive comments. We compute the average inference time of several small models on one A100 GPU.

Meanwhile, the average time for one inference of Qwen-VL-Chat and mPLUG-Owl is:

After incorporating the small models, the relative increase in inference time is an average of 21.78%. Moreover, in multi-turn dialogue scenarios, the image is only uploaded once, regardless of how many questions are asked. The small model only needs to perform inference once, making the cost of inference relatively smaller.

Q2: Domain Compatibility:

a) Is there any overlap between the datasets used for training the small target detection and scene graph generation models and the LVLM hallucination test set?
b) In the context of an open-world scenario where the small model might struggle to efficiently capture target and relation information, to what extent will the acquisition of false evidence affect the LVLM's otherwise accurate responses?

Ans for Q2:

a) Thanks for pointing out this potentially confusing point. There is no overlap between the datasets used for training small models and the LVLM hallucination test sets. We are using the open-source small models trained on the COCO or Visual Genome training set. The test set we have constructed is derived from the COCO validation set or Visual Genome test set. We have clarified this in the revision in Sec. 4.1.1.

b) Thanks for your insightful question. We conduct more evaluations and incorporate the discussions into our revision in Appendix A.1.

• Firstly, we collect 2,540 additional samples from open-world datasets and scenarios to further quantitatively evaluate the genaliation ability of our method. More details of these two out-of-domain datasets Object365 [A] and OpenImage [B]are added in the revision (Appendix A.1). In 20.6% of the images, small model captures incorrect or partial correct object or relation information. With these visual evidences, only 8% of the false evidence confuse the LVLM and change the response from collect to wrong.
• Secondly, besides Figure 3, we follow CLIP and select 2 samples from 10 open-world datasets (without groundtruth) for qualititive analysis. These 20 cases are in the revised Appendix F.1. From these results and cases, it can be seen that in open-world scenarios, incorrect evidence still has limited impact on cases that are already correct.
• Thirdly, The fifth prompt template in Table 3 in the draft shows that we can tell the LVLM that the evidence might be wrong and keep your answer if you think the evidence is wrong or missing.

In summary, quantitative and qualititive analysis show that false evidence has limited affect on LVLM's accurate responses with the help of carefully designed prompt.

[A] Objects365: A Large-scale, High-quality Dataset for Object Detection
[B] The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale

Dear reviewer rr7K,

We sincerely apologize for inconveniencing you, but we have no other choice at the end of the discussion period. We sincerely hope you can understand.

Thank you for taking the time to review our work. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work. Your support or feedback are very important to us. We greatly appreciate your constructive comments and efforts.

Best regards

Authors of #3272

Dear reviewer rr7K,

We sincerely apologize for inconveniencing you again, but we really have no other choice as the discussion period is drawing to a close (only 1 days left). We sincerely hope you can understand.

Thank you for taking the time to review our work. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work. Your support or feedback are very important to us. We greatly appreciate your constructive comments and efforts.

Sorry again for inconveniencing you.

Best regards

Authors of #3272

Dear reviewer rr7K,

Deep apologies for the repeated interruption. Sorry so much!

We are truly honored that you have reviewed our paper and provided valuable constructive suggestions. Sincerely thanks for these constructive suggestions which greatly aid in our paper’s refinement, we have diligently addressed every concern and question you raised during the initial review. We genuinely hope our responses have resolved your concerns and provided satisfactory explanations. We sincerely appreciate your dedication and valuable time.

To further improve our work, we sincerely hope for a valuable opportunity to discuss with you. It would be greatly appreciated if we could have a chance to hear your feedback. Your feedback is highly valuable to our paper and greatly contributes to the entire community.

Very sorry for inconveniencing you again. Hope you have a good day!

Best regards and many thanks,

Authors of #3272