Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuning

W1. The real-world applicability of adversarial questions, as demonstrated in the examples provided, seems limited. The likelihood of such adversarial scenarios occurring in practical use cases appears low, and hence, this might not be a widespread issue.

Dialogue hallucinations represent a significant real-world issue, particularly in critical domains such as healthcare, legal, and financial services, where the user chats with the services throughout multiple round of conversations so accurate and trustworthy responses are paramount. For example, if a user with diabetes discusses having a transient cold and seeks advice on diabetes management, it would be highly dangerous if the treatment were hallucinated based on this context.

W2. Many recent works, such as ShareGPT4V, have tackled hallucination issues with better training data. The paper does not evaluate its methods on models that are known to hallucinate less, leaving the effectiveness of the proposed method for stronger models unclear.

Thank you very much for introducing the important relevant work. As suggested, we evaluted ShareGPT4V on EvalDial with GQA and Flickr-30k dataset and the results are shown below. ShareGPT4V shows relatively better performance than LLaVA v1.5, but the performance on Adversarial dialogues are still not better than AIT. This shows our proposed methods is still valuable in improving model's capability in mitigating hallucinations.

W3. Section 3.1: Even by the paper’s own definition of hallucination (line 155: “the generated content that is nonsensical or unfaithful to the given source content”), the examples provided don’t clearly qualify as hallucinations. For instance, the responses in Figure 1 and Figure 2 make sense given the incorrect text from the previous dialogue. For example, in Figure 2 (mid-column), the answer "Antarctica" makes sense in relation to the mention of penguins from the prior round.

To be clear, line 155 “the generated content that is nonsensical or unfaithful to the given source content” is only a reference, but not part of our definition of the dialogue hallucination. We call a generated answer faithful without any dialogue but becomes unfaithful after some preceding dialogues, dialogue hallucination. In Figure 2, the original answer was "Asia". We acknowledge that there might be some ambiguity related to Figure 2. We presented a clearer example and updated Figure 2 to demonstrate three types of prepended dialogues.

Dear Reviewer k5BF, we appreciate your valuable comments on our paper. We have prepared a rebuttal with an updated manuscript and tried our best to address your concerns. We notice that the author-reviewer discussion period is coming to an end, and we are willing to answer any unresolved or further questions you may have regarding our rebuttal if time is allowed. If our rebuttal has addressed your concerns, we would appreciate it if you would let us know your thoughts. Additionally, we will be happy to answer any further questions regarding the paper. Thank you for your time and consideration.

W4&Q4. How does the DTAR values in Table 1 prove that the hallucinations are caused by prepended dialogues? It is suggested that the author test whether the DTAR value is affected by the relative position of the prepended hallucination dialogue and the current question.

We thank the reviewer for the careful comments. The DTAR score quantifies the proportion of influence of dialogue tokens over visual tokens. The DTAR usually exhibits low values since LVLM is dominantly influenced by visual tokens from the vision encoder. However, in the hallucinated case, the DTAR value increases so that it influences the latent features of the LVLM, hallucinating the output answer. That is, even with a seemingly small increase in absolute value, such as 0.19 to 0.37, it can significantly change the output answer, causing hallucination. For LLaVA, the high AUC-ROC score of 0.93 in Figure 4 indicates a substantial difference in DTAR score distribution between non-hallucinated and hallucinated cases.

Furturemore, per your suggestion, we conducted additional experiments measuring the DTAR value on prepended adversarial dialogues at different positions in a multi-round setting. Specifically, we set the number of round to 2, and prepended an adversarial dialogue in the first or second position (with a General dialogue placed correspondingly in the second or first position) for LLaVA-v1.5 model. As shown in the table below, the DTAR scores of hallucinated cases are also higher than that of non-hallucinated case. Also, the closer the dialogue is to the target question, the greater its impact.

W1&Q1. Could you provide a detailed analysis of how the information in the preceding dialogues leads to hallucinations in the model?

Thank you very much for your thorough assumptions and suggestions. From Table 2, we observe that prepending the three types of dialogues degraded the model's ability to varying extents. We believe the hallucinations are caused by a combination of incorrect predictions of the current question and conflicts between the prepended questions and the current question. At your suggestion, we divide the Adversarial type dialogues into one of three types: 1) incorrect information, 2) unrelated information, and 3) long information using GPT-4o. The distribution table of these three types of dialogues for each OKVQA and NoCaps dataset is provided below.

We then calculated the hallucination rates for the three types of preceding dialogues respectively on those datasets, and the results are shown in the Table below. From the table, LLaVA model shows similar performance across the three types of adversarial dialogues, with the worst results occurring with correct and long preceding dialogues. This is a valuable analysis and results, and we will include this part in our main paper to facilitate future study.

W2&Q2. In the benchmark, is a hallucinated prepended dialogue added before each question? If multiple rounds of dialogue are added before the current question, and the hallucinated dialogues are positioned differently within these rounds (e.g., at the beginning or just before the current question), or if the number of hallucinated dialogues varies, how would the model's performance change?

Thank you for your constructive comments. In our benchmark, for a single-round prepended dialogue case, we prepended the hallucinated dialogue before the target question (See Table 2). For multi-rounds dialogue case, we added the hallucinated dialogues for every round of dialogue before the target question, which can measure the effect of varying number of hallucinatory dialogues (See Table 6). The results shows that more prepended dialogues make the model less likely to answer the target question correctly. In addition, as per your suggestion, we conducted additional experiment with different position of the adversarial dialogues and shows the result in Table below. For each tested QA pair, we prepended four rounds of dialogues with adversarial dialogues appear at different positions and use General type dialogues to fill in the rest of the dialogues. From the result, the adversarial dialogue has more influence on the final accuracy as it positions closer to the final test question. We will include this result in the final version, and again, thanks for helping us improve our manuscript.

W3&Q3. It is suggested to add a statistical table to analyze the benchmark, such as the distribution of dialogue types, and average dialogue length.

Figure 6 of our draft illustrates the semantic distribution of each dialogue type within our benchmark. Specifically, it shows how far General, Random, and Adversarial dialogues are from the target question in the LVLM semantic space. The distribution indicates that adversarial questions are the closest to the target question, indicating them as subtle cases. Additionally, we provide the average token length of each dialogue type in our benchmark at the table below. We will add this statistics in the final version.

Dear Reviewer RNog, we appreciate your valuable comments on our paper. We have prepared a rebuttal with an updated manuscript and tried our best to address your concerns. We notice that the author-reviewer discussion period is coming to an end, and we are willing to answer any unresolved or further questions you may have regarding our rebuttal if time is allowed. If our rebuttal has addressed your concerns, we would appreciate it if you would let us know your thoughts. Additionally, we will be happy to answer any further questions regarding the paper. Thank you for your time and consideration.

W1. The paper’s focus on hallucinated dialogue scenarios limits the ability to gauge the model’s general performance in non-hallucinated contexts. A broader evaluation that includes non-adversarial scenarios would provide a more comprehensive view of the model’s robustness and applicability.

We thank the reviewer for this constructive comment. The model's overall performance in non-hallucinated contexts is already included in the "None" column of Table 2 in our draft. This refer to the non-adversarial scenarios without prepended dialogues, and our AIT shows comparable performance to the vanilla LLaVA-v1.5.

W2. The study relies heavily on the CIDEr metric for evaluating image captioning, limiting insights into overall performance. Results using additional metrics would provide a more balanced view of the model’s capabilities.

In Table 10 of Appendix D, our draft included the study with other captioning metrics, such as FAITHSCORE, which is a metric designed to evaluate the factual accuracy of outputs from LVLMs by verifying the consistency between generated text and visual inputs, and this metric demonstrates highly correlations with human judgments of faithfulness [1]. From the results, AIT is still more effective in maintaining the Faithscore than LLaVA v1.5, meaning that AIT's ability to reduce the dialogue hallucination is consistent with other metrics.

[1] FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models, ArXiv, 2024.

W3. While the Adversarial Dialogue is designed to be image-related and likely to induce hallucination in response to the original test question, it introduces content that is largely independent of the question itself. This raises the question: if the adversarial content is unrelated to the core test question, would simply resetting the chat between exchanges be a more straightforward solution?

Thank you very much for your insightful comment. What you proposed could be a simple fix if people are aware such dialogue hallucination exists. However, there are still many scenarios where previous unrelated dialogue history is essential and valuable. For example, a patient previously discussed dietary habits and now reports a specific symptom. Understanding the patient's diet can help the LVLM identify potential causes of the symptom, leading to more accurate advice. It is often unclear whether a previous content is related or not to the core test question. Therefore, merely resetting the chat between exchanges is likely not an appropriate solution in many cases, and our method that guides robust responses against adversaries is necessary.

Q1. Would AIT still demonstrate superior performance if prepend responses were correct, non-hallucinatory answers?

Our "General" type prepended dialogues could represent the correct, non-hallucinatory scenario, which is defined in Section 3.2. This type of question is about the general information of an image and each specific question is shown in Appendix A. As shown in Table 2 "Gen" column, AIT still shows better performance than LLaVa v1.5 with General type dialogues prepended.

Q2. Does the Masked Instruction Tuning approach risk overly discouraging reference to preceding dialogues?

This is an excellent question. We believe that our masked instruction tuning approach can actually encourage models to identify appropriate references from preceding dialogues. The adversarial question is unmasked and only the answer is masked, so our model is forced to learn the question-answer that is relevant to the target question. For example, in the training dataset, suppose there are preceding dialogues necessary to answer the target question, but there are also subtle adversarial dialogues prepended right before the target question. When we train an LVLM on this dataset, the model must identify the appropriate references in the preceding dialogues to answer the target question despite the presence of adversarial dialogues. This process may help the model become robust in distinguishing adversarial contexts and effectively finding relevant references.

Q3. It would be valuable to understand how the model’s performance in captioning tasks changes as the length of dialogue rounds increases.

Thanks for mentioning this experiment to make our paper more robust. We conducted multi-round experiments on Captioning task, particularly on Flickr-30k dataset and the result is shown in the Table below. Similar to the VQA task, more prepended adversarial dialogues make the model less likely to answer the target question correctly, and AIT maintains its performance on the Adversarial case. We will include this result in the final version.

We sincerely appreciate the reviewers' constructive comments and positive feedback on our manuscript.

Q1. Does AQG-generated data somehow reflect the real distribution of LVLM's hallucination behaviors? Or is it just a way to create fake adversarial dialogue that serves AIT's finetuning purpose?

As our objective listed in Section 3.3, AQG works to generate natural sounding dialogs. The distribution of real-world dialog are vast. However, we expect them to be natural sounding, and therefore AQG should be effective even in real world use cases.