Dear Reviewer 4B1M,

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point in the rebuttal.

Weaknesses

Weakness 1 (About additional latency)

Ans. Thank You for the comment. We would like to make a few points in our support:

• Improving baseline LVLMs typically requires: (i) high-quality datasets with cognitive prompts, which are challenging to collect; (ii) better architectures; and (iii) re-training the LVLM. Our proposed method eliminates the need for any of these and introduces a training-free technique that makes the first attempt at improving reasoning on cognitive prompts in LVLMs.
• Furthermore, our approach aligns with existing training-free hallucination mitigation techniques (e.g., VCD, LURE, PAI, HALC, etc.) in both motivation and methodology, though the algorithm itself is novel.
• Most existing methods (including [1,2,3] mentioned by the reviewer) address hallucinations by exploiting the high confidence of language hallucinations in language-only or corrupted image scenarios. While this is effective for mitigating object hallucinations, we demonstrate in Section 3 that these techniques are inadequate for cognitive prompts. Consequently, VDGD employs a distinct mechanism tailored to cognitive prompts, which, while slightly more computationally intensive, is necessary for addressing this unique challenge. We would like to show a quick computational analysis:

Computational Analysis on LLaVA 1.5 with 48GB GPU (on MMAU):

As we cam see, VDGD only leads to a slight increase computational complexity but also leads to substantial improvement in performance, especially on cognitive prompts (all other methods are meant for alleviating object hallucinations). We further show that VDGD is competitive with a small captioning model (which is not the LVLM) and the key is to generate an image caption that captures the visual elements. We hope you find this new results insightful. This shows that efficient captioning models also has the potential effectively reduce VDGD complexity.

We have cited this in the revised version of our paper and added a Discussion in the Appendix N to discuss the differences with these methods.

Weakness 2 (About comparison with CoT methods)

Ans. Thank You for the comment. We provide a comparison below with the cited methods:

• [4] Compositional Chain-of-Thought Prompting for Large Multimodal Models, CVPR 2024. Constructs a scene graph by asking the LVLM to look at the image and the given question and to identify three main properties: objects, object attributes and object relationships. The scene graph is then passed to the LVLM with the query again to get the final response. This method primarily solves compositionality in real world scenes. However the proposed method would be ineffective when the input does not contain real world objects, for example find the peak in a stock price graph.

• [5] Beyond Embeddings: The Promise of Visual Table in Visual Reasoning, EMNLP 2024. Proposes a Visual Table, which is trained on GPT4V generated data. The Visual Table generates hierarchical descriptions of visual scenes, featuring a scene description and multiple object centric descriptions covering categories, attributes, and knowledge. The input image is first passed to the Visual Table Generator and then the image query and generated data from VT are passed to the LVLM to get output. This method achieves performance improvement in real-world and non-real world scenes. The disadvantages are this method is not training free and requires significant computational resources.

• [6] Visual Evidence Prompting Mitigates Hallucinations in Multimodal Large Language Models Proposes using a small visual model which acts as an object detection model. The model also generates object description, object locations and object relations. The generated data along with input image and user query is passed to the LVLM. This paper suffers from the same disadvantages of CCoT where it is ineffective with non-real world scenes and datasets like MMMU, MathVista etc.

We have cited this in the revised version of our paper and added a Discussion in the Appendix N to discuss the differences with these methods.

We also provide a comparison of results below:

Continued

Weakness 3 (About the results on the defined categories)

Ans. Thank you for your question. We would like to clarify a potential misunderstanding here. The newly defined hallucination categories pertain to image descriptions and not cognitive QAs. These methods cannot be directly applied to categorize hallucinations in cognitive QAs – for example, instruction-response pairs in typical visual instruction tuning datasets do not contain cognitive prompts. Additionally, as shown in Fig. 6, unlike in image description tasks which have a mix of various hallucinations presented in Section 4, hallucinations in cognitive prompts are dominated by language hallucinations are predominant in. Thus, hallucinations for cognitive prompts cannot be effectively categorized into our mentioned types which is why we do not show results on the same.

But why is this analysis required here? To clarify, our analysis and categorization aim to investigate what is needed to mitigate hallucinations in cognitive prompts, thereby improving reasoning. Section 3 highlights that current mitigation techniques fail to address hallucinations in cognitive prompts. Section 4 categorizes hallucinations and suggests that existing methods, such as logit correction, are inherently tailored to mitigate language hallucinations. Finally, Section 5 demonstrates that (i) for image descriptions, hallucinated tokens are high-confidence single tokens in the logit space and can often be corrected by language-only models with the same prefix; (ii) in cognitive prompts, hallucinations arise from a top-$k$ probability space dominated by low-confidence tokens, making current methods ineffective (lines 70-75).

Building on these findings, VDGD addresses this challenge by grounding response generation to image descriptions, boosting the confidence of correct tokens and mitigating hallucinations more effectively.

Weakness 4 (About the overall flow and the requirement of the analysis)

Ans. Thank you for your observations. We would like to take this opportunity to clarify the flow of information in our paper. Our work is structured to first investigate what is required to effectively mitigate hallucinations for cognitive prompts and thereby improve reasoning.

• Section 3 explores why current mitigation techniques fail to address hallucinations in cognitive prompts. We conduct experiments using various hallucination mitigation methods, LVLMs, and datasets, demonstrating that existing approaches only reduce hallucinations in image description-based prompts, specifically for natural scenes.

• Section 4 delves into the reasons behind this limitation. We categorize the types of hallucinations observed and highlight that most existing methods are tailored to address language hallucinations. For example, techniques like logit correction target highly confident tokens in language-only or premature model outputs, which may inherently restrict their effectiveness to language hallucinations due to their design choices.

(continued in next part)

Dear reviewer 4B1M,

Thank you for taking the time to review our paper. We have addressed your concerns in our submitted response and provided a revised version of the paper. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!

Best,
Authors of Submission4948

I would like to thank the authors for their detailed response. I have carefully reviewed the rebuttal and appreciate your patience while awaiting my feedback. I am willing to consider raising the score if the following points are addressed:

W1.

I remain unconvinced by the latency analysis presented. Considering that most LVLM benchmarks typically require only a short binary answer (e.g., “Yes” or “No”) or a multiple-choice response (e.g., “A, B, C, D”), the output generation involves merely a single token. While VCD necessitates two forward passes to contrast probability distributions derived from the original and distorted images, it similarly requires generating only a single token for the final response (with the constraint of answering in one word, such as “please answer in one word”).

In contrast, the proposed method appears to have significantly higher latency. It requires pre-generating detailed image descriptions, which can span hundreds of tokens, followed by a prefill phase involving predefined prompts and the generated descriptions before arriving at the final output. This multi-step process is likely to result in considerably longer latency.

To strengthen the argument, I would suggest including a detailed latency breakdown for each step of the pipeline. Additionally, clarification on the “small captioning model” mentioned is needed, as its specifics have not been provided.

W2.

While the authors have stated that the listed methods are cited in the revised version, this does not appear to be the case upon review. Furthermore, I believe that the visual CoT methods discussed are highly relevant to the proposed approach and should be included in the main paper.

Ideally, the proposed method should also be compared with these methods more comprehensively. However, given the constraints of the review timeframe, I understand that a full comparison may not be feasible. At the very least, these related methods should be properly discussed in the main paper to provide necessary context and establish their relevance to the proposed approach.

Thank you once again for your efforts. I look forward to your clarifications on these points.

Thank you for your prompt response.

W1.

I am still not fully convinced by the latency analysis provided. While I understand that most models are not inherently designed to respond in a single word (e.g., “Yes” or “No”), it is possible to constrain their outputs (using "please answer in a single word") to such a format when the task allows for binary or multiple-choice responses. This can be achieved through careful prompting to ensure concise outputs in scenarios where single-word answers are appropriate.

[VCD]

• forward pass with original image: (# image tokens = 576 + # prompt length = 30~50) -> # output length = $N$
• forward pass with distorted image: (# image tokens = 576 + # prompt length = 30~50) -> # output length = $N$

[VDGD]

• first forward pass: (# image tokens = 576 + # prompt length = 30~50) -> # output length = $M$ (description length)
• second forward pass: (# image tokens = 576 + # prompt length = $M$ + 30~50) -> # output length = $N$

In the case of constrained decoding, $N = 1$ for VCD (e.g., binary or multiple-choice questions). While I recognize that this may not always apply to open-ended or detailed questions, VCD remains more efficient in its average-case scenario due to the shorter input/output requirement. In contrast, VDGD involves generating longer descriptions, with $M$ often far exceeding $N$. Additionally, VDGD incurs the extra cost of prefilling $M$ additional tokens during the second forward pass, which adds to the overall latency.

I am particularly curious about how the latency of the first forward pass (caption generation) for VDGD compares to that of the second forward pass (response generation). This discrepancy likely depends on the configuration of max_token_length and warrants further clarification.

As a result, I find the following statement from the rebuttal unconvincing: “VCD-like logit correction proves to be much more expensive than VDGD as the extended context length for each new token requires two passes now.”

Also, the “small captioning model” referenced in the paper should be properly cited.

Dear Reviewer 4B1M,

Thank You for the time in reading our rebuttal. We are glad we could clarify most of your concerns. We would like to take this opportunity to respond to the further weaknesses mentioned by you in your response to our rebuttal.

W1 (About breakdown of latency)

Ans. Thank You for your question. We would like to present a 3-way comparison of latency on MMMU and VaLLu for LLaVa-1.5. Before we go ahead with our analysis, we request you to note two important points:

• For the input prompt, beyond the length of the text prompt, the vision encoder adds 576 tokens to the total context length (in case of LLaVa 1.5 which employs CLIP ViT-L/336px)
• We request you to note that VDGD is comprised on all open-ended generation responses. MMMU on the other hand has 689 open-ended generation questions and not all single word answers. Finally, most models do not respond in just one word and sometimes add additional context to the response (e.g., Yes, ....)
• We also would like to re-iterate that a majority of results and analysis in our paper are focused towards evaluating and improving open-ended generations in LVLMs as they more closely reflect ideal real-world interactions with LVLMs.

Vanilla Greedy Sampling

• MMMU: Single Forward Pass: Time: ~1.3 seconds TFlops: 9.3
• VaLLu: Single Forward Pass: Time: ~1.6 seconds TFlops: 11.0

VCD (VCD analysis also reflects a majority of the logit correction methods proposed in literature)

• MMMU: Two consecutive forward passes for each token (assuming both copies are in the same GPU): Time: ~1.9 seconds TFlops: 14.6 for LVLM w/ image and 5.4 for LVLM w/o image (average ~10.2)

• VaLLu: Two consecutive forward passes for each token (assuming both copies are in the same GPU): Time: ~2.5 seconds TFlops: 18.9 for LVLM w/ image and 8.9 for LVLM w/o image (average ~13.6)

Please note that as output length increases in real-world open-ended generations, VCD-like logit correction provides to be much more expensive than VDGD as the extended context length for each new token requires two passes now

VDGD

• MMMU: First Forward Pass for Caption: Time: ~1 seconds TFlops: 9.3 (same as above greedy as image tokens take most processing and response tokens contribute minimally)
• MMMU: Second Forward Pass for Response Generation Time: ~1.4 seconds TFlops: 14.5
• VaLLu: First Forward Pass for Caption: Time: ~1 seconds TFlops: 9.3
• VaLLu: Second Forward Pass for Response Generation Time: Time: ~1.4 seconds TFlops: 14.5

Please note that for VDGD, our implementation (codebase in Reproducibility section of our paper) saves model logits during caption generation which is used directly by the model for response generation. Adding it to context during generation did not lead to score change.

VDGD with small captioning model

• MMMU: First Forward Pass for Caption: Time: ~0.4 seconds TFlops: 6.3
• MMMU: Second Forward Pass for Response Generation Time: ~1.4 seconds TFlops: 14.5
• VaLLu: First Forward Pass for Caption: Time: ~0.4 seconds TFlops: 6.3
• VaLLu: Second Forward Pass for Response Generation Time: Time: ~1.4 seconds TFlops: 14.5

Thus, we would like to conclude with 3 points:

• We hypothesize that the input token processing is responsible for a majority of the compute. This has also been discussed in prior art. In comparison, the captions generated are around ~30-50 tokens (only <10% additional tokens)
• In real-world cases with open-ended generations, VDGD is competitive in terms of compute taken for responding with other hallucination mitigation techniques proposed in literature.
• VDGD can be implemented in a variety of methods. For example, a small captioning model, or the logits can be saved prior to generation (which was done in our case for benchmark result generation)
• We also acknowledge in our rebuttal that VDGD iadds computational overhead. However, we expect the overhead to keep getting lower with advancements in captioning models.

Finally, for the small model, we employ: SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation (CVPR 2023)

Thank You again and we hope we have responded to your queries!

W2 (About discussion with visual CoT methods)

Ans. We have just updated our paper with these addition to the main paper! We have cited your mentioned works and added a short discussion in lines 101-107 of the revised version of our paper.

Thank You again for your time. Your feedback is invaluable to us in improving the quality of our paper.

Thank you for the detailed response throughout the discussion. My concerns are mostly resolved.

I am increasing my score to 8.

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point in the rebuttal.

Questions

Question 1 (about quantitive scores)

Ans. Thank You for your question. We provide the scores below:

Question 2 (about base rank in language hallucination)

Ans. Thank you for the question. We would like to clarify a potential misunderstanding. As you correctly noted, base ranks (BR) are non-negative. In Section 3.3, Algorithm 1 specifies that a BR = 0 signifies a language hallucination, not a BR < 0.

To clarify further: if BR > 0, it is not a language hallucination; otherwise (i.e., BR = 0), it is classified as a language hallucination. There is no case where BR < 0 in this context.

Question 3 (About the classification of IT instances)

Ans. Thank you for the question. We would like to address your concern by highlighting three key points that demonstrate the robustness of our method:

• IT hallucinations originate explicitly from the information transfer (IT) stage and can be directly matched with entries in the IT dataset. As noted in lines Section 3.3 of our paper, this makes IT hallucinations the most explicit and identifiable type, with strong evidence for their cause.

• Appendix K.5 provides ablations for various values of k, showing that while the count of IT hallucinations may vary slightly with k, the counts of other hallucination types remain stable. This demonstrates that categorizing IT hallucinations first does not impact the classification of visual or style hallucinations.

• The consistent count of IT hallucinations across different k values reinforces that IT hallucinations are not misclassified or inflated due to other causes. By categorizing them first, we ensure accurate and unbiased classification of other hallucination types.

Question 4 (Section 4.1 length of text prompts)

Ans. Thank you for the question. First, we would like to clarify a potential misunderstanding: the X-axis in Figure 6 represents the token position in the response, not the prompt length (as stated in the caption and lines 300-309). The figure shows a sharp decline in Base Rank after the second token, after which the curve flattens, with an average Base Rank between 0 and 1 throughout the rest of the response. Since the first two tokens are likely style tokens (e.g., This, The, etc [1]), they contain minimal content for the model to leverage. This sharp drop and sustained low Base Rank demonstrate that the model predominantly relies on language priors rather than attending to the image, a phenomenon we term the alignment gap.

While longer prompts naturally introduce more textual context, the model is not expected to rely heavily on earlier response tokens during auto-regressive generation. For example, in tasks like image description generation (e.g., AMBER), a model should ground its responses primarily in the image context rather than earlier response tokens. The observed pattern further highlights the misalignment between visual grounding and the model’s reliance on language priors, which we address in our work.

Dear reviewer cozw,

Thank you for taking the time to review our paper. We have addressed your concerns in our submitted response and provided a revised version of the paper. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!

Best,
Authors of Submission4948

Thank You for reviewing our response and the rebuttal. Your feedback is very important for improving the quality of our paper.

We would like to highlight a possible overlook: For various methods, AMBER has much higher performance improvement than DONUT in the Tables provided in our rebuttal:

• For Woodpecker, AMBER has an improvement of +0.30 , but DONUT only has +0.02
• For Opera, AMBER has an improvement of +0.18 , but DONUT only has +0.05
• For VCD, AMBER has an improvement of +0.25, but DONUT has -0.06 (performance decrease)
• For LURE, AMBER has an improvement of +0.15, but on DONUT has +0.10. We acknowledge that the gains are almost similar only in LURE (AMBER still has +0.05 improvement higher than DONUT). We hypothesize that LURE is a fine-tuning method and not a training-free method.

The relative gap in gains are substantial and is correlated with all other results presented in our paper.

---

Additional Thoughts.

All results are averaged across 3 runs.

Additionally, both benchmarks are visual element recognition benchmarks -- while AMBER is visual element recognition for natural scenes, DONUT is visual element recognition for document understanding (or OCR). Scores on DONUT are also overall lower than AMBER. None of them are cognitive QA benchmarks, which is the main focus of our analysis from Section 3.1. We would like to highlight two additional points:

• Our main motivation for this analysis is that current mitigation techniques are algorithmically built for mitigating only visual element hallucinations in natural scenes. Our hypothesis is detailed in lines 270 - 278 of our paper. An intriguing example is Woodpecker which depends on object recognition and grounding (not applicable in real-world scenes) and VCD where logit correction works due to extended descriptions with high confidence generated by the LVLM.

• Our motivation of VDGD is not grounded to this analysis. VDGD is motivated by the fact that current hallucination mitigation techniques do not perform well on mitigating hallucinations in cognitive prompts. This analysis can be seen in Section 3.1 where truly none of the hallucination mitigation techniques improve on cognitive QA benchmarks. Our analysis in Section 3.2 is just to further strengthen our analysis in Section 3.1 and investigate the cause for lower performance on cognitive QA benchmarks which ideally do not have real-world scenes.

---

We respectfully hope you can consider our response and we are happy we were able to resolve your other queries.

Best,
Authors of Submission4948

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point in the rebuttal.

Weaknesses

Weakness 1 (About reasoning on data with natural scenes)

Ans. Thank you for the insightful comment. We acknowledge that VDGD assumes visual content can be sufficiently represented in text. While this may seem more suited for structured scientific images like charts and tables, our evaluation in Table 2 also demonstrates that VDGD is effective across a range of non-scientific image benchmarks, such as MMVet, LLaVA-Bench, and Oven. VDGD leverages the LVLM's ability to identify key visual elements—such as objects, attributes, and their relationships—which ensures accurate descriptions. This capability allows VDGD to generalize effectively, reducing hallucinations in both structured scientific images and complex, unstructured natural images.

Additionally, recent advanced and foundational models (e.g., Qwen 2 Vision) have shown superior capabilities in capturing details in natural scene images. Finally, we as image captioning methods keep improving and their capabilities to capture every minute visual element improves, we hypothesize that the performance of VDGD will keep improving.

Weakness 2 (About definition of cognitive reasoning)

Ans. Thank you for the comment. We would like to clarify a potential misunderstanding. We would first to reiterate the VDGD process:

• As discussed in Section 4 of our paper, the top-k space of hallucinated tokens in responses to cognitive prompts (those requiring reasoning and knowledge) is dominated by low-confidence and equally likely tokens, a result of the alignment gap (lines 340-342). This finding is unique to cognitive prompts, and VDGD is specifically designed to address this issue.
• By grounding responses to image descriptions, VDGD increases the confidence of the correct token. This improvement is crucial during auto-regressive generation, as it prevents hallucination snowballing and enables accurate responses. VDGD operates on a simple yet intuitive principle, similar to how humans write down observations from an image to guide reasoning tasks.

To summarize, VDGD does not merely prepend descriptions to improve reasoning. Instead, it leverages grounding to boost token confidence, ensuring more accurate responses. Finally, as by VDGD’s performance on benchmarks like MMVet [4], LLaVa-Bench [5] and Oven [6] which involves reasoning beyond simple data representation.

We also state that" VDGD can be seen as analogous to how humans think – “Much like humans, who often write down their observations of an image in their own words and refer back to them while tackling complex reasoning tasks.

Weakness 3 (About other forms of reasoning)

Ans. Thank you for the comment. Our work focuses broadly on cognitive prompts, which require reasoning or knowledge to generate responses, and not on specific types of reasoning—this term has been consistently used throughout our paper. Benchmarks such as MMVet, LLaVA-Bench, and Oven include non-scientific instances that include cognitive prompts (or require reasoning). As stated earlier, if the description captures the details of the image, VDGD effectively guides generation to reduce hallucinations. Additionally, benchmarks like LISA [7], which involve segmentation tasks, and instances in the mentioned benchmarks demonstrate reasoning beyond scientific contexts.

Weakness 4 (Comparison with [2] on hallucination categories)

Ans. Thank you for the question! Upon reviewing [2] in detail, we find that while [2] categorizes hallucinations by type (e.g., object, attribute, and relation hallucinations, as referenced in lines 237-266 of our paper) and attributes their causes to broader factors like data, architecture, or connection modules, it does not categorize hallucinations based on their decoding-time origin or analyze and link their causes to behaviors in the logit space—a key focus of our work. Our contribution lies in proposing a novel approach that first categorizes hallucinations by type and then by their specific cause, including decoding-time factors and logit-space dynamics. This perspective is critical for understanding and mitigating hallucinations, as demonstrated in our analysis and findings.

Weakness 5 (About more ablations)

Ans. Thank you for your comment. The Appendix of our paper contains extensive ablation studies. For instance, Table 3 presents key ablations for VDGD, while Tables 5–9 include additional experiments:

• Scores of LVLMs on rephrased prompts without images (Appendix L.1)
• Performance on the VaLLu benchmark when only image descriptions are provided (Table 3)

Dear reviewer fYgL,

Thank you for taking the time to review our paper. We have addressed your concerns in our submitted response and provided a revised version of the paper. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!

Best,
Authors of Submission4948

Dear Reviewer fYgL,

Thank You for your time in reviewing and responding to our rebuttal. We are grateful and we are confident your comments and suggestions can help us improve the quality of our paper.

At your request, we have made 2 major additions and uploaded a revised version of our paper:

1. We have added evaluation results on the VaLLu benchmark for (i) VDGD + GPT-4 captions (a stronger captioning model) and (ii) VDGD + [1] as a captioning model (a smaller but more compute efficient captioning model). We present the results below and have added these rows to Table 3 in the Appendix with other ablations:

We also show VDGD (-) VCD results for comparison (VDGD without VCD-based decoding for caption generation. As we can see, VDGD shows notable performance gains with GPT-4 captions, highlighting the impact of high-quality captions. It also performs competitively with smaller captioning models (SmallCap -- https://arxiv.org/abs/2209.15323), suggesting future improvements in small and better captioning models can further enhance VDGD’s effectiveness and efficiency.

The results were accumulated as efforts to your questions and a similar question by Reviewer 4B1M.

2. We have explanation to two places in our paper for the same: (i) lines 509 - 511 in the main paper and (ii) lines 816 - 822 in the Appendix where we have thoroughly explained this phenomena.

We thank you again and request you to please let us know if we can clarify anymore of your concerns. We appreciate your willingness to raise our score and we look forward to it!

Best,
Authors of Submission4948

Dear Reviewer 9PSE,

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point in the rebuttal.

Questions

Question 1 (About VDGD results on newly defined categories)

Ans. Thank you for your question. We would like to clarify a potential misunderstanding here. The newly defined hallucination categories pertain to image descriptions and not cognitive QAs. These methods cannot be directly applied to categorize hallucinations in cognitive QAs – for example, instruction-response pairs in typical visual instruction tuning datasets do not contain cognitive prompts. Additionally, as shown in Fig. 6, unlike in image description tasks which have a mix of various hallucinations presented in Section 4, hallucinations in cognitive prompts are dominated by language hallucinations are predominant in. Thus, hallucinations for cognitive prompts cannot be effectively categorized into our mentioned types which is why we do not show results on the same.

But why is this analysis required here? To clarify, our analysis and categorization aim to investigate what is needed to mitigate hallucinations in cognitive prompts, thereby improving reasoning. Section 3 highlights that current mitigation techniques fail to address hallucinations in cognitive prompts. Section 4 categorizes hallucinations and suggests that existing methods, such as logit correction, are inherently tailored to mitigate language hallucinations. Finally, Section 5 demonstrates that (i) for image descriptions, hallucinated tokens are high-confidence single tokens in the logit space and can often be corrected by language-only models with the same prefix; (ii) in cognitive prompts, hallucinations arise from a top-$k$ probability space dominated by low-confidence tokens, making current methods ineffective (lines 70-75).

Building on these findings, VDGD addresses this challenge by grounding response generation to image descriptions, boosting the confidence of correct tokens and mitigating hallucinations more effectively.

Question 2 (About source of gains for VDGD and if VDGD goes through snowballing effect)

Ans. Thank you for the question. To address the first part: we agree that hallucinations in image descriptions can lead to hallucinated responses in cognitive questions. Lines 430-431 state that for all results in Table 1, VDGD uses image descriptions generated via the VCD method. VCD mitigates language hallucinations in image descriptions by correcting low-confidence tokens, which are often hallucinated. As shown in Table 3 (Appendix), using VDGD with vanilla decoding instead of VCD for image descriptions results in a performance drop. This highlights the importance of combining VDGD with effective language hallucination mitigation methods like VCD. VDGD is complementary to existing mitigation techniques. While methods like VCD effectively address language hallucinations in image descriptions, VDGD leverages these corrected descriptions to improve reasoning by grounding the generation process. It is important to note that VDGD is designed with a distinct motivation. Unlike methods that rely on logit corrections to counter low-confidence tokens, VDGD explores other aspects of the logit space, focusing on grounding through image descriptions. This approach allows VDGD to address reasoning tasks effectively, as demonstrated in our results.

Additionally, Figure 1 illustrates that while VCD addresses language hallucinations, it does not mitigate other hallucination types. VDGD complements such techniques by ensuring that corrected descriptions are used for reasoning, leading to overall performance improvements.

Question 3 (About computational analysis and applicability to beam search decoding)

Ans. Thank you for your suggestion. We have added below a comparison for latency and compute efficiency. We have also added this in the revised version of our paper in Appendix M.

Computational Analysis on LLaVA 1.5 with 48GB GPU (on MMMU):

We also show that VDGD is competitive with a small captioning model (which is not the LVLM) and the key is to generate an image caption that captures the visual elements. We hope you find this new results insightful. This shows that efficient captioning models can effectively reduce VDGD complexity.

Dear reviewer dSa3,

Thank you for taking the time to review our paper. We have addressed your concerns in our submitted response and provided a revised version of the paper. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!

Best,
Authors of Submission4948