Probing Visual Language Priors in VLMs

We greatly appreciate the detailed and insightful feedback. Below, we have carefully addressed the opportunities for improvement you highlighted.

---

Failure case analysis

We sincerely appreciate your suggestions and will include an in-depth failure case analysis section in revision. We observe several consistent failure patterns, as shown in the Figure. For deeper analysis, we additionally prompted the VLMs to output their reasoning before answering. Below, we summarize the failure patterns in the order corresponding to the images.

1. Shape recognition can fail in VLMs, causing them to revert to priors rather than accurately interpreting visual input.
2. Models sometimes struggle to count accurately. Instead of performing an actual count, they default to relying on learned priors to estimate quantities.
3. Models may refuse to accept visual information that contradicts their learned priors, whereas humans can comprehend hypothetical scenarios. For instance, the model recognizes the city as Beijing but rejects the correct answer because it expects the Eiffel Tower to be in Paris.
4. For images with creative concepts, the model may overly rely on its learned priors. As illustrated, a common prior is that microscopes are used to view organisms, leading the model to answer “microscope” rather than identifying the creatively depicted stapler.
5. For images with blended features, the model may rely mostly on text input while overlooking the visual cues. As illustrated, the VLM heavily depends on textual input leading to saxophone as the answer.

---

Fine-grained evaluation and LLaVA-OneVision performance

Thanks for the suggestions! Currently, we explicitly ask VLMs to produce a single-word response for evaluation, as it is (1) cost-efficient without calling LLM, (2) fast, and (3) reliable. For robustness, we implemented a comprehensive synonym-matching pipeline to validate synonymous responses accurately. We indeed penalize responses that include long sentences, as we believe instruction-following is a foundation capability of VLMs—thus, failure to comply warrants a penalty.

We acknowledge the value of your suggestion and have extended our evaluation system to include three categories: Correct, Wrong, and Invalid. “Invalid” refers to responses that do not meet the single-word requirement. Additionally, for “Invalid” outputs, we now provide an optional GPT-based evaluation that classifies them as either Invalid-Correct or Invalid-Wrong. The results is shown in Table. The new evaluation enables us to assess the model’s performance which generates longer sentences.

---

Regarding Winoground and HallusionBench benchmarks and unrealistic examples

For clarification, Appendix A.3 provides both quantitative and qualitative comparisons between our dataset and existing benchmarks such as WinoGround and HallusionBench, with key differences highlighted, ranging from high-level design principles to low-level formatting details.

Moreover, our human study shows that participants can identify unnatural elements like tigers or moons. This underscores the human ability to reason about unrealistic or novel scenarios—an essential goal for advancing VLMs—and positions our dataset as complementary to existing benchmarks.

---

Reference of [1]

Thanks for the suggestion, and we will discuss it [1] in the final revision. The key difference between ImageDPO and [1] is the training objective: while [1] follows the standard DPO setup by using preferred/dispreferred outputs for the same input, our approach fixes the output and varies the input conditions. This results in notable differences, like the distinct gradient behavior in Figure 13. Additionally, we introduce Proposition 1 and its theoretical proof in Appendix B, connecting our modified objective to the original DPO formulation.

[1] Self-Supervised Visual Preference Alignment

---

Connection between Image-DPO and the benchmark

Directly adjusting VLMs to improve specific abilities becomes challenging as the development of VLMs.. Our Image-DPO method addresses learned priors in a nuanced manner, where the proposed objective encourages the model to rely more heavily on the vision branch. Our objective drives a distinct shift in gradient directions when processing normal images $I_w$ compared to corrupted images $I_l$ (as illustrated in Figure 13). We appreciate your suggestion and will revise our wording to improve the connection.

---

Potential better approach

We explored a similar idea but faced a key limitation: existing image-editing models often introduce visible artifacts or unnatural transitions between edited regions and the original background, making it easier for VLMs to detect anomalies. Additionally, they frequently struggle to follow instructions accurately. As editing technology improves, we plan to revisit this approach for future dataset extensions.

We sincerely appreciate Reviewer j6Xe QTDS’s thoughtful and encouraging feedback. Due to constraints in time and computational resources, we have prioritized addressing your question and will incorporate the suggested additional improvements in future revisions.

---

Why is it necessary to fine-tune using additional synthetic images?

This is a very good point! We employ a comprehensive pipeline that uses multiple image-generation models to synthesize additional images from seed datasets. Our primary motivation is diversity: by incorporating a broader range of synthetic images, the VLM is exposed to more varied content and can, in turn, produce more diverse Question–Image–Answer (QIA) pairs. This process also implicitly leverages the knowledge locked in the existing image-generation models. Compared to relying solely on the original dataset, these additional QIA pairs enrich the training signal, helping the model learn more robust and generalizable representations.

To verify this point, we conducted an ablation study using the same dataset as in our ImageDPO experiments. Specifically, we directly generated QA pairs from images in the COCO, VG, and Text2VQA datasets without additional modifications or synthetic generation, i.e., no synthetic images. We then applied image corruptions to produce altered versions of these images and utilized these corrupted-image QA pairs for ImageDPO finetuning. The results, shown below, indicate notable performance drops. This outcome underscores that leveraging synthetic data, rather than relying solely on the original datasets, benefits the DPO training process.

---

The only weakness is the lack of comparison of their proposed Image-DPO to finetune more recent open VLMs (including LLaVA-NeXT, InstructBLIP, QwenVL and DeepSeekVL)

Thank you for this valuable suggestion! We agree that comparing Image-DPO with more recent VLMs (e.g., LLaVA-NeXT, InstructBLIP, QwenVL, DeepSeekVL) would provide additional insights. However, due to limited time and computational resources during the rebuttal period, we are unable to conduct these comparisons comprehensively. We appreciate this feedback and plan to include parts of the models in our future revisions. Nonetheless, we believe our current results on LLaVA-v1.5 and Cambrian sufficiently demonstrate Image-DPO’s effectiveness and hope they convey the potential of our approach.

We sincerely appreciate the positive review provided by Reviewer 1kEP, and are grateful that Reviewer 1kEP “likes the paper structure”, and “convinced with the methodology adopted in the paper”, believes “the ViLP dataset is well motivated”, and “the results are promising as well”. Below, we provide our detailed responses to your comments.

---

I have the following question: a) Since data generation for DPO training uses image-editing, I wonder if there is some quality control established to threshold what is an accepted corrupted image?

That's a good point! Our guiding principle is to eliminate human involvement in order to make the pipeline fully automatic and scalable. While the generated data may contain some noise, we believe the pipeline is robust enough to help improve VLM performance, which is supported by the improvements in the experiments. For corrupted images, we control the hyperparameters of image editing, such as the strength of Gaussian blur and pixelation, to ensure that the corrupted versions are noticeably worse than the original ones. In the case of semantic corruption, we calculate the number of pixels that are modified and discard cases where the changes affect less than 5% of the total pixels, as such minor alterations may be too subtle to detect.

---

The paper could do with a bit of re-writing and typos. For example, Line 1152.

Thank you for your thorough review! We have corrected the typo on line 1152 in the latest revision and will carefully proofread and refine the manuscript to enhance readability further.

We greatly appreciate the constructive feedback from Reviewer 4vBy. We seriously address your comments below.

---

Regarding the style of images.

We appreciate this valuable point and acknowledge that some generated images appear cartoon-like or synthetic, due to the image generation tools selected—among the best available at the time. However, since our objective is to benchmark VLMs using out-of-distribution QIAs that are not readily available online, we adopt image generation models to achieve this goal – a motivation appreciated by Reviewer j6Xe “Prior works focus on benchmarking real-world data…The proposed benchmark revisits and shifts this particular limitation”.

Regarding the cartoon-style concern, we would like to respectfully point out that current VLMs are commonly trained on datasets that include synthetic and cartoon-style images. For example, LAION, widely used in training VLMs, contains subsets focused on anime, cartoon, and aesthetics (filtered by community), reflecting the prevalence of synthetic imagery online. VLMs have shown robust performance on such data, and understanding synthetic visual representations remains an important capability, especially given humans’ ease in interpreting them.

To assess the impact of image style, we regenerated a subset of 45 QIA pairs using GPT-4o’s latest image generation model to enhance realism (Figure). We then measured changes in model correctness when using these realistic images, with negative values indicating performance degradation. The results show that increased realism slightly reduces performance in most cases for the “Score” metric, while most effects on “Prior” are zero. These ablation results suggest that incorporating more realistic images may increase task difficulty, highlighting an important direction for future research.

---

Regarding Proposition 1.

We would like to improve this point in the revision. For clarification, proposition 1 is the theoretical foundation of our Image DPO algorithm, showing that minimizing our proposed Image DPO loss corresponds to minimizing an upper bound on the RLHF loss. Unlike typical DPO modifications that adjust the output, our method fixes the output and alters the conditioning. Proposition 1 thus rigorously establishes the connection between our modified objective and the original DPO formulation.

---

Regarding the ambiguity and single-word evaluation.

We thank the reviewer for highlighting the inherent ambiguity in counterfactual visual reasoning. Although ambiguous scenarios occur, humans typically resolve them effectively after initial confusion, as evidenced by the strong human performance reported in our benchmark (Table 2), where evaluators had access to identical textual and visual inputs as the VLMs. Thus, the noted "ambiguity" intentionally examines whether VLMs can match human performance under controlled conditions.

Regarding single-word outputs, we highlight their evaluation advantages—including efficiency, speed, and reliability—when combined with our synonym and plural detection pipeline. In contrast, sentence-based evaluations relying on LLM inference incur API fees, computational costs, sensitivity to hyperparameters, and occasional unreliability due to incorrect reasoning.

We appreciate the reviewer's concerns about potential limitations of single-word answers. To explore this, we conducted ablation studies prompting GPT-4o in a "chain-of-thought" manner, instructing it to reason before giving a sentence-level final answer. We evaluated these answers against ground truths using GPT, comparing both reasoning steps and final sentences with original single-word responses.

Interestingly, allowing longer answers with more reasoning tokens did not yield performance benefits. As shown in (Figure), correct reasoning (second pic.) can still produce incorrect conclusions, and sometimes the reasoning process itself may be flawed (third pic.). Quantitatively, GPT-4o’s performance on "Prior" slightly improved from 91% to 92.67%, whereas on "Score" it decreased from 66.17% to 55.5%. Given that "Prior" aligns closely with language priors and "Score" represents out-of-distribution scenarios, we hypothesize that longer outputs may unintentionally increase reliance on priors during reasoning.