VaPR - Vision-language Preference alignment for Reasoning

We thank the reviewer for their insightful feedback, and for recognizing the contribution of our (a) dataset, particularly the novel idea of generating rejected responses with stylistic and length similarity to accepted ones, (b) extensive experiments and reliable empirical evaluation. Below, we address your questions and concerns.

Q1. While I do admit that releasing new datasets is a huge contribution, the data generation process in this paper is reasonable yet not very novel.

While several prior works have explored synthetic preference data, our contribution is the dataset and data generation framework, which distinguishes itself in the following key ways:

• Task-Aware Perturbations: Unlike prior works (e.g., POVID, SIMA, CSR, RLAIF-V), our VaPR framework generates hard negative rejected responses with task-aware semantic errors, while explicitly preserving stylistic and length consistency. This prevents models from exploiting superficial cues during preference optimization (Section 4.3.1, Table 1, 2 in the main paper).
• Response Editing Approach: While existing approaches rely on VLMs (the same or a large oracle VLM) to generate and/or score sampled responses for preference datasets, VaPR instead leverages large, high-quality open-source SFT datasets to generate targeted negatives by editing ground-truth responses use LLMs. This leverages LLM’s’ semantic understanding of text, which is typically more reliable than image-text comprehension in VLMs [1].

These contributions make our pipeline particularly effective for post-SFT preference alignment and distinguish it from existing approaches. As we demonstrated in response to Reviewers iz1D and W6PJ, our process generalizes to open source models, where models trained on VaPR-OS (generated using Qwen3-32b [2]) achieve ~ 99% of performance of those trained VaPR (generated using GPT-4o). This means that many researchers could adapt our work to their own internal SFT datasets, without relying on closed-source APIs and potentially gain improvements beyond the ones made possible by our other main contribution: the VaPR dataset.

Q2. To be honest, I would like to see more examples of the dataset and check the source code. I am not fully sure whether this is possible according to the COLM rule.

We appreciate the reviewer’s interest in seeing more examples of the dataset and accessing the source code. As detailed in Appendix Section B.2, our illustrations show the generation steps for various tasks. For tasks requiring a penalty list - color and counting (to ensure diversity of perturbations) and captioning (to ensure diversity of perturbed dimensions like size, spatial relation, etc), we provide a PDF with additional examples highlighting how the penalty list is applied. We use the color task as an example. Additionally, we include source code for generating samples with and without a penalty list using GPT-4o, with color and spatial relation serving as representative tasks, respectively. Note that the shared scripts assume task-specific samples are pre-filtered (Section 3.2) and handle model input format post-processing separately.

OSF Link - Contains the PDF and code scripts: https://osf.io/pfz72/?view_only=57f0fe6e0b384bf4a6b165cb69aea753

Q3. Could you explain why the authors decided to use this Figure 5 and how this relates to so-called scaling law discussed in the existing work?

We thank the reviewer for raising this point. We acknowledge that Figure 5 does not use a logarithmic scale, and that the x-axis spacing may give the impression of regular intervals despite the non-linear values (3k, 10k, 30k). Our intention was not to claim that our data necessarily follows the same scaling laws as pretraining but rather to illustrate how performance gains vary with increased data size within our budget constraints. We agree that the term “scaling law” may be misleading in this context and will clarify the wording and the figure in the revised paper to avoid confusion. Our goal is to provide initial insights that can inform the community to further explore scaling opportunities if appropriate.

Q4. Please properly use \citep or \citet for citations.

Thank you for pointing this out, we will correct citation formatting to consistently use \citep and \citet in the revised paper.

References

• [1] Guan, Tianrui, et al. "Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
• [2] Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).

Thank you for providing the authors' feedback. I have read other reviews and rebuttals. The responses mostly addressed my concerns, so keep my original rating of 6.

We thank the reviewer for their thoughtful and constructive feedback. We appreciate your positive assessment of our work and are pleased that you found the (a) dataset, (b) framework, (c) model comparison and (d) metrics & analysis to be meaningful and impactful. Below, we address your specific concerns.

Q1. There is no ablation on using weaker or open-source editors, leaving it unclear whether the pipeline is usable in constrained settings.

We agree that understanding the pipeline’s generalizability beyond GPT-4o is important. To this end, we have applied our VaPR pipeline using the open-weight Qwen3-32b model [1] - focusing on the following task categories: color, object, spatial, counting, size, background, existence and referential VQA.

Table 1: Performance is compared across LLaVA-v1.5-Instruct, Qwen2VL-Instruct, Qwen2.5VL-Instruct, and DPO models, all preference finetuned on VaPR (10K subset created with GPT-4o) or VaPR-OS (8K samples generated by the Open-source model Qwen3-32b). Aside from the model used for generation, the framework and prompts are identical. All models share the same hyperparameters. Higher scores indicate better performance, with the top result for each benchmark shown in bold. In case two models get the top scores, both are bolded, otherwise top score is bolded and the second highest score(s) is $\underline{underlined}$

We preference finetuned LLaVA-v1.5-Instruct-7B, Qwen2VL-Instruct-2B and Qwen2.5-VL-Instruct-3B on VaPR-OS (VaPR Open source), where the difference between the models are two aspects: (a) is missing reasoning and captioning samples (1K each), reason shared below (b) uses an open-weight model for generating responses. Key findings include:

• Dataset Quality: VaPR-OS rejected responses exhibit similar hard-negative properties to those generated by GPT-4o, with an average token length difference of 6 (compared to 3 in VaPR) and a Levenshtein distance of 10 (vs. 6 in VaPR).

We thank the reviewer for their thoughtful and constructive feedback, as well as for recognizing (a) the strong benchmark performance of VaPR-trained models, (b) our comparative evaluation against alternative datasets, (c) the human validation that ensures dataset quality, and (d) our detailed analysis of generated outputs, particularly for “Yes/No” questions. Below, we address your specific concerns.

Q1. The proposed pipeline doesn't introduce any meaningful novelty with respect to the state-of-the-art.

While we acknowledge prior works such as mmE5 that utilize LLMs for generation, our approach focuses on post-SFT data generation and distinguishes itself in the following key ways:

• Distinction from mmE5: Unlike mmE5, which generates hard negatives primarily for training upstream embedding models and focuses on plausibility of content, our approach targets post-SFT preference alignment. Importantly, mmE5 does not emphasize length and stylistic consistency, which can be exploited by DPO (Section 4.3.1, Table 1 in the main paper).

• Distinction from Preference dataset generation methods:

  • Task-Aware Perturbations: Unlike prior works (e.g., POVID, SIMA, CSR, RLAIF-V), our VaPR framework generates hard negative rejected responses with task-aware semantic errors, while explicitly preserving stylistic and length consistency. This prevents models from exploiting superficial cues during preference optimization (Section 4.3.1, Table 1, 2 in the main paper).
  • Response Editing Approach: While existing approaches rely on VLMs (the same or a large oracle VLM) to generate and/or score sampled responses for preference datasets, VaPR instead leverages large, high-quality open-source SFT datasets to generate targeted negatives by editing ground-truth responses use LLMs. This leverages LLMs’ semantic understanding of text, which is typically more reliable than image-text comprehension in VLMs [1].

These innovations make our pipeline particularly effective for post-SFT preference alignment and set it apart from prior approaches.

We thank the reviewer for their thoughtful and constructive feedback, as well as for recognizing the value of (a) our dataset, (b) clear presentation (c) thorough empirical analysis. Below, we address your specific concern.

Q1. It is not clear whether the contribution of a dataset, while self-contained and significant, is novel enough to warrant an acceptance

While several prior works have explored synthetic preference data, our contribution is the dataset and data generation framework, which distinguishes itself in the following key ways:

• Task-Aware Perturbations Unlike prior works (e.g., POVID, SIMA, CSR, RLAIF-V), our VaPR framework generates hard negative rejected responses with task-aware semantic errors while explicitly preserving stylistic and length consistency. This prevents models from exploiting superficial cues during preference optimization (Section 4.3.1, Tables 1 and 2 in the main paper).

• Response Editing Approach: While existing approaches rely on VLMs (the same or a large oracle VLM) to generate and/or score sampled responses for preference datasets, VaPR instead leverages large, high-quality open-source SFT datasets to generate targeted negatives by editing ground-truth responses using LLMs. This leverages LLMs' semantic understanding of text, which is typically more reliable than image-text comprehension in VLMs [1].

These contributions make our pipeline particularly effective for post-SFT preference alignment and distinguish it from existing approaches. As we demonstrated in response to Reviewers iz1D and W6PJ, our process generalizes to open source models, where models trained on VaPR-OS (generated using Qwen3-32b [2]) achieve ~ 99% of performance of those trained VaPR (generated using GPT-4o). This means that many researchers could adapt our work to their own internal SFT datasets, without relying on closed-source APIs and potentially gain improvements beyond the ones made possible by our other main contribution: the VaPR dataset.

References

• [1] Guan, Tianrui, et al. "Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
• [2] Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).