Systematic Reward Gap Optimization for Mitigating VLM Hallucinations

Q1: Pseudo code / flowchart of proposed TPR.

Thanks for your advice. We will add a pseudo code in revised manuscript. It will be like:

Algorithm 1 Topic-level Preference Rewriting.

Require: Reference model $\pi_\text{ref}$, Labeler model $\pi_\text{label}$, Source data $\mathcal{D}_\text{src}$ (Image $I$, Prompt $x$), Chosen strategy $\omega$ (e.g., greedy, curriculum).

Ensure: Preference data $\mathcal{D}_\text{pref}=\{(I,x,y_w,y_l)\}$.

1:  Initialize $\mathcal{D}_\text{pref}\leftarrow\varnothing$;

2:  for each ($I$, $x$) in $\mathcal{D}_\text{src}$ do

3:    Initialize initial responses $S_y\leftarrow\varnothing$, semantic units $S_u\leftarrow\varnothing$, topic clusters $S_C\leftarrow\varnothing$;

4:    for $i\leftarrow$ 1 to $M$ do

5:      $y_i\leftarrow$ Sample($\pi_\text{ref}$, $I$, $x$);

6:      Add $y_i$ to $S_y$;

7:      Add Decompose($\pi_\text{ref}$, $y_i$) to $S_u$;

8:    end for

9:    $S_C\leftarrow$ TopicCluster($\pi_\text{ref}$, $S_u$);

10:    for each cluster $C$ in $S_C$ do

11:      IntraTopicResample($\pi_\text{ref}$, $C$);

12:      Rank($\pi_\text{label}$, $C$);

13:    end for

14:    Initialize response template $y_k \leftarrow$ Randomly select from $S_y$, replacements $S_w\leftarrow\varnothing,S_l\leftarrow\varnothing$;

15:    for each unit $u \in C$ in $y_k$ do

16:      ($u_w$, $u_y$) $\leftarrow$ SelectAlternatives($\omega$, $C$);

17:      Add $u_w$ to $S_w$, Add $u_l$ to $S_l$;

18:    end for

19:    $y_w\leftarrow$ InContextRewrite($\pi_\text{ref}$, $y_k$, $S_w$);

20:    $y_l\leftarrow$ InContextRewrite($\pi_\text{ref}$, $y_k$, $S_l$);

21:    Add ($I$, $x$, $y_w$, $y_l$) to $\mathcal{D}_\text{pref}$;

22:  end for

23:  return $\mathcal{D}_\text{pref}$

Q2: More relevant references and baselines (e.g., HA-DPO, OPA-DPO, and mDPO).

We compare TPR with HA-DPO/OPA-DPO/mDPO below.

We will add these results and more references in revised version.

Q3: Experiments on larger base model (e.g., LLaVA-1.5-13B).

We provide a comparison between 7B and 13B model on 2k/4k/8k/16k/20k data. The results are presented below.

The results show 13B model demonstrates a substantial and consistent reduction in hallucinations across all data scales, confirming TPR is highly effective on larger base models. We will add this table to revised manuscript.

Q4: Details and Equation of DPO training loss.

Thanks for your advice. We will add equation of DPO loss and modify details about DPO in lines 128-130 in revised manuscript.

Q5: Capitalization of article titles in the references.

Thanks for your advice. We will standardize the format of article titles in references in revised manuscript.

Q6: About hallucination metric in Figure 1 (b).

In Figure 1 (b), we use an average of CHi/CHs in ObjHal as the metric to quantify "hallucination", because these two metrics simultaneously consider both coarse-grained and fine-grained hallucinatory contents.

Q7: Suppose a response y consists of two topics, $y_1$ and $y_2$, generated in sequence. The conditional probabilities $p(y_2|x,I,y_1)$ and $p(y_1|x,I,y_2)$ are generally not equivalent. How do the authors control the topic sequence in the proposed method to ensure consistency and coherence?

• To handle topic sequence and ensuring coherence/consistency, our method addresses it through a two-fold mechanism grounded in Selective Topic Rewriting pipeline. Specifically,

  • Our process does not generate topics in a new, arbitrary order. Instead, it is anchored by a response template, which is one of the initial, complete responses sampled from the reference model. This template naturally possesses a fixed topic sequence, e.g., $y=(y_1, y_2)$. During selective replacement, we perform an in-place modification of this template. Suppose we select alternative $y_1'$ for topic $Y_1$ and $y_2'$ for topic $Y_2$, then the new response $y'=(y_1',y_2')$ strictly preserves the original sequence. Therefore, the conditional dependencies remain consistent with template, i.e., the probability is always based on $p(y_1|x,I)$ and $p(y_2|x,I,y_1)$.
  • In additional to preserving topic order, we also ensure logical and linguistic coherence. After the replacement units are chosen, we employ the "In-Context Rewriting" step. Here, we instruct the reference model itself to seamlessly integrate the chosen semantic units into the template, with explicit instructions to preserve logical structure and linguistic style. This step ensures the resulting response is fluent and natural.

• For easy understanding, we have provided a simplified case study. Correct contents and hallucinations are highlight in different format respectively.

  • Chosen Template: In the image, there is a fork and a knife on the table (Topic-1), ready for use. There are two people in the scene, with one person located on the left side and the other person on the right side of the table (Topic 2).

  • Candidates for Topic-1: There is a fork and a knife on the table. / There are two forks and one knife on the table. / ...

    Candidates for Topic-2: There are two people in the scene, with one person located on the left side and the other person on the right side of the table. / Two people are located on the right side of the table. / ...

    * Note that other initial responses may include phrases like "the water bottle is on the right side." These topics will also be broken down and clustered. But since the chosen template does not include this topic, it will be excluded subsequently. And we omit it here for clarity.

  • Based on candidates on topic-1 and topic-2, we obtain the preferred and inferior response through rewriting the template with reference model.

    Preferred Response: In the image, there is a fork and a knife on the table, ready for use. There are two people in the scene, with both located on the right side of the table.

    Inferior Response: In the image, there are two forks and one knife on the table, ready for use. There are two people in the scene, with one person located on the left side and the other person on the right side of the table. Throughout the entire process, the topic sequence remains consistent due to the template, and logical/textual coherence is also ensured through in-context rewriting. We will add this clarification in revised manuscript.

Q8: Do "Intra-Topic Self-Resampling" and "In-Context Rewriting" utilize prompts $x_1$/$x_2$ that are different from prompts $x$ used during response generation? If so, $p(y|x,I)$ would likely differ from $p(y|x_1,I)$ and $p(y|x_2,I)$. Have the authors considered the impact of such differences on the proposed method?

• The prompt $x$ used for response generation is different from that used for self-resampling $x_1$ and in-context rewriting $x_2$. We provide the prompts in Appendix E (Table 5 and 6 in supp).
• Consequently, the conditional probabilities $p(y|x,I)$, $p(y|x_1,I)$ and $p(y|x_2,I)$ are not identical. However, we argue that the impact of these differences is acceptable.
  • Crucially, all three probabilities are sampled by the same reference model. While the prompts guide the model towards different sub-tasks, the generated text is always sampled from the same underlying model's capabilities and biases. The entire pipeline is therefore designed to explore the intrinsic success and failure modes of reference model itself.
  • Different from methods using external models like GPT-4V for rewriting, TPR only introduces a minimal and controlled distribution shift. As shown in Figure 4 (c), the hallucination patterns introduced by external rewriters can differ significantly from the model's own failure modes. Our in-house approach avoids this mismatch.

In summary, the impact of using different prompts is minimal and well-managed because the process remains self-contained within the reference model's distribution. We will add the discussion about this point in revised manuscript.

Q9: Limitations of complexity of proposed method.

We will add complexity of proposed method as one of the limitation revised manuscript. We are committed to improving the framework's efficiency in future by simplifying the pipeline, e.g., investigating the necessity of the clustering step, to further enhance its throughput and efficiency.

Q1: Technique novelty. Why TPR outperforms other fine-grained methods? Ablations on performance gains, from topic-level rewriting or a stronger rewriter model?

• Technique Novelty. We disagree the technical contribution is limited.
  • Topic-level selective rewriting to construct $y_w$ and $y_l$ with deliberate semantic differences is a new data curation paradigm. To our knowledge, this is the first work to actively sculpt the desired reward gap within preference data during data curation, rather than passively ranking VLM outputs or relying on black-box rewriters.
  • While TPR integrates established techniques like DPO and curriculum learning, we argue that their novel application for the purpose of systematically optimizing the reward gap configuration constitutes a significant contribution to mitigating VLM hallucinations. It's analogous to how major advancements are often achieved by innovatively combining mature methods. E.g., recent breakthroughs in reasoning models apply reinforcement learning to stimulate chain-of-thought processes. The value of such work lies not in inventing RL or CoT, but in their novel and effective application to unlock new capabilities, a principle that applies directly to our work.
  • Some work (e.g., β-DPO) in other domains highlight the importance of reward gap in DPO, but they primarily address it by tuning hyperparameters in loss function, leading to potential distribution shift and uncontrolled results (see response to Reviewer zQZ4 Q2 for details). In comparison, TPR provides a mechanism to control the reward gap during data curation, avoiding above issues.
• Ablations on Performance Gains. In Appendix D and Table 4 (supp), we provide experiments of using reference model (i.e., LLaVA-1.5-7B) itself as its labeler, named TPR-SL (self-labeling). It shows TPR-SL achieves substantial improvements over baseline LLaVA-1.5-7B model and remains highly competitive with other SOTA methods using much stronger labelers/rewriter like GPT-4V or LLaVA-NeXT-34B, demonstrating the performance gains are fundamentally driven by the design of TPR framework, and not solely by leveraging a more powerful labeler model.
• Comparison with Other Fine-grained Approaches. In addition to above ablation study on performance gains, another evidence that TPR outperforms other fine-grained approaches is data efficiency.
  • Compared to methods that do not rely on proprietary models (e.g., RLAIF-V), TPR constructs higher-quality responses through selective replacement. In contrast, while RLAIF-V improves the ranking process by aggregating the scores of all decomposed sub-responses, it still ranks existing, often flawed model outputs. This leads directly to the superior data efficiency we demonstrate in Figure 1 (b).
  • Compared to other methods that applies human annotators for fine-grained labeling, such as RLHF-V, although they can produce high-quality data, they are prohibitively expensive and not scalable. TPR provides an automated, scalable framework that surpasses the performance of human-annotated data at larger scales, as shown in our data efficiency analysis.

In summary, we believe TPR's technical contributions are significant, offering a new and effective perspective on data curation for preference alignment. This has also been recognized by other reviewers as a "novel" approach with "valuable contributions."

Q2: Comparison of computation costs between TPR and other baseline approaches.

We provide a detailed breakdown and comparison below.

• Computation Cost. The total computational cost for TPR generating full 20k dataset, run on 8 A100 GPUs, is detailed in the table below.

  Stage	TPR wo/ vLLM	TPR w/ vLLM
  Response Generation	10.27h	1.36h
  Decomposition	8.91h	3.57h
  Wh-question Convertion	7.95h	3.17h
  Self-resampling	7.48h	2.96h
  Topic Cluster	10.84h	6.71h
  Scoring & Ranking	23.43h	7.42h
  In-context Rewriting	2.17h	0.85h
  Total	71.05h	26.04h

  With vLLM acceleration, the cost is just 4.7 GPU-seconds per pair.

• Comparison with Other Baselines. We compare TPR with methods that curating data using open-sourced models (e.g., RLAIF-V) or proprietary models (e.g., POVID).

  • To ensure a fair comparison with RLAIF-V, we use official RLAIF-V code to measure its computation cost under the same conditions (without vllm acceleration, using 8 A100 GPUs). It takes approximately 66 hours to produce 22k preference data in RLAIF-V. While RLAIF-V has fewer steps, the most computationally expensive step is the scoring phase which uses a 34B labeler model. This is a common practice shared by methods using open-sourced models, including TPR and RLAIF-V. Additionally, iterative refinement in RLAIF-V introduces an additional cost: intermediate fine-tuning, i.e., retraining policy model after each data generation round to serve as new reference model for next round. The training process is generally more resource-intensive than data generation (inference) steps and cannot be accelerated by inference engines. In comparison, given the similar computational budget, TPR can consistently deliver superior performance across multiple benchmarks.
  • TPR is more cost-effective than methods relying on proprietary APIs. As detailed in Appendix C (Fig7 in supp), our analysis shows TPR achieves a comparable level of hallucination reduction at roughly one-third of estimated API cost required by a GPT-4V based rewriting approach.

We will add this discussion in revised manuscript.

Q3: Definition of "topic". Clarification why "topic" level was selected over other potential levels of granularity (e.g., sentence-level or entity-level).

• In TPR framework, the terminology "topic" is defined as a cluster of fine-grained semantic units that are determined as the same subject, based on textual consistency and visual correlation (see Section 3.2 and Appendix A).
• We choose "topic-level" over other granularities because it best describes the flexible, emergent semantic clusters in TPR.
  • Why Not "sentence-level". The granularity of "sentence-level" is too coarse. The decomposed units in TPR are often segments of sentences, not whole sentences.
  • Why Not "entity-level" or "attribute-level". These granularities are too rigid. The resulting clusters are broader than just single entities or attributes. They inclues entities, relations, related implications. Forcing our data into such a structure (only entity-level or attribute-level) would be limiting and would fail to capture the full spectrum of potential hallucinations.
  • Why Use "topic-level". The term "topic" effectively conveys the key insight of our method. We are identifying the different subjects being discussed and then performing resampling and rewriting within the context of that same subject. Using a term like "segment-level" would fail to capture the crucial clustering step that is central to our contribution.

We will clarify this definition of "topic" in revised manuscript.

Q4: Details about candidate sampling strategy.

We follow RLHF-V and RLAIF-V to sample initial candidate responses. Specifically, we instruct the reference model with the prompt provided in the Appendix E (Table 5 in supp). We set the parameters as: temperature=0.7,top_p=0.95,do_sample=True. Other parameters like top_k are left at their default values. Besides, as detailed in Appendix E, we generated 10 candidate responses for each image. After these responses are decomposed into semantic units, we resample one additional alternative per topic.

Q5: Data efficiency of FGAIF/HSA-DPO in Figure 1 (b).

The omission of FGAIF/HSA-DPO in Figure 1 (b) is because our local paper version was not updated to the latest during our experiments, so we cannot ascertain their precise data usage at that time. Given that they release their data in updated version, we have now conducted this comparison and will update our paper accordingly.

• Note that both FGAIF and HSA-DPO rely on GPT-4V to collect auxiliary datasets for essential intermediate training steps, which represent a significant cost. To provide a fair comparison, we argue it is crucial to consider the total data volume leveraged by each method, not just the size of the final preference dataset (Otherwise, TPR can also collect additional dataset to train a stronger labeler model).

  • FGAIF collects an auxiliary 3.5k dataset to train a reward model for its RL training stage, on top of its 14k preference data. The total effective data volume is about 17.5k.
  • HSA-DPO uses a detect-then-rewrite pipeline that first requires a 16k dataset to train a hallucination detection model. In addition to 8k preference data used in DPO training, the total effective data volume is about 24k.

• The results are present below. Note that we use average of CHs and CHi in ObjHal as the hallucination rate in Figure 1 (b), because these two metrics simultaneously consider both coarse-grained and fine-grained hallucination contents.

  Method	Data Scale	Hall (lower is better)
  TPR	16k	3.7
  TPR	20k	2.6
  RLAIF-V	22k	6.4
  FGAIF	17.5k	5.1
  HSA-DPO	24k	4.3

  As this data shows, both FGAIF and HSA-DPO perform similarly to RLAIF-V. Crucially, in the 16k-22k data range, TPR continues to demonstrate markedly superior data efficiency, achieving a significantly lower hallucination rate with less total data.

We will update Figure 1 (b) in the revised manuscript to include these new data points. We thank the reviewer for pointing out this omission.

Q6: Difficulty Score used in Curriculum Learning.

The "difficulty" in TPR refers to reward gap, it represents the learning difficulty.

Q7: Some Typos.

We will fix typos in revised version. Thanks for your advice.

Q1: Comparison between "utilization of a substantially larger and presumably more capable model for non-hallucinatory content generation or factual ascertion" and "knowledge distillation from a larger model to a smaller one." Ablation study on using a smaller, equaivalently capable model as $\pi_\text{label}$ instead of LLaVA-NeXT-34B.

• Ablation on Self-Labeling Provided in Appendix. For reviewer's concern about the absence of an ablation study on using reference model itself as labeler model, in fact, we provide this experiment and analysis in the supplementary material. As detailed in Appendix D and Table 4 (supp), we evaluate a variant named TPR-SL (self-labeling). The results show that TPR-SL achieves substantial improvements over the baseline LLaVA-1.5-7B model and remains highly competitive with other SOTA methods that use much stronger labelers like GPT-4o or LLaVA-NeXT-34B. This directly supports our claim that the significant performance gains are fundamentally driven by the architectural design of the TPR framework, and not solely by leveraging a more powerful labeler model.

• Using Capable Labeler. Our decision to use LLaVA-NeXT-34B in our main experiments is primarily to ensure a fair and direct comparison with contemporary methods. leveraging a more capable model or human annotators for supervision is a standard and established practice in the training-based vision hallucination migitation literature (RLHF/RLAIF). For example, rewriting-based methods like POVID using proprietary models like GPT-4V. RLHF-V relies on costly human annotator. And ranking-based baseline like RLAIF-V also employs LLaVA-NeXT-34B as its labeler. By using the same 34B labeler, we provide a fair comparison that isolates the benefits of our novel data curation strategy against existing work.

• Surpassing the Labeler Model after Alignment. Furthermore, the outcome of our alignment process goes beyond what is typically expected from knowledge distillation. We provide a comparison below.

  Method	ObjHal-CHs↓	ObjHal-CHi↓	MMHal-Score↑	MMHal-Hall↓	AMBER-acc↑	AMBER-F1↑	RefoMB-Trust↑	RefoMB-Win↑
  LLaVA-NeXT-34B	12.6	6.4	3.31	34.4	81.4	85.4	44.4	32.8
  TPR-CL-7B	3.4	1.8	3.06	30.2	82.7	87.8	61.1	32.3

  We can find that the student model (TPR-CL-7B) outperforms its teacher (LLaVA-NeXT-34B) on most of the hallucination benchmarks, demonstrating that TPR is not merely distilling knowledge.

Q1: Topic-level decomposition on more complex tasks, e.g., multi-step reasoning or commonsense knowledge.

There is currently no visual hallucination benchmarks for complex reasoning, so we use more complex general VLM benchmarks (i.e., MMMU/SEED-Bench) for evaluation, which to some extent can reflect hallucination of TPR and other methods exposed in complex reasoning scenarios. The results are present below.

We will add these results in revised manuscript.

Q2: Theoretical grounding for TPR.

We provide a mathematical grounding:

1. DPO Gradient

   The gradient of the DPO loss function, $\nabla\_\theta\mathcal{L}\_\mathrm{DPO}$, with respect to the policy model's parameter $\theta$ is given by:

   $\nabla\_\theta\mathcal{L}\_\mathrm{DPO}=-\beta\cdot\sigma(-\beta M_{\pi_\theta})[\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)].$

   where $\sigma$ is sigmoid function, $\beta$ is trade-off hyperparameter, $M_{\pi_\theta}$ is policy model's estimated reward gap, defined as:

   $M_{\pi_\theta}=\left(\log\frac{\pi_\theta(y_w|x)}{\pi_\mathrm{ref}(y_w|x)}-\log\frac{\pi_\theta(y_l|x)}{\pi_\mathrm{ref}(y_l|x)}\right)$

   The gradient consists of a direction vector and a magnitude scalar, $\sigma(-\beta M_{\pi_\theta})$. This scalar term, with a value between 0 and 1, directly controls the magnitude of the gradient update and is dependent on the model's current estimated reward gap, $M_{\pi_\theta}$.

2. Curating Data with a Desired True Reward Gap $\Delta r^\*$ In TPR

   Let $r^*(y,x)$ be true reward function that perfectly captures human preferences. For any pair ($y_w,y_l$) we curate, we define the true reward gap: $\Delta r^\*=r^\*(y\_w,x)-r^\*(y\_l,x)$. $\Delta r^\*$ is a fixed property of curated preference data pair that represents its difficulty. A high $\Delta r^\*$ is an "easy" problem, while a low $\Delta r^\*$ is a "hard" problem. Note that true gap is distinct from the model's estimated gap $M_{\pi_\theta}$ mentioned above. Using data with different true reward gaps has a direct impact on $M_{\pi_\theta}$. Let's compare two training strategies, assuming the same initial model:

   • Constant $\Delta r^\*$. If we train model on data with a constant true reward gap (e.g., only "easy," high-gap pairs), the model will quickly learn to make $M_{\pi_\theta}$ large for these types of problems. As $M_{\pi_\theta}$ grows, the gradient magnitude $\sigma(-\beta M_{\pi_\theta})$ shrinks towards zero, leading to the learning signal diminishes.

   • Gradually Lower True Gap. After model mastering easy problems (warm-up stage), it is presented with "hard" pairs. A lower true reward gap means more difficulty in distinguishing between $y_w$ and $y_l$. It leads to $M_{\pi_\theta}$ being close to zero for challenging pairs. Accordingly, an $M_{\pi_\theta}\approx0$ places the learning process in the steepest region of the loss curve, maximizing the gradient magnitude. This forces the model to learn the fine-grained details it was previously ignoring.

3. How About Changing $\beta$?

   The $\sigma(-\beta M_{\pi_\theta})$ term indicates that one can also influence gradient by changing $\beta$, e.g., dynamically adjusts $\beta$ based on $M_{\pi_\theta}$. However, if we want to achieve similar results that creates a strong learning signal for hard preference pairs by dynamically changing $\beta$, we would aggressively lower the value of $\beta$. Note that $\beta$ is originally designed to control the strength of KL penalty during DPO learning. A low $\beta$ could induce an unforeseen distribution shift. In contrast, TPR uses reference model for every creative step, ensuring the process remains grounded in the capabilities of the original reference model. We will add this discussion on revised manuscript.

Q3: Total computation cost.

We provide a detailed breakdown and comparison below.

• Computation Cost. The total computational cost for TPR generating full 20k dataset, run on 8 A100 GPUs, is detailed in the table below.

  Stage	TPR wo/ vLLM	TPR w/ vLLM
  Response Generation	10.27h	1.36h
  Decomposition	8.91h	3.57h
  Wh-question Convertion	7.95h	3.17h
  Self-resampling	7.48h	2.96h
  Topic Cluster	10.84h	6.71h
  Scoring & Ranking	23.43h	7.42h
  In-context Rewriting	2.17h	0.85h
  Total	71.05h	26.04h

  With vLLM acceleration, the cost is just 4.7 GPU-seconds per pair.

• Comparison with Other Baselines. We compare TPR with methods that curating data using open-sourced models (e.g., RLAIF-V) or proprietary models (e.g., POVID).

  • To ensure a fair comparison with RLAIF-V, we use official RLAIF-V code to measure its computation cost under the same conditions (without vllm acceleration, using 8 A100 GPUs). It takes approximately 66 hours to produce 22k preference data in RLAIF-V. While RLAIF-V has fewer steps, the most computationally expensive step is the scoring phase which uses a 34B labeler model. This is a common practice shared by methods using open-sourced models, including TPR and RLAIF-V. Additionally, iterative refinement in RLAIF-V introduces an additional cost: intermediate fine-tuning, i.e., retraining policy model after each data generation round to serve as new reference model for next round. The training process is generally more resource-intensive than data generation (inference) steps and cannot be accelerated by inference engines. In comparison, given the similar computational budget, TPR can consistently deliver superior performance across multiple benchmarks.
  • TPR is more cost-effective than methods relying on proprietary APIs. As detailed in Appendix C (Fig7 in supp), our analysis shows TPR achieves a comparable level of hallucination reduction at roughly one-third of estimated API cost required by a GPT-4V based rewriting approach.

We will add this discussion in revised manuscript.

Q4: Practical scalability for large-scale training regimes or for models serving many languages and domains. Feasibility when scaling to millions of pairs or much larger models.

While TPR involves a multi-step pipeline, it is practical for large-scale regimes because:

• TPR pipeline can be decoupled with parallelizable workflow: generation (response generation/decomposition/resampling), processing (scoring/ranking/selective replacement/rewriting) and training (curriculum learning). It allows us to execute data generation and processing steps in parallel across various languages, domains. The outputs can then be aggregated for a unified training phase, making the framework scalable for large-scale regimes.
• Additionally, TPR is designed for the post-training phase for alignment, not pre-training. This phase inherently operates on a much smaller data scale, typically in the order of hundreds of thousands to millions of preference pairs, rather than the trillions of tokens used for pre-training.

In summary, given TPR's parallelizable workflow and its application in less data-intensive alignment phase, we believe in TPR's practicality and scalability. Also, as detailed in our response the your question (Q3) on computation cost, the per-pair generation cost is manageable and can be significantly accelerated with inference engines like vLLM, further strengthening the case for its feasibility when scaling to millions of pairs or larger models. We are also committed to improving the framework's efficiency. Future work will explore simplifying the pipeline, for instance by investigating the necessity of clustering step, to further enhance its throughput and efficiency.

Q5: Potential negative societal impacts, such as misuse of generated "wrong" responses or environmental cost of large-scale data curation.

We agree a discussion of these aspects is essential for responsible research. In revised manuscript, we will add our response to these concerns in Appendix G. It will include:

• Misuse of Synthetic Data. The synthetic "wrong" responses generated by TPR, which are designed to contain hallucinations, could be misused for disinformation if taken out of context. To mitigate this risk, we commit to the following actions for any public release of our generated preference data:
  • The data will be released under a strict research-focused license, which prohibits commercial use and derivative works intended for malicious purposes.
  • The released dataset will be accompanied by a detailed README file. It will prominently state that the negative examples are synthetically generated for the purpose of model alignment, and are not suitable for any other application. We will explicitly warn against their use in any public informational systems.
• Environmental Cost: As discussed in our response to Q4, TPR is applied during the post-training alignment phase. While not zero, the environmental impact is therefore substantially lower than that of training a VLM from scratch. Our work on improving data efficiency and future plans to simplify the pipeline leads to a smaller environmental footprint.