Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning

Thank you for your valuable feedback. We respectfully address your concerns as follows:

1. Response to Weakness 1 & Question 1: Regarding correctness of distilled cold-start data

1. In our work, we ensure the correctness and reliability of the distilled CoT reward samples through two effective strategies:

   1. Multidimensional Evaluation and Scoring Thinking:
      We design a multidimensional analysis and scoring CoT process to structure reasoning. Specifically, we prompt GPT-4o to evaluate each image along several well-defined dimensions, and then aggregate the intermediate scores to arrive at a final preference decision. This step-by-step scoring mechanism enforces alignment between reasoning and outcome, helping ensure that the final decision is grounded in interpretable and logically coherent reasoning.

   2. Human-Aligned Ground Truth Supervision: We distill CoT reasoning samples from 10K human-preference-annotated image generation pairs, randomly sampled from four datasets: HPD, OIP, EvalMuse, and OpenAI-4o_t2i_human_preference. Each sample contains a prompt, a pair of generated images, and a human-annotated ground-truth (GT) label indicating the preferred image. We prompt GPT-4o to perform explicit CoT reasoning and make a comparative judgment. If GPT-4o's final decision matches the GT label, we retain the generated CoT reasoning and decision. This filtering process yields a final distilled dataset of approximately 5K high-quality CoT reward samples, which we use to cold-start our reward model training.

   These measures jointly help ensure that the distilled reasoning traces are not only logically consistent but also aligned with human preference signals.

2. Furthermore, in our pipeline, the cold-start CoT data serves primarily to teach the model the structure and format of CoT-style reasoning, rather than to enhance reward decision quality. The generalization and strengthening of the model’s reward discrimination capability are achieved through the subsequent stages of rejection sampling and GRPO, which operate on large-scale unified multimodal vision preference data and iteratively reinforce correct reasoning trajectories.

3. Our experimental results in the table also show that even when using distilled CoT data from weaker models (Qwen2.5VL-72b) instead of GPT-4o for cold-start, the model can still achieve comparable performance after subsequent reinforcement through rejection sampling and GRPO.

---

2. Response to Weakness 2 & Question 2: Regarding reliability of the model-generated answers in rejection sampling

We would like to clarify that our rejection sampling process does not involve any additional human annotation. Similar with cold-start data, we ensure the correctness and reliability of the rejection sampling data through two effective strategies . Specifically:

1. Multidimensional Evaluation and Scoring Thinking
   To ensure the reliability of the CoT process during sampling, we design a multidimensional analysis and scoring CoT process. Specifically, the model is guided to assess each candidate (image/video/response) along multiple well-defined dimensions, and then aggregate these intermediate scores to produce the final preference decision. This step-by-step structure enforces alignment between intermediate reasoning and the final outcome, helping ensure that correct decisions are derived from reliable and interpretable reasoning trajectories.

2. Human-Aligned Ground Truth Supervision:

   1. Generation Tasks :We conduct sampling on datasets such as HPD and VideoDPO, where each sample consists of a prompt, a pair of candidate images or videos, and human-labeled ground-truth (GT) preferences. The model is prompted to perform explicit CoT reasoning to compare the two candidates. If the final prediction matches the GT label, we consider the generated CoT reasoning and decision to be correct and apply rejection sampling to reinforce these correct reasoning patterns.

   2. Understanding Tasks: We apply a similar approach on datasets like LLaVA-Critic and ShareGPTVideo-DPO, where each sample includes a prompt and two candidate responses, along with labeled GT preferences. Again, if the model’s CoT-guided final prediction aligns with the GT, we regard the reasoning as correct and apply rejection sampling.

---

3. Response to Weakness 3 & Question 3: Regarding computational cost of GRPO fine-tuning and scalability

(1) GPU-hours and Memory Overhead of GRPO

During the GRPO fine-tuning phase, we trained on approximately 40K data points using ZeRO-3 distributed training across 64 H100 GPUs for 2 epochs, which took around 12 hours. Each GPU utilized approximately 60GB of memory.
The inference speed is around 3 seconds for image reward tasks and 7 seconds for video reward tasks.

In comparison, the UnifiedReward baseline was trained on 236K samples using ZeRO-2 across the same number of GPUs for 3 epochs, requiring around 10 hours, with 70GB memory usage per GPU. The inference speed is around 1 second for image reward tasks and 4 seconds for video reward tasks.

While GRPO introduces a slightly higher training overhead, this is justified by its ability to support long, multi-dimensional CoT reasoning, which leads to consistent and significant performance improvements across both image and video reward tasks.

(2) Scalability

1. Our framework is designed to scale effectively. In this work, we have curated and unified a wide range of open-source datasets covering both image and video generation/understanding tasks.
2. We believe that the model continues to benefit as more data becomes available. We are committed to continually integrating more newly released datasets to enhance the scalability and robustness of our approach.

---

4. Response to Weakness 4 & Question 4: Regarding analysis of downstream biases

We appreciate the reviewer’s thoughtful comment on fairness and bias. We would like to note the following:

1. Our method itself does not introduce new sources of societal bias. Any potential biases in the CoT traces would likely originate from the open-source datasets used for training.

2. In real-world tasks, we recommend incorporating human verification or filtering mechanisms when sensitive applications are involved. This can effectively mitigate unintended stereotypes that may appear in the data or model outputs.

3. As a research work, we did not have the resources or capacity to conduct a dedicated fairness audit. Nonetheless, we acknowledge the importance of fairness evaluation and view it as a valuable direction for future exploration.

Thank you for your feedback. We respectfully address your concerns as follows:

1. Response to Weaknesses 1&2: Regarding novelty

1. Our core innovation lies not in proposing a new technique, but in strategically addressing a fundamental and underexplored challenge in multimodal reward model-based RL: the unreliability of reward signals caused by direct response or shallow reasoning in existing vision reward models.

2. To the best of our knowledge, this is the first work to introduce a unified multimodal CoT reward model, capable of performing long CoT reward reasoning across both image and video generation and understanding tasks.

3. To address the challenge, we posit that incorporating explicit, multi-step CoT reasoning into the vision reward modeling process significantly improves the reliability and robustness of the reward signals. However, this is not a trivial application of existing techniques. Several unique challenges may arise in this context:

   1. Lack of multidimensional reward evaluation: Resultant models may tend to provide oversimplified or single-facet CoT reward reasoning, which is inadequate for complex visual reward tasks. For instance, in image generation, a comprehensive evaluation should consider not only semantic consistency, but also dimensions such as authenticity and aesthetics. Neglecting any of these aspects can lead to unreliable or biased reward signals.
   2. Inconsistency between reasoning and final decision: A widely observed issue in CoT-based models, where the reasoning appears implausible but the final prediction is correct. This mismatch directly undermines the reliability of the reward model, as it leads to inaccurate supervision signals during reinforcement learning.

4. To address these issues, we introduce several key innovations in CoT reward scoring strategy design:

   1. Multidimensional Evaluation Thinking:
      We design the model to perform multidimensional evaluations for each candidate (image, video, or textual response), based on the specific requirement of vision reward tasks. This enables the model to reason over diverse and task-relevant aspects, thereby enhancing the accuracy and reliability of the reward signal.

   2. Reliable Reasoning and Decision Alignment:
      The model is designed to explicitly assign scores to each evaluation dimension and then compute a final aggregated score across all dimensions. The candidate with the highest total score is selected as the preferred choice. This design enforces alignment between the intermediate reasoning process and the final decision, thereby mitigating a common issue in CoT-based models, i.e., the misalignment between reasoning and final prediction.

5. Based on these reward designs, we strategically adapt and integrate post-training techniques to elicit and reinforce CoT reasoning:

   1. Cold Start: We first construct a distilled dataset of 5k CoT reward reasoning samples for image generation, serving as a cold start to help the model learn CoT reasoning structures.
   2. Rejection Sampling: We then extend this capability to all vision tasks (image/video understanding and generation), by applying our model to large-scale unified multimodal reward datasets that contain only preference annotations (image/video/response X is better): we prompt our model to generate explicit CoT reasoning for each data. If the generated reasoning results in final prediction that aligns with the ground-truth preference, we apply rejection sampling to reinforce these correct reasoning patterns.
   3. GRPO: Conversely, when the reasoning leads to incorrect predictions, we apply GRPO, which encourages the model to explore diverse reasoning paths and optimize toward outputs that better match the verified ground-truth preferences.

6. It is important to highlight that solely relying on cold start and GRPO is insufficient for this task due to following challenges:

   • Scalability: Directly applying GRPO over a large scale of preference-only samples after cold start would incur high computational costs.
   • Low rollout quality: After cold start, the model’s performance is still unsatisfactory as evidenced in Tables 3 and 4, making it difficult to generate correct answers during RL rollout in hard samples.
   • Low RL benefit on known samples: For cases the model can already handle well after cold-start, GRPO provides limited learning gains while incurring high computation.

   Therefore, we introduce rejection sampling as an intermediate stage to:

   • Filter out easy samples that the model already gets right, reducing redundancy in GRPO, by consolidating high-quality correct reasoning patterns through these samples.
   • Allocate more RL effort to hard or ambiguous cases where exploration is needed through GRPO.

   We additionally include ablation results as shown in the table below. The results demonstrate that removing rejection sampling leads to a consistent drop in performance.

6. Empirically, our model shows substantial improvements across multiple vision reward benchmarks. Interestingly, we observe that once the model acquires long CoT reasoning ability, it can still make more accurate reward judgments even without explicitly outputting reasoning traces, suggesting it has internalized implicit logical reasoning mechanisms.

7. We believe our work leads to meaningful and impactful improvements in multimodal reward models and presents a valuable contribution to the community that should not be overlooked.

---

2. Response to Question 1: Regarding correctness of cold-start data

1. In our work, we ensure the correctness and reliability of the distilled CoT reward samples through two effective strategies:

   • Multidimensional Evaluation and Scoring Thinking:
     We design a multidimensional analysis and scoring CoT process to structure reasoning. Specifically, we prompt GPT-4o to evaluate each image along several well-defined dimensions, and then aggregate the intermediate scores to arrive at a final preference decision. This step-by-step scoring mechanism enforces alignment between reasoning and outcome, helping ensure that the final decision is grounded in interpretable and logically coherent reasoning.
   • Human-Aligned Ground Truth Supervision:
     We distill CoT reasoning samples from 10K human-preference-annotated image generation pairs, randomly sampled from four datasets: HPD, OIP, EvalMuse, and OpenAI-4o_t2i_human_preference. Each sample contains a prompt, a pair of generated images, and a human-annotated ground-truth (GT) label indicating the preferred image. We prompt GPT-4o to perform explicit CoT reasoning and make a comparative judgment. If GPT-4o's final decision matches the GT label, we retain the generated CoT reasoning and decision. This filtering process yields a final distilled dataset of approximately 5K high-quality CoT reward samples, which we use to cold-start our reward model training.

   These measures jointly help ensure that the distilled reasoning traces are not only logically consistent but also aligned with human preference signals.

2. Furthermore, in our pipeline, the cold-start CoT data serves primarily to teach the model the structure and format of CoT-style reasoning, rather than to enhance reward decision quality. The generalization and strengthening of the model’s reward discrimination capability are achieved through the subsequent stages of rejection sampling and GRPO, which operate on large-scale unified multimodal vision preference data and iteratively reinforce correct reasoning trajectories.

3. Our experimental results in the table also show that even when using distilled CoT data from weaker models (Qwen2.5VL-72b) instead of GPT-4o for cold-start, the model can still achieve comparable performance after subsequent reinforcement through rejection sampling and GRPO.

---

3. Response to Question 2: Regarding GPT-4o's performance

1. While GPT-4o is a powerful general-purpose multimodal model, it is not specifically optimized for reward modeling, especially in terms of aligning its reward reasoning process with ground-truth human preferences across diverse image and video tasks. In contrast, achieving reliable reward decisions requires more than general reasoning capability. In our work, this is accomplished through our carefully designed multidimensional evaluation and scoring reward mechanism, combined with a strategically structured three-stage training pipeline. Together, these components ensure the generalizability and reliability of our model across both image/video understanding and generation reward tasks.
2. Since GPT-4o is a closed-source model, we are unable to directly apply our CoT-based reward modeling training pipeline to it. Nonetheless, we believe that integrating our approach into GPT-4o would likely lead to significant improvements in its performance on this tasks.

We appreciate the reviewer’s valuable feedback. We respectfully address your concerns as follows:

1. Response to Weakness 1 & Question 2: Regarding correctness of model-generated answers in rejection sampling

In our work, we ensure the correctness and reliability of CoT reward samples in rejection sampling through two effective strategies:

1. Multidimensional CoT Scoring Mechanism: We design a multidimensional CoT analysis and scoring process to structure reasoning process. Specifically, the model is guided to assess each candidate (image/video/response) along multiple well-defined dimensions, and then aggregate these intermediate scores to produce the final preference decision. This step-by-step scoring mechanism enforces alignment between reasoning and outcome, helping ensure that the final decision is grounded in interpretable and logically coherent reasoning.

2. Human Preference-Aligned Ground Truth Supervision:

   1. For generation tasks, we conduct sampling on datasets such as HPD and VideoDPO, where each sample consists of a prompt, a pair of candidate images or videos, and human-labeled ground-truth (GT) preferences indicating which candidate is better. During sampling, the model is prompted to perform explicit CoT reasoning to compare the two candidates. If the final prediction of the model matches the GT label, we consider the generated CoT reasoning and decision to be correct and apply rejection sampling to reinforce these correct reasoning patterns.

   2. For understanding tasks, we apply a similar approach on datasets like LLaVA-Critic and ShareGPTVideo-DPO, where each sample includes a prompt and two candidate responses, along with labeled GT preferences. Again, if the model’s CoT-guided final prediction aligns with the GT, we regard the reasoning as correct and apply rejection sampling.

---

2. Response to Weakness 2: Regarding the performance improvement in Table 2

1. We would like to point out that while there is still room for improvement, our model has already achieved significant gains across most vision reward tasks, also acknowledged by Reviewer HVvo.. Specifically:

   • Image Generation Reward Task (GenAI-Bench):
     Our model achieves a +1.6% gain with CoT and +1.0% gain without CoT over the best baseline.

   • Video Generation Reward Task (GenAI-Bench):
     Our model outperforms the baseline by a significant margin of +6.1% with CoT and +5.4% without CoT, demonstrating its effectiveness in handling the complexity and temporal reasoning required for video evaluation.

   • Multimodal Understanding Task (VLRewardBench):
     On this challenging and widely adopted understanding benchmark, our model achieves a +6.3% improvement with CoT and +5.6% without CoT in Overall Accuracy, showing robust generalization beyond generation tasks.

---

3. Response to Weakness 3: Regarding the novelty

1. Our core innovation lies not in proposing a new technique, but in strategically addressing a fundamental and underexplored challenge in multimodal reward model-based RL: the unreliability of reward signals caused by direct response or shallow reasoning in existing vision reward models.

2. To the best of our knowledge, this is the first work to introduce a unified multimodal CoT reward model, capable of performing long CoT reward reasoning across both image and video generation and understanding tasks.

3. To address the challenge, we posit that incorporating explicit, multi-step CoT reasoning into the vision reward modeling process significantly improves the reliability and robustness of the reward signals. However, this is not a trivial application of existing techniques. Several unique challenges may arise in this context:

   1. Lack of multidimensional reward evaluation: Resultant models may tend to provide oversimplified or single-facet CoT reward reasoning, which is inadequate for complex visual reward tasks. For instance, in image generation, a comprehensive evaluation should consider not only semantic consistency, but also dimensions such as authenticity and aesthetics. Neglecting any of these aspects can lead to unreliable or biased reward signals.
   2. Inconsistency between reasoning and final decision: A widely observed issue in CoT-based models, where the reasoning appears implausible but the final prediction is correct. This mismatch directly undermines the reliability of the reward model, as it leads to inaccurate supervision signals during reinforcement learning.

4. To address these issues, we introduce several key innovations in CoT reward scoring strategy design:

   1. Multidimensional Evaluation Thinking:
      We design the model to perform multidimensional evaluations for each candidate (image, video, or textual response), based on the specific requirement of vision reward tasks. This enables the model to reason over diverse and task-relevant aspects, thereby enhancing the accuracy and reliability of the reward signal.

   2. Reliable Reasoning and Decision Alignment:
      The model is designed to explicitly assign scores to each evaluation dimension and then compute a final aggregated score across all dimensions. The candidate with the highest total score is selected as the preferred choice. This design enforces alignment between the intermediate reasoning process and the final decision, thereby mitigating a common issue in CoT-based models, i.e., the misalignment between reasoning and final prediction.

5. Based on these reward designs, we strategically adapt and integrate post-training techniques to elicit and reinforce CoT reasoning:

   1. Cold Start: We first construct a distilled dataset of 5k CoT reward reasoning samples for image generation, serving as a cold start to help the model learn CoT reasoning structures.
   2. Rejection Sampling: We then extend this capability to all vision tasks (image/video understanding and generation), by applying our model to large-scale unified multimodal reward datasets that contain only preference annotations (image/video/response X is better): we prompt our model to generate explicit CoT reasoning for each data. If the generated reasoning results in final prediction that aligns with the ground-truth preference, we apply rejection sampling to reinforce these correct reasoning patterns.
   3. GRPO: Conversely, when the reasoning leads to incorrect predictions, we apply GRPO, which encourages the model to explore diverse reasoning paths and optimize toward outputs that better match the verified preferences.

As shown in Tabs. 3 and 4, applying GRPO directly to the baseline without our CoT reward scoring strategy and the proposed three-stage training pipeline yields only limited performance gains. In contrast, our full method achieves substantially better results.

6. Empirically, our model shows substantial improvements across multiple vision reward benchmarks. Interestingly, we observe that once the model acquires long CoT reasoning ability, it can still make more accurate reward judgments even without explicitly outputting reasoning traces, suggesting it has internalized implicit logical reasoning mechanisms.

7. We believe our work leads to meaningful and impactful improvements in multimodal reward models and presents a valuable contribution to the community that should not be overlooked.

---

4. Response to Weakness 4 & Question 1: Regarding why the learning can be generalized to different tasks

1. In this work, our unified multimodal CoT reward model is designed to support both vision generation and understanding tasks, across both images and videos. Although the downstream tasks may differ, the input-output structure to the reward model is unified: in all cases, the model receives a combination of visual content (image or video) and caption or textual response, and outputs a reward judgment signal indicating the relative quality or preference between candidate outputs.

   Because of this consistent structure, the reward model learns to evaluate visual-text pairs in a task-agnostic manner, allowing it to generalize across different modalities and task types. This design enables us to train the reward model jointly on diverse datasets and apply it effectively across vision reward tasks.

2. However, equipping reward models with CoT reasoning using traditional training schemes like SFT poses a highly challenging problem due to the scarcity of large-scale multimodal CoT reward data.

3. We posit that VLMs already possess latent capabilities for multi-step reasoning. The key is designing a scalable and effective approach to elicit and reinforce such reasoning behavior across different vision reward tasks.

4. To this end, our pipeline strategically incorporates three stages:

   1. Cold Start: Learning CoT Reward Format
   2. Rejection Sampling: Unified CoT Reward Generalization Fine-tuning
   3. GRPO: Unified CoT Reward Reinforcement Fine-tuning

5. Through this three-stage strategy of learning, consolidation, and reinforcement, our model successfully transfers the CoT reasoning ability learned from a small number of image generation reward tasks to all visual reward tasks.

Thank you for your valuable feedback. We respectfully address your concerns as follows:

1. Response to Concern 1: Regarding novelty

1. Our core innovation lies not in proposing a new technique, but in strategically addressing a fundamental and underexplored challenge in multimodal reward model-based RL: the unreliability of reward signals caused by direct response or shallow reasoning in existing vision reward models.

2. To the best of our knowledge, this is the first work to introduce a unified multimodal CoT reward model, capable of performing long CoT reward reasoning across both image and video generation and understanding tasks.

3. To address the challenge, we posit that incorporating explicit, multi-step CoT reasoning into the vision reward modeling process significantly improves the reliability and robustness of the reward signals. However, this is not a trivial application of existing techniques. Several unique challenges may arise in this context:

   1. Lack of multidimensional reward evaluation: Resultant models may tend to provide oversimplified or single-facet CoT reward reasoning, which is inadequate for complex visual reward tasks. For instance, in image generation, a comprehensive evaluation should consider not only semantic consistency, but also dimensions such as authenticity and aesthetics. Neglecting any of these aspects can lead to unreliable or biased reward signals.
   2. Inconsistency between reasoning and final decision: A widely observed issue in CoT-based models, where the reasoning appears implausible but the final prediction is correct. This mismatch directly undermines the reliability of the reward model, as it leads to inaccurate supervision signals during reinforcement learning.

4. To address these issues, we introduce several key innovations in CoT reward scoring strategy design:

   1. Multidimensional Evaluation Thinking:
      We design the model to perform multidimensional evaluations for each candidate (image, video, or textual response), based on the specific requirement of vision reward tasks. This enables the model to reason over diverse and task-relevant aspects, thereby enhancing the accuracy and reliability of the reward signal.

   2. Reliable Reasoning and Decision Alignment:
      The model is designed to explicitly assign scores to each evaluation dimension and then compute a final aggregated score across all dimensions. The candidate with the highest total score is selected as the preferred choice. This design enforces alignment between the intermediate reasoning process and the final decision, thereby mitigating a common issue in CoT-based models, i.e., the misalignment between reasoning and final prediction.

5. Based on these reward designs, we strategically adapt and integrate post-training techniques to elicit and reinforce CoT reasoning:

   1. Cold Start: We first construct a distilled dataset of 5k CoT reward reasoning samples for image generation, serving as a cold start to help the model learn CoT reasoning structures.
   2. Rejection Sampling: We then extend this capability to all vision tasks (image/video understanding and generation), by applying our model to large-scale unified multimodal reward datasets that contain only preference annotations (image/video/response X is better): we prompt our model to generate explicit CoT reasoning for each data. If the generated reasoning results in final prediction that aligns with the ground-truth preference, we apply rejection sampling to reinforce these correct reasoning patterns.
   3. GRPO: Conversely, when the reasoning leads to incorrect predictions, we apply GRPO, which encourages the model to explore diverse reasoning paths and optimize toward outputs that better match the verified preferences.

As shown in Tables 3 and 4, applying GRPO directly to the baseline without our CoT reward scoring strategy and the proposed three-stage training pipeline yields only limited performance gains. In contrast, our full method achieves substantially better results.

6. Empirically, our model shows substantial improvements across multiple vision reward benchmarks. Interestingly, we observe that once the model acquires long CoT reasoning ability, it can still make more accurate reward judgments even without explicitly outputting reasoning traces, suggesting it has internalized implicit logical reasoning mechanisms.

7. We believe our work leads to meaningful and impactful improvements in enhancing the reliability of reward signals in multimodal reward models, and presents a valuable contribution to the community that should not be overlooked.

---

2. Response to Concern 2: Regarding CoT quality during GRPO

1. We agree that evaluating the quality of CoT reasoning is a critical aspect of ensuring the reliability of reward modeling. Indeed, rigorously assessing reasoning quality remains a broadly acknowledged challenge across CoT-based models. To date, there is no widely adopted and effective solution for systematically evaluating the correctness and robustness of intermediate reasoning steps during training, especially in complex multimodal tasks.

2. In our work, we mitigate this concern by designing a multidimensional analysis and scoring CoT process. Specifically, the model is guided to assess each candidate (image/video/response) along multiple well-defined dimensions, and then aggregate these intermediate scores to produce the final preference decision. This step-by-step structure enforces alignment between intermediate reasoning and the final outcome, helping ensure that correct decisions are derived from reliable and interpretable reasoning trajectories.

3. Thanks to this design, during GRPO training, we treat the correctness of the final prediction as a proxy for the quality of the CoT reasoning path. This allows us to supervise the CoT process implicitly but effectively, without requiring manual annotation of each reasoning step, which demands substantial human annotation effort.

4. In summary, our method implicitly favors reasoning chains that consistently lead to reward-aligned outcomes, which also have been discussed in Appendix A.

5. We fully acknowledge the importance of explicit post-hoc evaluation of CoT correctness, as you suggested. This is an excellent direction, and we plan to incorporate such evaluation in future iterations of our work.

---

3. Response to Concern 3: Regarding Tau metric

1. We would like to first clarify the interpretation of the Tau metrics reported in Table 2. In both the GenAI-Bench and VideoGen benchmarks, the Tau metric are specifically designed to evaluate model behavior in tie cases, where two images or videos are equally preferred.

2. In this work, our approach is explicitly focused on enhancing the model’s discriminative ability, that is, its capacity to distinguish between better and worse samples. While existing reward models are generally effective at identifying outputs with large quality differences, they often struggle to make fine-grained distinctions between two high-quality candidates. To address this gap, we introduce a multidimensional CoT reasoning process, aimed at improving the model's sensitivity to subtle differences in quality. As a result, we intentionally exclude tie cases from our training data, and our method is not optimized for these scenarios. This design choice is deliberate and aligns with practical objectives in vision-based reinforcement learning, where reward models are expected to make clear and decisive judgments between competing outputs in order to effectively guide policy optimization.

3. In contrast, baseline models were specially trained on datasets that include such tie scenarios, enabling them to perform well in such situations.

4. Although baseline models may achieve higher Tau scores, our method, both with and without CoT reasoning, consistently demonstrates superior performance in Diff. metrics where a quality difference exists between the two candidates.

5. We believe these results convincingly demonstrate the effectiveness of our proposed training approach, which is also acknowledged by Reviewer HVvo.

Hi,

Could you elaborate on your comments above? In particular, why do you think the work lacks technical contribution given the author's response, and what concrete additions do you think would improve the paper to be "above the bar"?