Generative RLHF-V: Learning Principles from Multi-modal Human Preference

Dear reviewer vM9w:

Thank you for your high recognition and affirmation of our work! We are greatly encouraged. We will address each of your comments point by point, and hope that our responses will strengthen your confidence in our work.

Q1: As noted, the use of RL means the approach is vulnerable to reward hacking.

Response 1: Clarification of Reward Hacking

You raise a very insightful question. Reward hacking is indeed an unavoidable challenge in reinforcement learning approaches [1, 2]. It essentially occurs when the policy model discovers vulnerabilities in the reward model and exploits them to obtain high rewards, while fundamentally contradicting human intentions.

Our proposed generative RLHF-V aims to provide a more robust reward for reinforcement learning fine-tuning of MLLMs, thereby improving model performance. This has been thoroughly demonstrated in our paper's experiments and has also received your recognition.

Beyond performance improvements, the contribution of generative RLHF-V is not merely an algorithm, but rather an empirical study on the application of multimodal generative reward models. Therefore, we candidly acknowledge that as a reinforcement learning method, generative RLHF-V is also susceptible to reward hacking issues. We do not wish to conceal this fact and are willing to expose the reward hacking problems that may exist in extreme scenarios with our method.

We hope this will inspire subsequent research, whether in more thorough mechanistic analysis or more robust algorithm design.

For example, during the rebuttal period, we implemented the four novel scoring methods of social technical: Spectral Ranking[3] and the three scoring methods from social choice techniques[4]: Borda count, Instant Runoff, and Ranked Pairs. We tested them in two scenarios: reward model scoring and reinforcement learning fine-tuning.

We selected the batch ranking task from MLLM-as-a-Judge, which requires the reward model to rank multiple responses, using the minimum edit distance from human expert results as the metric (lower is better). The results are shown in Table 1, where the left is the initial Qwen2.5-VL-Instruct-7B, and the right is the model trained with our proposed method:

Table 1: Performance Comparison of Different Scoring Methods on MLLM-as-a-Judge Batch Ranking Task

1. Social technical methods represented by Ranked Pairs can indeed improve the performance of grouped comparison methods, outperforming our implemented Average Score method in the paper.
2. Models fine-tuned with our method show the most significant improvements on both Average Score and Ranked Pairs methods.
3. While the Average Score method is not optimal, it still achieves relatively good performance.

Based on the above results, we find that social technical methods can indeed enhance the upper limit of grouped comparison methods, which is encouraging; meanwhile, our adopted Average Score method also achieves relatively good performance and can serve as an excellent baseline.

Finally, thank you again for your kind understanding and support.

[1]: Gao L, Schulman J, Hilton J. Scaling laws for reward model overoptimization[C]//International Conference on Machine Learning. PMLR, 2023: 10835-10866.

[2]: Ji J, Qiu T, Chen B, et al. Ai alignment: A comprehensive survey[J]. arXiv preprint arXiv:2310.19852, 2023.

[3]: Vigna S. Spectral ranking[J]. Network Science, 2016, 4(4): 433-445.

[4]: Conitzer V, Freedman R, Heitzig J, et al. Social choice should guide ai alignment in dealing with diverse human feedback[J]. arXiv preprint arXiv:2404.10271, 2024.

---

Q2: It was not clear to me where the model learns the explicit principles “accuracy” and “comprehensiveness”. Is it the case that this is implied only because of the chosen/rejected responses? If this is the case, might it be that these are not the actual preferences captured, but rather the model representation of the principles is some entanglement of accuracy and comprehensiveness (i.e., not interpretable to humans, but equivalently meaningful to the model)? Lines 161-164: If the point about entangled representations above is correct, for the case of GRM+RL might it be that performance is harmed when providing principles as a hint because the provided principles do not align with the model interpretation of the learned principles? If it was due to redundancy because the principles are already learned/accounted for (as suggested) then might one just expect the model to ignore that redundant input?

Response 2: Clarification of Principles Learning

Thank you very much for your thoughtful and constructive feedback on our paper. We are grateful for your positive assessment and for the insightful questions you raised, which have helped us to clarify and strengthen the core contributions of our work.

We would like to address the two excellent points you made regarding how the model learns principles and why providing them as hints can be detrimental.

1. On the Implicit Learning of Principles (e.g., "Accuracy" and "Comprehensiveness")

You asked whether principles like "accuracy" and "comprehensiveness" are learned implicitly from the preference data, and suggested that the model’s internal representation might be an "entanglement" of these concepts rather than a discrete, human-interpretable one.

Your intuition is entirely correct, and we thank you for articulating it so clearly. This is precisely the mechanism at play. Our model learns these principles implicitly. The RL objective (lines 103-104) simply rewards the Generative Reward Model (GRM) for correctly predicting the human-preferred response. To succeed at this task across a diverse dataset, the model must learn to identify the underlying features that consistently lead to a positive preference.

The principles it generates in its analysis (as seen in Figures 3 and 11) are its attempt to translate these complex, learned features into human-readable language. The model does not have a discrete "accuracy" module. Instead, as you insightfully suggest, its internal representation is a high-dimensional entanglement of features that, when decoded into text, are best described by words like "accuracy," "completeness," or "step-by-step guidance." These are emergent concepts, not pre-programmed ones.

We will revise the manuscript to make this point more explicit for the reader. We agree that your framing of this as an "entangled representation" is a more precise and powerful description of the process.

2. On Performance Degradation When Providing Principles to GRM+RL

You raised a very sharp point regarding our findings in lines 161-164 and RQ6, where providing principles as hints harmed the performance of the GRM+RL model. You hypothesized that this might be because the provided generic principles do not align with the model's internally learned, specific principles, rather than being merely redundant.

This is an excellent point, and we believe your hypothesis is a more accurate explanation than our initial suggestion of simple redundancy. The GRM+RL model, through its training, develops a sophisticated policy for generating highly specific and context-dependent principles for any given pair of responses. When we provide a static, generic principle from the dataset's annotation guidelines, we are not just giving it redundant information; we are imposing a constraint that conflicts with its learned optimal policy. This distracting constraint forces the model away from its flexible, tailored analysis and toward a more rigid one, which ultimately degrades its performance on out-of-distribution tasks.

We will update the discussion in lines 161-164 and the analysis in RQ6 to reflect this more nuanced and accurate explanation.

If you feel that our response in this round has not fully addressed your concerns, we are very willing to engage in further communication with you. Please feel free to ask questions at any time.

Once again, we thank you for your time and for providing such valuable and thought-provoking feedback. Your comments have given us a clearer lens through which to view our own results and have helped us significantly improve the paper.

Suggestions:

Figure 4 — in the figure where the scores are shown (Left, Right, Average) replace with (Left, Right, Mean)

It seems the symbol “r” has dual meanings — rewards and reasoning traces. Symbols should not be reused to avoid confusion.

Remove the period after D before the equation above Line 103.

Line 107: “to evaluates multiple'” > “to evaluate multiple”

Line 111: Remove the repeated sentence that begins “Each generated response ..”

Line 119 (and subsequently throughout the paper): Capitalize references to subsections, i.e., Subsection 4.2.

In text, for numbers less than ten, write them out in full. E.g., Line 182 “4” > “four”, etc.

Response 3: Thank you for your detailed and helpful suggestions

It's difficult to put into words the delight we felt when we saw your detailed feedback. We have made corresponding modifications point by point in our paper and believe they are of tremendous help. We would like to express our heartfelt gratitude once again.

Dear Reviewer vM9w,

Your support for us is reflected not only in the scores but also in a long series of detailed and insightful feedback. Words cannot adequately express our gratitude to you. As the discussion period draws to a close and our communication nears completion, we hope to summarize this pleasant discussion.

W1: As noted, the use of RL means the approach is vulnerable to reward hacking.

We clarified that the main contribution of our paper is an empirical study of GRMs for multi-modal RLHF. Therefore, while reward hacking behavior does indeed affect RL performance, it is also one of the findings of our empirical research.

W2: Clarification of Principles Learning

Your questions about the principles learning section reflect that there is still room for improvement in our writing. We have provided detailed feedback and commit to updating the corresponding explanations in the revision to further enhance readability.

W3: Presentations and Minor Issues

We are deeply moved to receive such detailed and thorough feedback from you, which makes our paper of higher quality.

We are greatly honored to see your recognition of our rebuttal content, maintaining your strong support for us, and even evaluating our work as a valuable contribution. Thank you once again for your recognition and support!

Dear reviewer NwUp:

We are grateful for your recognition of the clarity of our writing, method innovation, and the breadth of our experiments. You must have carefully read our paper to propose such detailed and insightful feedback, and we will address each point. If you believe we have resolved your concerns, we sincerely and kindly hope that you will give us higher support. Also, if you have additional questions about our responses or other aspects you would like to discuss, we also welcome further communication with you.

W1: The authors propose using group comparisons based on model-generated responses to train reinforcement learning. Have the authors considered exploring alternative RL algorithms, such as GRPO, for training the reward models?

---

Response 1: Exploration of Alternative RL Algorithms

You are certainly experienced in RL experiment research! In fact, during the early research stage, we simultaneously attempted GRPO and PPO RL algorithms to train the generative reward model, but GRPO performed slightly better in reward modeling, and because GRPO did not need to train an additional critic model, it consumed less computational resources, so we finally chose GRPO. The experimental results are as shown in Table 1 and Table 2.

Table 1: Performance Comparison of GRM+GRPO vs GRM+PPO on Distribution-in Test

Table 2: Performance Comparison of GRM+GRPO vs GRM+PPO on Distribution-out Test

We believe the reason for this might be that the reward for the preference learning task is relatively "simple", and the model only needs to provide a higher score for the human-preferred response to obtain the reward. Therefore, this task does not require a critic model to provide additional information, and averaging the trajectory reward values within the group can achieve good results.

---

W2: It is unclear which specific algorithm the authors use to train the MLLM based on the proposed reward models. Do the authors employ PPO, GRPO, or another RL algorithm for this purpose?

Response 2: Clarification of RL Algorithm

Thank you for pointing this out! We use GRPO algorithm to train the multi-modal generative reward model. We will supplement this in section 4.1 Experiment Setup. We believe this will make the article clearer.

---

W3: The authors mention the emergence of self-praise behavior in the experimental results, which appears similar to reasoning patterns induced by reinforcement learning. Is this observation specific to the Qwen-based model, or can it be generalized to other multi-modal large language models (MLLMs) as well?

Response 3: Exploration of "Self-Praise" Behavior

Thank you for your attention to our exploration of the "self-praise" behavior. Extending this conclusion to other models will undoubtedly further strengthen our contribution. During the rebuttal period, we conducted additional experiments on LLaVA-1.5-7B, with results as follows:

We found that LLaVA-1.5-7B exhibits a different reward hacking behavior from the Qwen-VL series models under over-training conditions. Compared to the behavior mentioned in our main text where models add extra text to praise their own responses, LLaVA tends to use overly complimentary vocabulary to describe the image itself. Here is an example:

...[Complete Response]...The composite image of all the elements of the branded outfit, including the custom-made Revo-Knit pink dress, the matching headband and necklace, the Wilson head and racket, the Delta Duck logo on the left side, the W logo on the right side, the multiple accessories, the white wristband,socks and bracelets, all contribute to the overall powerful and memorable delivery of the brand's message, making it an emphatic signature for both the athlete and the endorsed brand. It effectively conveys the essence of the Wilson brand to the audience, creating a lasting impression of the brand's values, commitment, and the quality of their products, ultimately serving as a powerful promotional tool.

We found that this behavior cannot hack higher performance in MLLM-as-a-Judge evaluations, and the actual performance of over-trained models is not good, as shown in Table 3.

Table 3: Performance Comparison of Over-trained Models on LLaVA-Wild and MIA-Bench

We believe this may be because the capabilities of the LLaVA-1.5-7B model are limited and not sufficient to explore reward hacking behaviors similar to those observed in the Qwen-VL series. We will add discussions on reward hacking behaviors in models with different architectures to the future work section, and acknowledge in the paper that this addition is thanks to the kind reminder from reviewer NwUp. We sincerely appreciate your insightful suggestions once again.

Dear Reviewer NwUp,

Thank you once again for your continued support and detailed suggestions! As the rebuttal period draws to a close, it appears our communication with you is also nearing completion. Please allow us to provide a final summary to conclude this pleasant exchange.

We thank you again for your recognition of our writing, experiments, and methodological novelty, and we are even more grateful for your suggestions regarding supplementary details of our methodology.

W1: Exploration of Alternative RL Algorithms

We conducted an empirical comparative analysis of RL algorithms (PPO, GRPO) used for GRM training, and explained our rationale for choosing GRPO as the RL algorithm.

W2: Clarification of RL Algorithm

We sincerely acknowledge the details we omitted in the paper and commit to making corresponding changes in the revision.

W3: Exploration of "Self-Praise" Behavior

We are delighted that you are interested in "Self-Praise" behavior and other reward hacking behaviors. During the rebuttal period, we explored new reward hacking behaviors on the LLaVA-1.5-7B model.

Your response informing us that we have clarified your concerns and that you are willing to maintain your support fills us with excitement. Thank you once again for your support and feedback!

Dear reviewer ecEr:

We appreciate the reviewers' recognition of the effectiveness and novelty of our method. In response to the reviewers' concerns about robustness and applicability, we have supplemented our results with experiments on LLaVA-1.5-7B and provided detailed analysis. Regarding the novelty of our contributions, we have also provided comprehensive discussions.

We hope these responses address your concerns. And we sincerely and kindly hope that you can provide greater support. If you have additional questions about our responses or other aspects you would like to discuss, we also welcome further communication with you.

W1: The main experiments are conducted only on the Qwen-VL family, so it remains uncertain how well the pipeline transfers to models with different architectures or modalities. Including results on several diverse model series would give a clearer picture of the method's robustness and broader applicability.

Response 1: Additional Experiments on LLaVA-1.5-7B

You are certainly a rigorous and responsible reviewer! Our experiments cover two versions of Qwen-VL (2.0 and 2.5), two modes (pretrained and instruction-following models), and three sizes (2B, 3B, and 7B).

Nevertheless, we sincerely agree with your suggestion and have supplemented experiments on LLaVA-1.5-7B during the rebuttal period, reproducing our key experimental RQs from our paper, as detailed below:

RQ1 in paper: Does the GRM+RL facilitate more generalizable principle learning from preferences?

Table 1: Principle Learning Performance Comparison on LLaVA-1.5-7B Across Different Training Methods

The results show that on LLaVA-1.5-7B, GRM + RL still demonstrates the highest scoring accuracy and optimal principle learning capability across three out-of-distribution datasets, consistent with the experimental conclusions in the paper.

It is worth noting that despite LLaVA-1.5-7B's limited model capacity, our method still achieved significant performance improvements. This further expands the applicability scope of our approach.

RQ3 in paper: Are GRM+RL and grouped comparison competitive methods for multi-modal RLHF?

Table 2: Multi-Modal RLHF Performance Comparison on LLaVA-1.5-7B Using Different Training Methods and Reward Models

The results show that on five mainstream benchmarks, the method combining GRM + RL with grouped comparison achieves optimal results, consistent with the experimental conclusions in the paper.

RQ5 in paper: What is the reward hacking behaviors of an over-trained Generative RLHF-V model?

We found that LLaVA-1.5-7B exhibits a different reward hacking behavior from the Qwen-VL series models under over-training conditions. Compared to the behavior mentioned in our main text where models add extra text to praise their own responses, LLaVA tends to use overly complimentary vocabulary to describe the image itself. Here is an example:

...[Complete Response]...The composite image of all the elements of the branded outfit, including the custom-made Revo-Knit pink dress, the matching headband and necklace, the Wilson head and racket, the Delta Duck logo on the left side, the W logo on the right side, the multiple accessories, the white wristband,socks and bracelets, all contribute to the overall powerful and memorable delivery of the brand's message, making it an emphatic signature for both the athlete and the endorsed brand. It effectively conveys the essence of the Wilson brand to the audience, creating a lasting impression of the brand's values, commitment, and the quality of their products, ultimately serving as a powerful promotional tool.

We found that this behavior cannot hack higher performance in MLLM-as-a-Judge evaluations, and the actual performance of over-trained models is not good, as shown in Table 3.

Table 3: Performance Comparison of Over-trained Models on LLaVA-Wild and MIA-Bench

We believe this may be because the capabilities of the LLaVA-1.5-7B model are limited and not sufficient to explore reward hacking behaviors similar to those observed in the Qwen-VL series. We will add discussions on reward hacking behaviors in models with different architectures to the future work section, and acknowledge in the paper that this addition is thanks to the kind reminder from reviewer ecEr. We sincerely appreciate your insightful suggestions once again.

---

W2: The core ideas are closely related to prior work in the text-only NLP literature, and thus the novelty of the contribution is somewhat limited (though I acknowledge that the paper thoughtfully extends generative reward modeling and grouped comparison to the multi-modal setting and provides useful insights).

Response 2: Clarification of Novelty

Thank you for your recognition and questioning of our work. As you pointed out, Generative RLHF-V is an extension based on existing methods in the text-only NLP field: self-principled-critique-finetuning (SPCT). However, we would like to further clarify that this work is not simply extending SPCT from text modality to multimodal modality, but rather building a complete pipeline around the multimodal human principles learning problem, including reward modeling and grouped comparison (as you mentioned in the strengths, this is also a thoughtful methodology that provides useful insights). Within this pipeline, we have also conducted more in-depth and unique analysis of Generative RLHF-V's learning capabilities for multi-modal principles.

1. We found that compared to SPCT's approach of providing reference principles, directly letting the model actively infer the principles behind human preferences can achieve stronger and more accurate principle learning effects. We demonstrate this conclusion in Figure 5 and Table 3.

2. We found that in multimodal scenarios, integrating grouped comparison and GRM+RL near linearly enhances multi-modal RLHF performance across various settings of candidate responses. We demonstrate this conclusion in Figure 7.

3. We found that over-trained GRM+RL models exhibit self-praising reward hacking behavior on Qwen-VL series models, while during the rebuttal period we discovered that LLaVA-1.5-7B exhibits reward hacking behavior of overly praising the image itself. As shown in Figure 8, these analyses provide unique insights into the use of generative reward modeling in multimodal scenarios.

In summary, we believe that the contribution of this work goes far beyond algorithmic improvements, but rather represents the first systematic empirical study of the entire pipeline from reward modeling, grouped comparison, to various ablation analyses on the multimodal principles learning problem. We again thank the reviewer for their tolerant attitude and positive rating of our work. Your attention has prompted us to more clearly summarize and express the contributions of our work, and we hope the above supplementary explanations can effectively address your concerns.

Dear Reviewer ecEr,

Thank you once again for your strong support and insightful feedback! As the rebuttal deadline approaches, it appears our communication is nearly complete. We hope to provide a final summary to conclude this pleasant discussion.

We greatly appreciate the your recognition of our method's effectiveness and the comprehensiveness of our experiments. In response to the your concerns about applicability and novelty, we have supplemented experiments on LLaVA-1.5-7B during the rebuttal period and provided detailed clarifications regarding the novelty of our work.

W1 (Generalizability of Model Architecture): We have supplemented key experiments on LLaVA-1.5-7B. The results demonstrate that our GRM+RL approach still exhibits the highest scoring accuracy and optimal principle learning capability on three out-of-distribution datasets, consistent with the experimental conclusions in the paper.

W2 (Novelty Limitations): We clarify that this work is not simply extending SPCT from text modality to multimodal, but rather constructs a complete pipeline around the multimodal human principle learning problem. Our contributions include:

1. Discovering that having models actively infer principles behind human preferences achieves stronger learning effects than providing reference principles
2. In multimodal scenarios, the integration of grouped comparison and GRM+RL approximately linearly improves multimodal RLHF performance
3. Finding that over-trained models exhibit unique reward hacking behavior

We are pleased to see that you have expressed recognition of our rebuttal and correspondingly updated the score. We commit to updating the corresponding changes in the revision, confident that this will further enhance the quality of the paper. Thank you once again for your thorough review and the valuable energy and time you have invested.

Dear reviewer mrSo:

Thank you for your invaluable and insightful feedback. Your thoughtful comments and constructive suggestions have been instrumental in helping us enhance the clarity and quality of our paper. We sincerely appreciate the time and effort you dedicated to reviewing our work. We have carefully considered all your points and have revised the manuscript accordingly.

We have provided a detailed point-by-point response to your questions. If you feel we have addressed your concerns, we sincerely and kindly hope you will give us higher support.

---

W1: Some parts of the paper was hard to follow. A key contribution, grouped comparison, is only explained halfway through. A short description like "we generatr multiple candidates at the same time, compare each pair of candidates to each other, and use all available comparisons to calculate average scores for all the candidates" could've been mentioned earlier on, in the introduction. Some motivational statements in the introduction seemed out-of-place to me. For example, claiming "Pair-wise comparison is blocking multi-modal principles from generalizing to learnable rewards" seemed imprecise, as the authors still use pairwise comparisons, just multiple ones simultaneously, rather than list-based or rank-based methods which would be a more stark contrast to pairwise comparisons. Another example of a writing issue, in RQ6 of section 4, the first sentence ("SPCT mentions that providing principles as a reference within the user prompt.") is not a proper sentence. I suggest revising the paper to proofread other possible errors that can lead to lack of clarity.

Response 1: Clarification of "Grouped Comparison"

We completely agree with your excellent suggestion that a clearer, earlier explanation of "grouped comparison" would significantly improve the reader's understanding of our core contributions. We have revised the introduction to include a concise description of this mechanism, much like the one you proposed. This ensures that the concept is clearly defined from the outset, providing a better foundation for the sections that follow.

Also, we thank you for pointing out the imprecision in some of our motivational statements. Upon reflection, we concur that the claim "Pair-wise comparison is blocking multi-modal principles from generalizing to learnable rewards" could be misconstrued. Our method indeed leverages and extends pairwise comparisons in a novel way rather than abandoning them. We have refined this statement in the introduction to more accurately articulate that our contribution overcomes the limitations of traditional single pairwise comparisons by aggregating information from multiple simultaneous comparisons within a group.

---

W2 & Q1: For the grouped comparison part, I mentioned how the authors still use pairwise comparisons. There are more structured approaches to predicting scores from a pairwise comparison matrix as such as spectral ranking (which works well with cardinal scores). (ref: https://arxiv.org/pdf/0912.0238). It would be interesting to see if calculating rewards in that manner shows any visible improvement. Another possible venue of work to explore would be the use of social choice technique in AI alignment (which suggests using more than pairwise preferences). See here: https://arxiv.org/pdf/2404.10271

Response 2: Additional Experiments For Structured Prediction

We are delighted to see your insightful comments. This series of research will undoubtedly become a very valuable improvement direction for generative RLHF-V in the future, and demonstrates its reference value for other fields.

During the rebuttal period, we implemented the four novel scoring methods you mentioned: Spectral Ranking and the three scoring methods from social choice techniques: Borda count, Instant Runoff, and Ranked Pairs. We tested them in two scenarios: reward model scoring and reinforcement learning fine-tuning.

• Experiment (1/2): Reward Model Scoring

We selected the batch ranking task from MLLM-as-a-Judge, which requires the reward model to rank multiple responses, using the minimum edit distance from human expert results as the metric (lower is better). The results are shown in Table 1, where the left is the initial Qwen2.5-VL-Instruct-7B, and the right is the model trained with our proposed method:

Table 1: Performance Comparison of Different Scoring Methods on MLLM-as-a-Judge Batch Ranking Task

1. Social technical methods represented by Ranked Pairs can indeed improve the performance of grouped comparison methods, outperforming our implemented Average Score method in the paper.
2. Models fine-tuned with our method show the most significant improvements on both Average Score and Ranked Pairs methods.
3. While the Average Score method is not optimal, it still achieves relatively good performance.

Based on the above results, we find that social technical methods can indeed enhance the upper limit of grouped comparison methods, which is encouraging; meanwhile, our adopted Average Score method also achieves relatively good performance and can serve as an excellent baseline.

• Experiment (2/2): Reinforcement Learning Fine-tuning

Table 2: Multi-Modal Benchmark Performance of Different Scoring Methods in Reinforcement Learning Fine-tuning

We find that in the context of reinforcement learning fine-tuning, the improvements from the above social technical methods are not significant. We believe this is because the Score Average method can provide more fine-grained scoring for various answers, while methods like Ranked Pairs can only provide coarse-grained rankings, making it difficult to provide sufficiently fine-grained supervisory signals for reinforcement learning.

Still, we believe that the social technical methods have the potential to enhance the performance of grouped comparison methods, and we hope our empirical study can inspire more research in this direction or other applications. We will include the above experiments and analysis in the appendix: potential research directions section, denoting that this is inspired by the reviewer mrSo's suggestion. We expect researchers in fields such as social choice theory to further explore the applications of these methods in alignment scenarios.

---

W3 & Q2: The authors mention how there's almost a linear increase in some benchmark scores as number of candidate responses in the grouped comparison method. The authors don't really mention the computational tradeoffs here. Or how long the linear growth sustains.

Response 3: Clarification of Computational Tradeoffs

Thank you for your thorough and rigorous feedback. We present the computational overhead of the grouped comparison method. The data in Table 3 represents the computation time for a single step, including rollout, scoring, and parameter updates, with batch size and other hyperparameters consistent with those in the appendix. The experiments were conducted on 16×H800 GPUs.

Table 3: Computational Overhead of Grouped Comparison Method with Different Numbers of Candidate Responses

We sincerely acknowledge that as the number of candidate responses increases, the computational overhead of the grouped comparison method also increases significantly. We commit to updating these results in the limitations section of our paper.

We are willing to engage in further discussion on this issue. Specifically, we believe that the primary source of computational overhead comes from the $A_n^2$ comparisons performed on $n$ candidate responses, which leads to a significant increase in computation time as $n$ grows. However, it may not be necessary to perform pairwise comparisons on all responses to achieve results comparable to, or even better than, those obtained through $A_n^2$ comparisons. We believe that methods from social technical and other fields may offer valuable insights to help us reduce the number of comparisons and computational overhead. We will also discuss this in the appendix: potential research directions, denoting that this is inspired by the reviewer mrSo's suggestion. Thank you again for your insightful comments.

Dear reviewer mrSo,

Thank you for your timely and constructive feedback! We are very pleased that our previous rebuttal with limited character count addressed some of your concerns. We are very willing to provide more detailed responses to the new questions you have raised.

It's nice to see that the other methods for using rank data are showing some promise.

For experiment 1, can you explain in more detail what the two columns are indicating? Is this equivalent to Table 1 in the original manuscript?

We will add an introduction to all three tasks in the MLLM-as-a-Judge benchmark in Appendix Section 6 Experiment Details of the paper, with the following specific content:

MLLM-as-a-Judge benchmark introduces three distinct tasks to evaluate the judgment capabilities of MLLMs.

• Scoring Evaluation, requires the MLLM to rate a single response on a scale of 1 to 5 based on criteria like relevance, accuracy, and creativity.
• Pair Comparison, involves the model comparing two responses to the same prompt and determining which is superior, with the option of declaring a tie.
• Batch Ranking, tasks the MLLM to arrange multiple responses in descending order of quality based on a given instruction, without the possibility of a tie.

Our experiments on MLLM-as-a-Judge in the main paper primarily include its Scoring Evaluation (Table 1 and Figure 6 in our main paper) and Pair Comparison (Figure 5, (c). component) components.

To explore the potential of structured prediction methods, we supplemented experiments with the Batch Ranking evaluation set, during the rebuttal period, as it naturally requires ranking multiple responses.

For example, what is the exact difference between Init + Borda and Ours + Borda (or any of the other combination) in an MLLM-as-a-judge paradigm?

We sincerely apologize for not clearly explaining the details of these method abbreviations in the first round. We will make the following clarification:

• Init represents the starting point for training the generative reward model, which is Qwen-2.5-VL-7B-Instruct.
• Ours represents the generative reward model trained using the GRPO algorithm starting from Qwen-2.5-VL-7B-Instruct.

An example of the evaluation dataset is:

{
    "id": "1",
    "image": "...",
    "question": "...",
    "responses": [
        {"model_name": "gpt4o", "response": "..."},
        {"model_name": "gemini-2.5-pro", "response": "..."},
        {"model_name": "llava", "response": "..."},
        {"model_name": "cogvlm", "response": "..."},
    ],
    "ground_truth": "ABCD"
}

Where id is the unique identifier for the evaluation, responses are the model-generated answers, and ground_truth is the human-annotated correct answer. In this case, it indicates that gpt4o's response (A) is the best, and cogvlm's response (D) is the worst.

In the specific implementation, we split data with the same id containing $n$ responses into $A_n^2$ pairwise responses, where the data format is:

{
    "id": "1",
    "image": "...",
    "question": "...",
    "responses": [
        {"model_name": "gpt4o", "response": "..."},
        {"model_name": "gemini-2.5-pro", "response": "..."},
    ]
},
{
    "id": "1",
    "image": "...",
    "question": "...",
    "responses": [
        {"model_name": "gpt4o", "response": "..."},
        {"model_name": "llava", "response": "..."},
    ]
},

Subsequently, we have the generative reward model, either Init or Ours, perform pairwise comparison on each piece of data. We then use different methods for ranking statistics, where the methods include: {Average, Borda, Instant Runoff, Ranked Pairs, Spectral Ranking}, resulting in the normalized minimal editing distance in the named suffix results in Table 1 (our Round 1 rebuttal).

In the same vein, and I apologize for not pointing this in the original review, for completeness sake, it might be better to briefly describe the MLLM-as-a-judge paradigm in the full paper before describing the results as well.

Excellent suggestion! We indeed overlooked the introduction to the MLLM-as-a-Judge benchmark. As mentioned above, we will add the corresponding introduction in Appendix Section 6 Experiment Details of the paper.

Following up on this, will this result be presented in the updated paper? In that case, the authors should mention some plans on which results will change.

We are very willing to share our revision plans with you. We will add a new section called Potential Directions in the appendix. We will discuss the potential of using the aforementioned social choice methods in Batch Ranking tasks and Reinforcement Learning Fine-tuning tasks, without making changes to the existing results in the main paper.

Finally, we greatly appreciate your quick feedback and suggestions. We hope our response in this round can address your concerns, and we look forward to further communication with you.

Dear Reviewer mrSo,

We would like to once again express our sincere gratitude for the time and effort you have dedicated during the rebuttal period. Throughout this round of rebuttal, you engaged in multiple rounds of discussion with us, and we are pleased to see that your concerns have progressively diminished. In fact, by the end, you even expressed your willingness to support the acceptance of our paper. However, it appears that one final point of discussion remains without your response.

We deeply value your feedback, as it has been immensely helpful in improving our paper. Therefore, we sincerely request that you take the time to review our responses and provide your comments, so that our paper can be further enhanced.