MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

Concern 1: MM-RLHF-Reward Model Training Details

We sincerely apologize for any lack of clarity regarding our reward model training process. Here are the key details:

1. Base Model: We initialized our reward model from LLaVA-OV-7B, following common practice in both LLM and MLLM research where reward models are typically derived from capable base models.

2. Training Setup:

• Hardware: 32× A800 GPUs
• Training Time: 8 hours
• Dataset Size: 120K samples

3. Training Objective: The reward model was trained exclusively using:

• The critique loss (Eq. 2 in our paper)
• Standard binary classification loss for reward modeling (Eq. 3 in our paper)
• Neither DPO nor MM-DPO was involved in reward model training

We would be happy to clarify any additional aspects of this process that may need further explanation.

---

Concern 2: LLaVA-Critic Fine-tuning Baseline

We appreciate this suggestion and have conducted additional experiments:

1. Data Adaptation: We reformatted our MM-RLHF data to match LLaVA-Critic's input requirements (pairwise responses with GPT-4o evaluations).

2. Training Variants:

   • Basic LLaVA-Critic-MM-RLHF (direct GPT-4o critiques)
   • Enhanced version with human-annotated critique expansions

The final results are as follows. We observe that fine-tuning LLaVA-Critic with our data yields a significant improvement, whereas directly using GPT-4o’s critic as a training objective shows limited gains. Even with the expansion of generated critiques based on human annotations, the performance does not surpass that of GPT-4o, though it does serve as a strong baseline. Additionally, we found that this training strategy is highly dependent on the model’s instruction-following capability, and at times, it fails to produce the expected comparative results during evaluation, requiring complex regularization matching.

---

Concern 3: It is expected that their model would perform (Tab. 2) the best on their MM-RLHF-RewardBench since the benchmark is sampled from their model’s training set!

First, the data used in MM-RLHF-RewardBench and our model’s training set do not overlap. As a standard practice in deep learning, we ensure by default that the training and test datasets are kept separate. To further clarify this, we will explicitly mention this in the main text.

To further prevent overfitting to the MM-RLHF dataset, we also evaluated our model on VL Reward Bench, as shown in Table 3. In this comparison, MM-RLHF-Reward-7B performs similarly to Claude-3.5-Sonnet, significantly outperforming the LLaVA-OV-7B baseline. This demonstrates the strong generalization capability of our critic-reward model.

Finally, in Table 2, when directly fitting the model to our training data, the results on the test samples are not as strong, at best matching GPT-4o. What we want to emphasize is the potential of the critic-based reward model; with optimal critic generation, the reward model could achieve up to 93% average accuracy.

---

Concern 4: The authors claim that the MM-DPO is useful but I cannot find any evidence of that in the main text.

Actually, we compare DPO with MM-DPO in Figures 1 and 11, where MM-RLHF refers to training with the MM-RLHF dataset using DPO loss, not SFT on high-rated examples. For a more detailed answer to the reviewer’s question, please refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17). We compared our approach to various baselines, including LLaVA-Critic, beta-DPO, SIMPO, and MPO, and found that existing methods showed limited gains on our high-quality preference dataset. Additionally, we compared DPO training results across multiple datasets (e.g., VL Feedback, RLAIF, LLaVA-Critic, MPO-Data), and the results demonstrated that MM-RLHF provides more comprehensive and significant performance improvements compared to existing datasets.

---

Concern 5 The authors should consider restructuring their paper.

Given the large amount of content, it's challenging to fit everything within the limited page count. We have decided to follow the reviewer’s advice.

First, we will condense the related work section, highlighting key content in the main text and providing comparisons. Second, we will include important experimental settings—such as baselines, model implementation details, and computational overhead—in the main text, while moving less critical experiments to the appendix.

Concern 1: Critical Baseline Missing: There is no direct comparison between MM-DPO and standard DPO, making it impossible to quantify the benefit of dynamic reward scaling itself.

Actually, we compare DPO with MM-DPO in Figures 1 and 11, where MM-RLHF refers to training with the MM-RLHF dataset using DPO loss, not SFT on high-rated examples. For a more detailed answer to the reviewer’s question, please refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17). We compared our approach to various baselines, including LLaVA-Critic, beta-DPO, SIMPO, and MPO, and found that existing methods showed limited gains on our high-quality preference dataset. Additionally, we compared DPO training results across multiple datasets (e.g., VL Feedback, RLAIF, LLaVA-Critic, MPO-Data), and the results demonstrated that MM-RLHF provides more comprehensive and significant performance improvements compared to existing datasets.

---

Concern 2 Reward Model vs Base Model Gap:

First, using an untrained model directly as a reward model performs poorly (see LLaVA-OV-7B in Tables 2 and 3) and is practically unusable for our training objectives. This necessitates training a dedicated reward model rather than relying on raw MLLMs. Numerous studies have explored specialized reward model training rather than naively repurposing MLLMs as reward models [1,2,3].

Furthermore, to mitigate overfitting to the MM-RLHF dataset, we evaluated our model on an independent reward model benchmark (Table 3). Results show that MM-RLHF-Reward-7B performs comparably to Claude-3.5-Sonnet on the VL Reward benchmark and significantly outperforms the LLaVA-OV-7B baseline. This demonstrates strong generalization of our critic-reward model across diverse datasets.

[1] Aligning large multimodal models with factually augmented RLHF

[2] LLaVA-Critic: Learning to evaluate multimodal models

[3] Self-generated critiques boost reward modeling for language models

---

Concern 3: Modest Gains: Reported improvements over SFT baselines are relatively small

We emphasize that "MM-RLHF" specifically refers to training with the MM-RLHF dataset using DPO loss—we did not include an SFT-only baseline. Please refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17) for General Response 2. Comparison of SFT Baselines and DPO Sample Selection Strategies. The results that preference-based training (DPO/MM-DPO) is essential for robust performance. MM-DPO further outperforms standard DPO, highlighting its effectiveness:

---

Concern 4: Contextualization with Early RLHF Pipelines and Safety Alignment We appreciate the reviewer’s constructive feedback on relating our work to broader literature.

1. Early Text-Based RLHF (e.g., InstructGPT): A key limitation of traditional RLHF methods lies in their sensitivity to hyperparameters and dependence on base model capabilities. For instance, when testing PPO with LLaVA-Ov-7B (actor) and our MM-RLHF-Reward-7B (critic), we observed only marginal improvements in dialogue tasks, alongside the need for meticulous tuning to avoid training instability. In contrast, MM-RLHF’s high-quality responses (e.g., from Qwen2-VL-72B) make DPO-based methods more intuitive and stable for achieving consistent gains.

2. Safety Alignment in Vision-Language Models: While prior work [1] focuses on trade-offs between safety and model capability (often at the cost of performance), our pipeline demonstrates that proper data construction can simultaneously enhance both safety and general capabilities.

[1] Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models

---

Concern 5: Evaluation Benchmark Credibility

We acknowledge concerns about potential bias in our custom benchmarks (MM-RLHF-RewardBench and MM-RLHF-SafetyBench). To address this:

1. Data Leakage Prevention: Strict separation between training and test data was enforced.

2. Reward Model Generalization: As shown in Table 3, MM-RLHF-Reward-7B matches Claude-3.5-Sonnet on the VL Reward benchmark and vastly outperforms LLaVA-OV-7B, confirming cross-dataset robustness.

3. SafetyBench Provenance: MM-RLHF-SafetyBench was curated from existing benchmarks with no overlap with training data.

4. General Knowledge Evaluation: We tested on diverse, independent benchmarks (e.g., LLaVA-Wild, OCRBench) to ensure reliability.

We welcome further discussion on benchmark design if needed.

---

Concern 6: Failure Case Analysis in Safety-Critical Scenarios

We agree that safety failure analysis is crucial. While visual examples cannot be included here, please refer to (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17) Reviewer xZJU: Analysis on failure cases, especially in safety-critical scenarios for a detailed failure case.

Concern 1: The paper acknowledges this limitation but doesn't provide strong solutions.

It's important to note that not all models show minimal improvements on these benchmarks. For example, InternVL2 performs better on RealWorld tasks (from 43.1 to 44.9), which may relate to its image segmentation strategy. We recently discovered that the limited number of high-resolution images in the training data may be a key reason for this. Most public datasets have few high-resolution images, and our use of CLIP features for clustering reduced the number further. To address this, we sorted the dataset by resolution and re-annotated the top 1k highest-resolution samples. This led to significant improvements, with LLaVA-OV-7B increasing from 55.3 to 56.8. Based on these results, we believe increasing the number of high-resolution samples will improve performance on such tasks. We plan to continue adding more high-resolution data in the future.

---

Concern 2: Dynamic Reward Scaling technique is a relatively incremental advancement over DPO methods. There are already similar ideas proposed for reward model training [1].

At first, the reference [1] in the reviewer's question is missing.

In fact, we discuss the differences between Dynamic Reward Scaling, DPO, and its improved versions in line 304 of the original paper. Overall, there are three key distinctions:

1. Novelty: We are the first to propose the dynamic beta adjustment framework for MLLMs.

2. Methodological Advance: We demonstrate that instance-level beta tuning is viable with a robust reward model, contrary to prior beliefs [1].

3. Empirical Gains: Our approach outperforms existing methods (Figure 11(a)).

[1] Beta-DPO: Direct preference optimization with dynamic beta.

Additionally, we have already elaborated on the differences between similar reward model training methods and our approach in line 261. We conclude the main content as follows: In the MLLM community, there is no unified framework for designing reward models. Some approaches use traditional reward models with limited interpretability, while others rely on LLMs for ranking, which often leads to high variance. Additionally, other works focus on improving the reliability of model-generated critiques, but with a different goal. Our study is the first to explore how MLLMs can effectively leverage human annotations to enhance both interpretability and scoring ability.

For a detailed discussion, please refer to Appendix F (Comparison to Existing Methods on Beta Adjustment in LLMs and MLLMs) and Appendix E (Discussion of MM-RLHF-Reward Model).

---

Concern 3: This paper did not compare itself or discussed some previous RLHF methods or datasets for multimodal LLM training, like MIA-DPO and PPLLaVA, LLaVA-OneVision-Chat

PPLLaVA directly uses DPO loss without any improvements, so a direct comparison is not feasible. For a more detailed answer to the reviewer’s question, please refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17). We compared our approach to various baselines, including LLaVA-Critic, beta-DPO, SIMPO, and MPO, and found that existing methods showed limited gains on our high-quality preference dataset. Additionally, we compared DPO training results across multiple datasets (e.g., VL Feedback, RLAIF, LLaVA-Critic, MPO-Data), and the results demonstrated that MM-RLHF provides more comprehensive and significant performance improvements compared to existing datasets.

---

Concern 4: Is all possible pairs necessary? and What does “MM-RLHF” refer to—SFT on high-rated examples or the reward model’s output?

Please refer to this link (https://anonymous.4open.science/r/mm-rlhf-rebuttal-BE17) for General Response 2. Comparison of SFT Baselines and DPO Sample Selection Strategies.

---

Concern 5: Figures would benefit from error bars or statistical significance indicators.

Thank you for the suggestion. Since we use fixed seeds during training and temperature=0 for deterministic generation during evaluation, performance variance is generally small. However, we agree this is a valuable improvement. Due to the time limit, we will report the mean and standard deviation across three runs with different seeds in the final version.

---

Concern 6: Why train the MM-RLHF reward model if it’s not used for PPO or other RL methods?

The reward model is essential in our MM-DPO method for computing reward margins.
We did experiment with PPO as well, but similar to the observations in Concern 3 about multi-stage DPO, PPO requires online sampling and fine-grained hyperparameter tuning. We only observed improvements in relatively simple dialogue tasks with a 7B LLaVA-ov model, and training stability was a challenge. In contrast, DPO-based methods offer more robust performance, especially given the high-quality responses (e.g., from Qwen2-VL-72B) in the MM-RLHF dataset, making them a more practical and effective choice.

Thank you for your time and for acknowledging our work. We will address each of your concerns below:

---

Concern 1: This work does not seem to explicitly incorporate a mechanism that prevents test data leakage/contamination

Thank you for raising this important issue. In our work, we have implemented several measures to minimize the risk of leakage between the training and test datasets:

1. Strict separation between training and test data: During the data sampling process, we manually ensure that the selected samples are from the training set and have no overlap with the evaluation subset.
2. Experimental validation: We tested our results on more than twenty benchmarks, and observed consistent improvements across domains, with no signs of overfitting in any specific domain. The safety-related data showed the largest improvements because the model had minimal exposure to safety-related issues during pretraining. However, our safety data was newly constructed, clearly distinct from the benchmarks used.

---

Concern 2: This work does not include an evaluation of the annotation consistency of different human annotators.

In fact, our annotation process involves multiple rounds of interactive validation (at least two rounds). We will include the following details in the main text:

1. Clear annotation guidelines and training: As shown in Appendix B, we provided detailed annotation guidelines and training for annotators, ensuring that they could consistently understand and execute the annotation tasks. This helps reduce annotation inconsistency among different annotators.
2. Annotation review and iteration: To further improve consistency, we implemented a multi-step review process. The first annotator performs the annotations, then another annotator reviews them to ensure agreement. In cases of inconsistency, a third annotator is introduced, and the final decision is made by selecting the most suitable annotations.

---

Concern 3: There seems to be no clear evidence that the dataset has sufficient geographical diversity.

This is an interesting point. In Figure 3, we show the task richness of the dataset, and the images involved may contain buildings or natural landscapes from various geographical locations around the world.

However, since most existing public datasets are in English, our mm-rlhf model is also affected by this issue. The majority of the scenarios are still based on English-language contexts. As the reviewer pointed out, there may be a lack of sufficient geographical diversity. We have recognized the importance of geographical diversity and are actively working on collecting multimodal data from diverse geographical locations and languages, including Chinese (Mandarin), French (French), and other commonly spoken languages, to further enhance the geographical diversity of our dataset.

---

Concern 4: Please improve the resolution of Figure 1.

Thank you for the suggestion! We have updated the original image to a PDF format to avoid the issue of low resolution.