Calibrated Self-Rewarding Vision Language Models

Thank you for your valuable feedback. We've addressed your questions below and would appreciate it if you could let us know whether our responses meet your expectations.

Q1: The ablation of the weight of instruction-following score and image-text alignment score is missing.

A1: We evaluate the effectiveness of varying $\lambda$ on the performance. The results are shown in Table R1 and indicate that placing greater emphasis on visual calibration significantly improves performance, further strengthening our contribution to reward calibration.

Table R1: Performance vs $\lambda$

---

Q2: Inverstigating the impact of different image-text alignment scoring approaches.

A2: To further demonstrate the compatibility of CSR, we conducted an experiment using GPT-4o to calculate the image-response relevance score, which offers stronger image perception capabilities. We used the first iteration of CSR training on LLaVA-1.5-7B as an example, and the results are shown in Table R2. The results indicate that using a model with stronger image perception capabilities to calibrate the initial reward can provide improvements. We plan to include the performance of using more vision-centric models to calibrate the initial reward in the next version.

Table R2: Comparison between different models that are used to calculate image-response relevance score

---

Q3: Distinguish 1) comprehensive, 2) general VQA and 3) hallucination scores in the main ablation.

A3: We have split the ablation results and reported them in Table R3. The results indicate that relying solely on the image-response relevance score increases the hallucination score and slightly negatively impacts, or at best, only marginally improves, the VQA and comprehensive benchmarks (likely affecting only the VQA performance in these benchmarks).

Table R3: Fine-grained results of ablation study

---

Q4: The terminologies “sentence group” and “option” in Figure 2 are confusing.

A4: We have polished Figure 2 and put it in the supplementary PDF (see Figure R2). Here is an explanation of this figure: Since CSR uses step-level reward modeling, we initially generate a few sentences (five for illustration). Each of these sentences is then used to sample five possible next sentences, resulting in a total of 25 sentences, which is named “Option”. To improve efficiency, only the top 3 sentences (indicated by green dots in the figure) and the bottom 2 sentences (indicated by red dots) are retained, which is named as “Sentence Group”. The retained five sentences are then used to generate the third step sentence, and this process is repeated until the response is complete.

---

Q5: What is the consideration to use the same examples in different iterations?

A5: In CSR, we use the same samples for iterative learning for two main reasons:

• A single round of preference learning may not fully resolve all issues. Multiple iterations allow for the incremental correction of the model's errors on these samples, leading to further improvements in model performance.

• Using the same samples helps us investigate whether the model's responses to these samples improve across different iterations. This consistency is useful for analytical experiments (e.g. Figure 5).

Additionally, we conducted an experiment during the second iteration of CSR with LLaVA-1.5-7B, where we selected 13,000 images that were entirely distinct from those used in the first iteration. The results of this experiment are reported in Table R4. It can be observed that using different samples across iterations performs similarly to using the same samples. We plan to conduct further investigations on this topic in the next version.

Table R4: Results w/ random image batch over different iterations

Thank you for your constructive comments. Below are our responses to your questions. Please let us know if they address your concerns.

Q1: In the limitation section, the authors mention that they could only run three iterations due to computation issues. However, in Section 4.1, … supplement the claim of increased performance concerning iterations made in Section 4.2.

A1: Using LLaVA-7B as an example, we performed two additional rounds of iterative training on top of the three rounds of CSR training detailed in Table 7. The results are presented in Table R1. According to the results, we observe that more iterations can still improve model performance, especially in mitigating hallucination. However, the overall improvement tends to slow down with additional iterations (see Figure R5 in Supplementary PDF), indicating performance convergence.

Table R1: Results of more rounds of evaluation

---

Q2: Concern about the CLIP model in capturing details.

A2: First, we would like to clarify that our proposed CSR framework is general enough to incorporate different vision-centric models used to calculate image-response relevance scores, even though CLIP is used in this paper. To further demonstrate the compatibility of CSR, we conducted an experiment using GPT-4o to calculate the image-response relevance score, which offers stronger image perception capabilities. We used the first iteration of CSR training on LLaVA-1.5-7B as an example, and the results are shown in Table R2. The results indicate that using a model with stronger image perception capabilities to calibrate the initial reward can provide improvements compared to using CLIP, although CLIP is sufficiently strong and lightweight. We plan to include the performance of using more vision-centric models to calibrate the initial reward in a future version.

Table R2: Comparison of different models used to calculate image-response relevance scores

---

Q3: how varying $\lambda$ terms affect the score.

A3: We evaluate the effectiveness of varying $\lambda$ on the performance. The overall training settings are the same as in Table 1, with three rounds of iteration. The results are shown in Table R3 and indicate that placing greater emphasis (i.e., $\lambda=0.9$) on visual calibration performs best, further strengthening our contribution to reward calibration.

Table R3: Performance w.r.t. $\lambda$

---

Q4: Can a rough estimate of how different the computing resource would be, including the time to gather the preference data for previous methods?

A4: In CSR, the data is constructed by the target LVLMs themselves and calibrated using the CLIP model, which eliminates the need for any human effort or costly closed-source models (e.g., GPT-4). In terms of the time required to gather the data, using LLaVA-1.5-13B as an example, it takes around 12 hours to obtain 13,000 pairs of high-quality preference data on an A6000ada GPU node. In contrast, manual data correction represented by RLHF-V [1] significantly involves human effort, as well as time and monetary costs. For POVID [2], constructing one dataset (which may not be optimal) requires approximately $300-400 in GPT-4 API costs.

---

Q5: (minor) The overall framework figure (Figure 2) seems hard to understand.

A5: We have polished Figure 2 and put it in the supplementary PDF (see Figure R2). Here is an explanation of this figure: Since CSR uses step-level reward modeling, we initially generate a few sentences (five for illustration). Each of these sentences is then used to sample five possible next sentences, resulting in a total of 25 sentences. To improve efficiency, only the top 3 sentences (indicated by green dots in the figure) and the bottom 2 sentences (indicated by red dots) are retained. The retained five sentences are then used to generate the third step sentence, and this process is repeated until the response is complete.

---

References

[1] Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.

[2] Aligning modalities in vision large language models via preference fine-tuning

Thank you for your valuable feedback. We have answered your questions below, and we would appreciate it if you could let us know whether our responses address your concerns.

Q1: The methodological contribution is not sufficient as it basically follows self-rewarding language models [11] and applies it in another domain. The proposed techniques such as using CLIPScore to calibrate the self-generated scores are incremental changes to me.

A1: We would like to highlight our contribution in comparison to LLM self-rewarding. Compared with self-rewarding in LLM, the fundamental difference in LVLM is that using the target LVLM as both the generator and judge (as is done in LLM self-rewarding) can amplify the misalignment issues between text and image modalities. This happens because the judge itself also tends to be biased toward the contextual information, often ignoring the input image. By incorporating visual constraints, we can effectively mitigate this issue, as it encourages better alignment between text and image modalities within the judge. We believe this is the fundamental difference between our approach and LLM self-rewarding. We believe these contributions are significant.

---

Q2: The empirical results do not seem significant in many of the datasets. In Table 1, on popular benchmarks such as GQA and SEED, the performance improvements are marginal (one exception may be LLaVA-W, but this dataset is small and relies on GPT evaluations). Results on VQAv2 and MMStar should also be added.

A2: We would like to emphasize that the primary goal of CSR is to enhance the alignment between image and text modalities and minimize hallucination. CSR has demonstrated substantial improvements over other methods across the majority of the datasets evaluated. These improvements are significant; we achieved average improvements of 8.23% and 2.59% compared to the original 7B model and the strongest baseline (i.e., self-rewarding), respectively.

To further evaluate the effectiveness of CSR, we evaluated LLaVA-1.5 7B and 13B of CSR for VQAv2 and MMStar, following the setup in Table 6. The results are shown in Table R1, including the original LLaVA, the strongest baseline (self-rewarding), and CSR. The performance gain of CSR on these additional benchmarks further demonstrates the effectiveness of CSR.

Table R1: Performance on VQAv2 and MMStar

---

References

[1] Zhou Y, Cui C, Rafailov R, et al. Aligning modalities in vision large language models via preference fine-tuning[J]. arXiv preprint arXiv:2402.11411, 2024.

[2] Yu T, Yao Y, Zhang H, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13807-13816.

[3] Sun Z, Shen S, Cao S, et al. Aligning large multimodal models with factually augmented rlhf[J]. arXiv preprint arXiv:2309.14525, 2023.

[4] Li L, Xie Z, Li M, et al. Silkie: Preference distillation for large visual language models[J]. arXiv preprint arXiv:2312.10665, 2023.

Dear Reviewer bHbK,

Thank you for your response and for increasing your score. We're delighted that our answers addressed your questions. We'll continue exploring ways to further calibrate reward scores in the future.

Thank you for reviewing our paper and providing valuable feedback. We address your concerns point by point below and would appreciate knowing if our responses address them.

Q1: Technical Novelty: The primary distinction of the proposed method … differs from and improves upon existing methods.

A1:

• Compared with self-rewarding in LLM, the fundamental difference in LVLM is that using the target LVLM as both the generator and judge (as is done in LLM self-rewarding) can amplify the misalignment issues between text and image modalities. This happens because the judge itself also tends to be biased toward the contextual information, often ignoring the input image. By incorporating visual constraints, we can effectively mitigate this issue, as it encourages better alignment between text and image modalities within the judge. We believe this is the fundamental difference between our approach and LLM self-rewarding.
• Compared with other modality alignment enhancement approaches, CSR does not rely on additional VLMs or human annotators to generate preference data, which allows it to better capture and correct the inherent preferences of the target LVLM.

We believe our contributions are significant and fundamentally different from existing methods.

---

Q2: It would be beneficial to explore the reasons behind these limited improvements … further enhancement.

A2: Based on Table 1, CSR performs significantly better on hallucination benchmarks that evaluate image captioning quality compared to closed-ended VQA benchmarks. This is because preference optimization may be better suited for enhancing the performance of open-ended questions. In preference optimization, the preferred and dispreferred responses in the preference data are possibly partially correct, with the preferred response simply being better than the dispreferred one. Optimizing the model with this preference data can strengthen its ability to distinguish between both responses and accurately capture more details in long-form, open-ended image captioning tasks. On the other hand, some closed-ended VQA tasks use a fixed set of response options (e.g., yes/no, multiple-choice), which rely more on the visual model's perception capability and may not be as easily improved by preference optimization compared to long-form image captioning. Similar characteristics are exhibited across POVID [1], RLHF-V [2], Human-Prefer [3], and Vlfeedback [4].

---

Q3: How is the improvement of approximately 7.62% computed?

A3: As mentioned in Line 204, the improvements refer to the average percentage increase across all benchmarks when comparing CSR (7B) and LLaVA-1.5 7B. The method we used to calculate the average percentage increase is as follows:

First, to calculate an average score on a 100-point scale, we adjusted the original values: MME$^P$ was divided by 16, and MME$^C$ was divided by 4, according to # categories in MME. Additionally, since a lower CHAIR value indicates better performance, we standardized all metrics to follow a "higher is better" approach by transforming the CHAIR$_S$ and CHAIR$_I$ metrics into 100 - CHAIR$_S$ and 100 - CHAIR$_I$. Then, we calculated the average score by averaging the standardized values, which were used to compute the average percentage increase.

Regarding the specific value, we apologize for a computational error in calculating the average percentage increase using the data from Table 1. The correct value is 8.23%.

---

Q4: What is the upper bound in the performance gain of CSR? A case study would be helpful.

A4: We have included two case studies selected from the CSR-generated datasets in Figure R1 in the supplementary PDF. As the CSR iteration increases, we observe performance gains, the alleviation of challenging hallucinations (such as counting issues), and enhanced fine-grained perception. Combined with the qualitative improvements of CSR in various benchmarks, these additional case studies further support the effectiveness of CSR.

---

Q5: Figure 2 is unclear.

A5: Since CSR uses step-level reward modeling, we initially generate a few sentences (five for illustration). Each of these sentences is then used to sample five possible next sentences, resulting in a total of 25 sentences. To improve efficiency, only the top 3 sentences (indicated by green dots in the figure) and the bottom 2 sentences (indicated by red dots) are retained. The retained five sentences are then used to generate the third step sentence, and this process is repeated until the response is complete. We have refined Figure 2 in the supplementary PDF (see Figure R2).

---

Q6: visualize the generated preference data

A6: We visualized a case of preference data generated by the model itself in the CSR process in the supplementary PDF (see Figure R3).

---

Q7: Additionally, Figure 1 appears confusing.

A7: Figure 1’s radar chart shows the scores of self-rewarding, LLaVA-1.5-7B, and CSR-7B across all benchmarks. A larger area in the radar chart indicates higher scores. To clarify it, we polish Figure 1 in the supplementary PDF (see Figure R4).

---

Q8: The potential negative societal impact of the work should be considered

A8: CSR depends on the quality of self-generated preference data. Although it achieves overall higher performance, there is still a possibility that the data can contain errors, which may misguide the learning process in some situations and ultimately lead to erroneous decisions in safety-critical applications like healthcare.

---

References

[1] Aligning modalities in vision large language models via preference fine-tuning.

[2] Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.

[3] Aligning large multimodal models with factually augmented rlhf.

[4] Silkie: Preference distillation for large visual language models.

Thank you for your valuable feedback to help us improve our paper. We detail our response below and please kindly let us know if our response addresses your concerns.

Q1: How did the performance improve over each preference data generation stage?

A1: In the first round of training, our model (referred to as model-iter0) generated a batch of preference data based on the approach discussed in Section 3.2.1. Then, by employing preference optimization, we optimized model-iter0 and obtained model-iter1. Compared to model-iter0, model-iter1 exhibits better performance and generates responses that are better aligned with the image input. In the second round of training, similarly, we used model-iter1 to generate a batch of preference data. Due to the improved performance, this batch of data is of higher quality compared to the previous round’s preference data and is more challenging to distinguish between preferred and dispreferred data. Consequently, using this new preference data to perform preference optimization on model-iter1 yields a stronger model, i.e., model-iter2.

In this way, we can perform more rounds of iteration until convergence (see empirical results in Figure 3 in the paper).

In summary, because the quality of preference data can improve over different rounds and it becomes increasingly difficult for the model to distinguish preferences, performing preference optimization each round will enhance the model’s performance.

---

Q2: .Scaling study-Gao et al., "Scaling Laws for Reward Model Overoptimization", ICML 2023

A2: Since CSR utilizes the model itself to establish an initial reward and then calibrates this initial reward by incorporating an image-response relevance score computed by the CLIP model, we analyzed the effect of the reward model's (CLIP model) size. We conducted experiments with different sizes of CLIP models to calibrate the initial reward. Due to rebuttal time constraints, all experiments were conducted with only one round of CSR training on LLaVA-1.5-7B. The results are reported in Table R1. The experimental results show that using larger and more powerful CLIP models as the reward model reduces the model's hallucinations and improves comprehensive benchmark performance. The results meet with our expectations, as stronger models can better align image and text responses.

Table R1: Performance with respect to different sizes of CLIP models

Thanks a lot for increasing your score, we are happy that our response addresses your concerns. For the follow up questions, we detail our response below and please kindly let us know if our response addresses your concerns.

Q1: Following A1, is it possible to add iter-4 results (even it's negative)? Since for the 13B model it seems the performance not yet converges.

A1: We added additional iteration results for the LLaVA-7B and 13B models, which are presented in Tables R1 and R2, respectively. Due to the time constraints of the discussion period, we conducted four additional iterations for the 7B model (iter-4 through iter-7) and two additional iterations for the 13B model (iter-4 and iter-5). As shown in the "Avg score" column in both Table R1 and Table R2, the overall improvement tends to slow down or even fluctuate with each additional iteration, indicating that the performance is converging.

Table R1: Results of additional iteration evaluation of 7B model

Table R2: Results of additional iteration evaluation of 13B model

---

Q2: do you observe "model collapse" such that the reward keep increasing yet the benchmark score drops

A2: We analyzed the relationship between the reward score and the final average performance of LLaVA-7B, with the results presented in Table R3. In the later stages of training, it can be observed that a slight increase in the reward score coincided with a slight decrease in model performance. One possible reason is that the model's performance had mostly converged, yet it continued to overfit the training data in pursuit of a higher reward score. Another explanation could be the inherent limitations of the target LVLM and the reward-calibrating model, making it increasingly challenging to accurately assign rewards to certain difficult data points. Early in training, these mis-rewarded data points may not have had a significant impact because the majority of the data was useful. However, as the model's capabilities improved, the mis-rewarded data began to affect overall performance. We believe that introducing more diverse vision-centric tools for reward calibration, along with a more thoughtful data selection mechanism, could potentially alleviate this issue. We plan to systematically explore these methods further in the future.

Table R3: Results of rewards w.r.t. average score