Multimodal Large Language Models Make Text-to-Image Generative Models Align Better

Thank you for your detailed, helpful feedback. We address your feedback point by point below.

---

Q1: It is better if the authors could provide some quantitative metrics.

A1: Thank you very much for your insightful feedback :). Firstly, in Table 2 of the main text, we provide a comparison of the preference prediction accuracy scores of our reward model VP-Score, with other state-of-the-art reward models. Additionally, we present the numerical results for win count & win rate under PPO and DPO experiments in Tables 4 and 5 in the appendix. These quantitative results demonstrate the effectiveness of using MLLMs to generate preference feedback.

Moreover, in response to your valuable suggestion, we conducted additional measurements. We compared the FID values (a widely recognized metric for evaluating the quality of images generated by models) under PPO and DPO experimental settings for generative model optimized with our data and model against those optimized with other models. As shown in Tables A and B, we found that the FID scores of ours are significantly lower than those of the other comparison groups, indicating the effectiveness of using MLLMs-generated feedback to align the generation model.

We will include these findings in the revised version of our paper to enrich the content. If you have other quantitative metrics of interest, please feel free to let us know :).

Table A. FID score of generative models optimized with VP-Score compared to other reward models for PPO experiments.

Table B. FID score of generative models optimized with VisionPrefer compared to other human preference datasets for DPO experiments.

---

Q2: A significant contribution of this paper lies in the dataset. However, examples of annotations in VisionPrefer can only be found in the appendix. It would be more intuitive if these were included in the main paper.

A2: Considering your insightful recommendations, in the revised paper, we will move the examples of annotations in VisionPrefer to the main body to make the central focus of the article clearer and more reader-friendly :).

Thank you for your detailed, helpful feedback. We address your feedback point by point below.

---

Q1: Figure 3 indicates that VP-Score converges slower than others.

A1: In fact, Figure 3 does not represent a comparison of the convergence speeds of VP-Score and other baselines. Instead, it illustrates how the human preference quality of a generative model, which uses VP-Score as a reward model for alignment, changes during the fine-tuning process (as noted in lines 173-177 of the main text). This is assessed by various reward models such as ImageReward and HPS v2.

As shown in Figure 3, the quality of human preferences in the generative model increases as the fine-tuning steps progress. This improvement is reflected in the increasing reward score of models such as HPS v2, which are trained on human preference data. This indicates that VP-Score can serve as a reliable reward model for aligning generative models, further proving the reliability of using MLLMs to generate human preference feedback.

Additionally, inspired by your feedback, we observed that the growth in metrics for ImageReward, PickScore, and Aesthetic is not very pronounced. We hypothesize that these three models may struggle to capture variations in the quality of human preferences in images. In contrast, HPS v2, currently the most accurate and unbiased model for reflecting human preference quality, exhibits a clear upward trend. This suggests that our VP-Score aligns better with HPS v2, further validating the effectiveness of VP-Score.

Furthermore, we appreciate your suggestion. All authors agree that the convergence speed of using human preference reward models for aligning generative models is an intriguing research question, and we plan to explore this in future work.

---

Q2: The authors should analyze why VisionPrefer fails to deliver uniform performance gains and identify aspects where it falls short.

A2: We first analyzed the differences between GPT-4V and human annotations at a fine-grained level: we randomly selected 1,000 samples from VisionPrefer and invited six human experts to rate them on four dimensions (Prompt-Following, Aesthetic, Fidelity, and Harmlessness). The correlation between human experts' preference judgments and GPT-4V's judgments is shown in Table A below. We found that GPT-4V's preference annotations align more closely with human experts in the areas of Prompt-Following and Fidelity (the same phenomenon can be observed in Figure 10 e in the main text). This suggests that GPT-4V is better aligned with human preferences in these two aspects. This finding might indicate that we can reduce the weight of Aesthetic and Harmlessness scores when constructing VisionPrefer to achieve more accurate preference annotations.

Furthermore, we hypothesize that using the average score across the four aspects as the final preference score in VisionPrefer might differ from human preference evaluations. Humans may prioritize certain aspects more heavily, such as Prompt-Following and Aesthetic, which could be more significant than Harmlessness. Supporting this hypothesis, Table 2 in the main text shows that VP-Score achieves the highest accuracy on ImageRewardDB but not on the other two datasets. ImageRewardDB uses an averaged score across multiple aspects for human preference evaluation, similar to our VisionPrefer construction, while Pick-a-Pic and HPD involve direct human preference scoring without fine-grained evaluation across multiple aspects.

Based on these insights, we envision two possible improvements for GPT-4V annotations: (1) Require GPT-4V to provide a confidence estimate for each aspect-specific preference score, using a weighted average of the four aspects as the final sample score. (2) Modify GPT-4V’s annotation process to evaluate all four aspects simultaneously for each text-image sample, outputting a final score rather than an average.

In summary, your question is very intriguing, and we will continue to explore and experiment with these ideas in future research.

Table A.

---

Q3: Bias in Annotations.

A3: All authors consider your suggestion both important and intriguing. In response, we have reviewed and searched for related research [1-4], including analyses of biases in AI-generated annotations and methods for detecting them. We will focus on these issues in future work to further optimize our VisionPrefer and VP-Score. Additionally, we will include your suggestion in the Limitations section to enrich the paper.

Reference

[1] Fang, Xiao, et al. "Bias of AI-generated content: an examination of news produced by large language models.".

[2] Fan, Zhiting, et al. "BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs.".

[3] Morehouse, Kirsten, et al. "Bias Transmission in Large Language Models: Evidence from Gender-Occupation Bias in GPT-4.".

[4] Hajikhani, Arash, and Carolyn Cole. "A critical review of large language models: Sensitivity, bias, and the path toward specialized ai.".

---

Q4: Combining AI and Human Annotations.

A4: Thanks. In response, we conducted the following experiment: we combined our VisionPrefer dataset (AI annotations) with a randomly selected subset from the Pick-a-Pic dataset (human annotations). We then retrained a new reward model, named VP-Score+x, where x represents the percentage of Pick-a-Pic data used, while keeping the training steps constant. The preference prediction accuracy scores are shown in Table B below. We found that mixing AI and human annotations further improves the reward model's accuracy in reflecting human preferences.

Table B.

Thank you for your feedback. We address your questions point by point below:

---

Q1: the overall novelty is a bit limited as it heavily relies on existing large models to generate the dataset and to train additional models.

A1: In fact, the main motivation for our research is to explore: Can Multimodal Large Language Models act as a Human-Aligned Preference Annotator for Text-to-Image Generation? (as noted in line 38 of the main text). We aim to demonstrate that MLLMs can generate reliable human preference annotations for text-to-image generation results. To this end, we explored the performance of several state-of-the-art MLLMs (GPT-4V, Gemini-pro-V, and LLaVA 1.6) in generating human preference annotations (see Figure 10a in the main text). Using the best-performing GPT-4V, we constructed VisionPrefer, the largest fine-grained preference dataset to date, and developed the reward model VP-Score derived from VisionPrefer. We validated the effectiveness of VisionPrefer and VP-Score under PPO and DPO experimental settings.

We believe our research has the following significance:

• Since human annotation of preference labels is expensive and time-consuming, proving the reliability of MLLMs in generating human preference annotations allows humans to easily scale existing human preference datasets, thereby achieving better text-to-image generative alignment.
• It further supports the notion that AI-generated synthetic data is a reliable way to create augmented datasets and provides a large dataset for further research related to AI-generated synthetic data.
• It shows that existing AI models may exhibit capabilities similar to those of human experts in some interesting areas, e.g., image aesthetics, which is exciting and worthy of further exploration.

---

Q2: the paper seems to have a diverse set of focuses. For example, the section regarding editing images does not seem to correlate well with the main proposed method, thus seem irrelevant.

A2: We included the section on editing images to demonstrate that using MLLMs for preference annotation offers more than just aligning generative models to produce images that match human preferences, similar to existing human-annotated preference datasets. A key advantage of MLLMs is their ability to provide detailed textual feedback for each text-to-image sample, which can guide further image editing to produce images that better align with human preferences. This is another critical advantage of MLLMs beyond their ability to construct reliable preference labels quickly and cost-effectively. In this section, to further validate this advantage, we designed a simple pipeline (see in the Figure 8 in the maintext) using MLLM-generated textual feedback for image editing. Therefore, this section does not indicate a lack of focus in our paper but rather highlights the advantages of our approach.

---

Q3: The authors only showed the empirical strength of the proposed method with a randomly sampled subset of the images. I recommend the authors to repeat the experiments with the standard large-scale training regime.

A3: We demonstrated the effectiveness of our conclusions and methods in two ways:

• For the accuracy of the reward model's preference prediction (see Table 2 in the main text), we computed quantitative metrics on three public test sets without any random components, ensuring reliable results.
• To validate the model alignment under PPO and DPO settings, we followed existing research [1]. We generated 64 images for all aligned generative models and selected the top 3 images using the corresponding reward model. We then invited ten human annotators to evaluate these top 3 images. This process was repeated on three test sets for both PPO and DPO settings, with a workload and rigor that exceeds previous studies [1] (only three annotators on one test set). Therefore, we believe the qualitative and quantitative results presented in this paper substantiate the validity of our conclusions and methods.

Unfortunately, we did not fully understand your suggestion to "repeat the experiments with the standard large-scale training regime." We look forward to further discussions during the diffusion stage and welcome more specific feedback to conduct additional experiments.

Reference

[1] Xu, Jiazheng, et al. "Imagereward: Learning and evaluating human preferences for text-to-image generation." NeurIPS 2024.

---

Q4: To show the true value of the method, the authors might have to use the method and GPT-4V to train a bigger model.

A4: The primary motivation of our research is to explore: Can Multimodal Large Language Models act as a Human-Aligned Preference Annotator for Text-to-Image Generation? We aim to demonstrate that MLLMs can generate reliable human preference annotations for text-to-image generation results. Proving that MLLMs can reliably generate these annotations offers many benefits, such as constructing preference labels quickly and cost-effectively.

Therefore, we naturally rely on the powerful capabilities of MLLMs (such as GPT-4V), as stronger models may produce higher-quality preference annotations, which is a desirable outcome. Unfortunately, we did not fully understand your specific suggestion regarding "training a bigger model to show the true value of the method." We look forward to further discussions with you during the diffusion stage and are eager to conduct additional experiments based on these more detailed discussions.

---

Q5: Require for an explicit limitation section.

A5: Thank you for your suggestion. We will include an explicit limitation section in the latest version, summarized as follows: Due to resource constraints, although our VisionPrefer is the largest fine-grained preference dataset for text-to-image generative alignment, we believe it can be further scaled to achieve better alignment performance. We plan to expand it further in the future.

Dear reviewer,

thanks for your participation in the NeurIPS peer review process. We are waiting for your response to the rebuttal. You gave a reject rating (3), with 2 weaknesses and 2 questions.

Is the response from authors satisfactory?

• If yes, are you planning to increase your score?
• If no, could you help the authors understand how they can improve their paper in future versions?

Thanks, AC