Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models

Dear Reviewer fRcb,

We sincerely appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below. Please let us know if you have any further questions or concerns.

---

[W1] Additional aspects of text-image alignment in prompt synthesis.

We instruct the LLM to generate variations with an emphasis on the categories and quantities of objects, as they are common alignment issues with text-to-image models. However, we also utilize different sets of few-shot examples tailored to the type of input prompt to guide the LLM towards generating relevant variations. As an example, for the prompt “a red box on top of a blue box” in Table 14, the set of synthesized prompts includes variations such as “two blue balls on top of a pink box” and “a black triangle on top of a green box”, exploring alternative color attributes for the objects. We have revised the draft to add these details in Section 4.1.

---

[W2] Qualitatively comparable results.

Note that Figures 1 and 4 contain examples that further highlight the shortcomings of the baseline models, including BLIP-2. We also added Figure 8 with an additional example that better showcases the limitations of BLIP-2 in the revised draft. Given the variability in performance across prompts, we remark that it is important to supplement the qualitative results with a human evaluation, as we have done in the work.

---

[Q1] Details on the baseline models in human eval.

Thank you for pointing this out. To clarify, PickScore is used as the baseline for the composition and counting prompts, while ImageReward is the baseline for the comprehensive prompts. We have also revised our draft to include the details.

Dear Reviewer exWV,

We sincerely appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below. Please let us know if you have any further questions or concerns.

---

[W1] Are the prompts in the benchmark sufficient for comparing the reward models?

We believe our benchmark offers a sufficiently diverse collection of prompts for evaluating reward models across various aspects of text-image alignment. We introduced separate composition and counting sets to specifically assess the ability of the reward models to evaluate compositional generation, a known challenge for text-to-image models, particularly when involving multiple objects [1]. To increase prompt diversity, we added the comprehensive set, which contains 100 prompts spanning nine fine-grained semantic categories: creative, spatial, text rendering, style, counting, color, composition, location, and open-ended. As a future direction, we aim to expand the scope of our benchmark by incorporating additional prompts, images, and human labels.

---

[W2] Do the rankings in Figure 3 align with human assessments?

Figure 3 indeed summarizes the overall alignment between the reward models and human assessments, as measured by the expected calibration error (ECE) computed with the human labels from the benchmark. Therefore, it is reasonable to conclude that a reward model for which more of the prompts fall within lower ECE ranges is overall more aligned with human preferences.

---

[Q1] Are the models fine-tuned with different data in Section 5.2?

The same offline dataset of images was considered for all fine-tuned models. However, filtered samples for each reward model may vary depending on how the model evaluates them. This training approach was used for two reasons: (a) using synthetic images from generative models filtered for quality is a common practice for downstream task training [2,3,4], and (b) we empirically found fine-tuning on filtered samples to be significantly more effective than using the entire dataset, which may contain low-quality samples, for optimizing for the given reward.

---

[Q2] Failure cases of TextNorm

Our method employs a set of semantically contrastive prompts to enhance calibration by normalizing reward scores over these prompts. In case the set contains semantically equivalent or irrelevant prompts, normalization can result in worse calibration. Below are the results from an ablation experiment where we compare semantically contrastive prompts to random prompts in terms of calibration. This experiment underscores the importance of prompt set construction, highlighting that a poorly chosen set could yield worse results.

---

References

[1] Feng, Weixi, et al. “Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis.” ICLR 2023.

[2] He, Ruifei, et al. “Is synthetic data from generative models ready for image recognition?” ICLR 2023.

[3] Fan, Ying, et al. “DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models.” arXiv 2023.

[4] Schuhmann, Christoph, et al. “LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs.” arXiv 2021.

Dear Reviewer 9bKW,

We sincerely appreciate your review with thoughtful comments. We have carefully considered each of your questions and provide detailed responses below. Please let us know if you have any further questions or concerns.

---

[W1] Evaluating overoptimization

In our study, we consider human assessment of text-image alignment as the true objective and base our investigation on human annotations and evaluations. For our fine-tuning experiments, we evaluate the reward models against this true objective by conducting a human evaluation (the true objective) and report the results in Figure 6. The calibration analysis, which is also based on the human annotations from our benchmark, is for understanding the limitations of existing rewards and the associated risk of overoptimization.

---

[W2] Connection to early stopping

Thank you for your pointer. While early stopping is another viable approach (orthogonal to ours) to addressing overoptimization, it has several limitations. One challenge is determining the optimal stopping point, especially when evaluating the true objective is expensive. As you pointed out, the authors of AlignProp use early stopping in their experiments to guard against “loss of image-text alignment if optimized above a specific score” [1]. However, it is difficult to justify whether the 10th epoch is the optimal point at which to stop training, without conducting many human evaluations on multiple checkpoints. The authors of DDPO [2] recognize that overoptimization is an issue in text-to-image generation but do not discuss any specific methods for addressing it.

We believe that our method of enhancing reward model calibration through alignment with human assessment can be also integrated into early stopping to more accurately determine when to halt optimizing against proxy rewards. We have revised our draft to include a discussion comparing early stopping and our proposed method.

---

[W3] Evaluation of the HPS v2 reward model.

We evaluated HPS v2 on our benchmark based on calibration metrics and include the results in the table below.

HPS v2 is indeed a competitive baseline compared to the other reward models we consider in this work. However, TextNorm still outperforms HPS v2 on our benchmark. We have revised our draft to include the results in Appendix.

---

[Q1] Softmax CLIP objective vs. sigmoid CLIP objective.

While our proposed TextNorm (Eq. 1) has a notational similarity to the softmax-based CLIP objective, it is important to note that we are not claiming that the softmax-based objective is superior to the SigLIP objective [1] in terms of calibration. Our main focus is to evaluate the effectiveness of normalizing reward scores across contrastive prompts on improving calibration. The proposed method does not depend on the training objective of reward models. Nevertheless, we think evaluating SigLIP-based models is an interesting future question to explore.

---

[Q2] Use of semantically contrastive prompts.

The underlying rationale behind our prompt set design, which employs semantically contrastive prompts that are syntactically similar, is based on the following hypothesis: prompts $x$ that differ from the input prompt $x_0$ in both syntax and semantics are unlikely to generate high reward values $r_{\phi}(x, y)$. Random prompts serve as an example of such $x$. As we show in our response to [Q3] in this letter, the prompt set using our method outperforms a random set of prompts, supporting the validity of the hypothesis. We have clarified this point in the revised draft.