Scalable Ranked Preference Optimization for Text-to-Image Generation

We thank the reviewer for taking the time to provide a detailed review of our work. We address the concerns raised by the reviewer below:

Clarification of Reference Model: We thank the reviewer for pointing out the lack of clarity. We have updated the paper (Sec. 3.1) to describe this process in more detail. Briefly, in Reinforcement Learning from Human Feedback (RLHF), we aim to maximize a reward function $r(.)$, which in our case are models that rate how good a generated image is for a given text prompt. However, naively optimizing the reward can lead to collapse. Therefore, RLHF methods enforce a KL-regularization to the original base model from which we start the optimization process. This original model checkpoint, i.e the starting point, is referred to as the reference model, and the weight of the KL-regularization term controls how much we change from the original/reference model.

Effectiveness and Limitations of Ground Truth: In Eq. 5 (Eq. 6 in the updated paper), we have the model ($\epsilon_\theta$) that is being trained, the original reference model ($\epsilon_\text{ref}$), and the winning and losing sample. In our case, the winning and losing samples are decided based on off-the-shelf reward models. In earlier works, these samples are collected using human-labeling, i.e., user-collected data. Naturally, user-collected data can be noisy, and several off-the-shelf reward models have shown a better correlation to human judgment than a single user. Further, we decide the winning and losing sample by aggregating scores/votes from five reward models, making the "ground truth" fairly robust. The role of the reference (original) model in Eq. 5 is to help measure how much the denoising is improved for the model when compared to the reference/original model that we start from. The overall goal of DPO in diffusion models is to improve the denoising for the preferred (winning) samples while reducing the denoising for the less preferred (losing) samples when compared to the reference/original model.

Explanations behind Scalability: Our primary motivation behind this work was to be able to construct an effective preference dataset for enhancing text-to-image models. The biggest bottleneck in this process is collecting, verifying, and performing quality control of human annotations. We show that purely synthetic preferences from off-the-shelf reward models provide an effective alternate solution to the costly human-labeled datasets. This allows us to arbitrarily scale up the size of preference datasets for text-to-image models. We also show that our dataset Syn-Pic allows for far superior results than the existing Pick-a-Picv2 dataset even though it has 1/4th the images, showcasing its effectiveness. In this work, we use the same prompts set used in Pick-a-Picv2 dataset for a fair comparison. However, we can create a much larger preference dataset by increasing the size of the prompts set, with significantly lower annotation cost than human labeling (cheaper by 2-3 orders of magnitude). Therefore, we wish to see future work building upon our work and increasing the size and quality of future preference datasets.

Extensions to Videos/3D: The reviewer also makes an important point about extending this framework for text-to-{video, 3D} models. We would like to point out that recent reward models (e.g., VQAScore) have shown promise for aligning with human preferences for videos and 3D modalities. Therefore, our framework would be a promising option for curating preference datasets for enhancing video/3D models.

Comparison of training costs: We already discuss training cost for RanDPO in Sec. 4.2 (Discussion of Computation Cost) and Tab. 8 in Appendix. In terms of the training cost, the only overhead RankDPO has over the standard DPO objective is computing the DCG weights. In our case, this amounts to an argsort operation over four entries, which does not have a meaningful overhead. However, due to the improvements of our Syn-Pic dataset both in image and labeling quality, we require fewer iterations until convergence (since the dataset has 4x fewer images), while providing superior results on all downstream evaluations. We provide this comparison in Tab. 8 of the paper.

We thank the reviewer for taking the time to provide thoughtful feedback on our paper. We address the concerns raised by the reviewer below:

Novelty: We describe the novel aspects of our work below.

Synthetically Labeled Preference Dataset (Syn-Pic) As described in the introduction, the biggest bottleneck of existing DPO works (e.g., DiffusionDPO) is the lack of a high-quality preference dataset. Human annotation of preference datasets is both costly and cumbersome. Additionally, these datasets quickly become outdated due to the rapid advancements in T2I model capabilities. For instance, the Pick-A-Picv2 dataset, which was manually labelled, primarily contains 512-resolution images produced by variants of the SD1.5 model. This has already been surpassed by newer T2I models such as SDXL and SD3.

In this work, we propose replacing human labeling with reward model scores to quickly iterate on new preference datasets. This approach provides several benefits:

• No human labeling required: Avoids noisy labels and reduces costs significantly.
• Stronger Preference Signal: Allows for aggregating preferences from multiple models and generating ranked preference lists for the same prompt.
• Scalability: Enables the use of a much larger prompt set, as annotation costs are drastically reduced.

Ranking-based Preference Optimization (RankDPO): Earlier DPO works for T2I models only had access to pairwise comparisons. However, our Syn-Pic dataset provides ranked preference lists of images for each prompt. This enables us to propose an updated direct preference optimization method that incorporates the ranking objective into its formulation.

Extensive Experiments to Show Benefits of Syn-Pic and RankDPO:

• Syn-Pic + RankDPO improves T2I models: Syn-Pic combined with RankDPO significantly improves the performance of models such as SDXL and SD3 (see Tables 1, 2, 3). Even though the publicly available SD3 model has already undergone DPO training with a larger dataset, our proposed method further enhances its performance.
• Individual contributions yield non-trivial gains: Both components individually lead to notable improvements (see Table 4).
  • DPO(Pick-a-Picv2) vs DPO(5 rewards): Demonstrates the superiority of the Syn-Pic dataset over the Pick-a-Picv2 dataset.
  • DPO(5 rewards) vs RankDPO: Shows the improvements of RankDPO over DiffusionDPO. This gap widens when using the GenEval benchmark (see Table 6 in the updated paper).

Baseline Methods: We thank the reviewer for pointing this out. For Syn-Pic, we generate images from SDXL, SD3-Medium, Stable Cascade, and PixArt-Sigma. We have now updated Tables 1-3 to include results from all four models. We see the same trends as before : (a) SD3-Medium is the strongest model, and (b) consistent improvements are provided to SDXL and SD3-Medium by our method.

Quality/FID evaluation: We thank the reviewer for bringing up this aspect of evaluation. We predominantly focused on evaluating visual quality through the Q-Align aesthetic score model and user studies. Fig. 3 shows that users consistently prefer our model over SDXL/DPO-SDXL. However, we now also perform an analysis of the FID scores using the MJHQ-30K benchmark, by comparing the FID against a set of 30K high-quality images from Midjourney. Existing SDXL+Refiner model achieves an FID of $9.55$, while our model achieves an FID of $7.99$. We also provide the full table in Tab. 10 of the paper.

Benefits from the ranking objective: We agree that the biggest source of performance gains is from preference-tuning on our Syn-Pic dataset, a key contribution of the paper. However, we would point out that the increase in performance from our RankDPO is a significant improvement. We further evaluate this on GenEval where we continue to see consistent improvements for SDXL and SD3-Medium (Tab. 6 of the updated paper)

We thank the reviewer for their positive evaluation and recognition of the solid contributions of our work. Below, we address the questions raised by the reviewer:

Choice of Evaluation Benchmarks: As the reviewer indicates, there are several benchmarks for evaluating text-to-image models (e.g., GenEval, T2I-Compbench, GenAI Bench, ConceptMix, HRS-Bench, TIFA). While we would be keen on adding more evaluations to further strengthen our claims, our choice of evaluation benchmarks were driven by the following criteria:

a. Having several top-performing recent models already evaluated,

b. Focus on accurately evaluating prompt-following and visual quality with a variety of evaluation strategies such as using object detectors and visual question answering models.

Our choice of evaluation benchmarks (GenEval, DPG-Bench, T2I-Compbench) have been used very frequently in the recent state-of-the-art T2I models to judge their quality. It includes models such as DALL-E 3, SDXL, SD-3, PixArt, SANA, etc. Further, it also reduces our computational burden since previous works have already evaluated and scored their T2I model on these benchmarks.

Evaluation of Reward Models: The reviewer brings up the critical question of how we evaluate reward models in text-to-image generation, especially given their growing usage in preference optimization and evaluation. Typically, reward models at the time of introduction are benchmarked on a variety of small evaluation sets which have human preferences and therefore can be used to measure correlation with human judgment. For instance, HPSv2.1, ImageReward, MPS, and Pickscore have been evaluated on the HPD, ImageReward and Pick-a-Pic test sets where these models show good correlation with human judgment (in some cases even outperforming individual users). Similarly, VQAScore has also been evaluated on several benchmarks where not only does it show a high quality of measuring image-text alignment, but is also overall aligned with human preferences. Based on these experiments, we decided to choose these models, and opted for an ensemble strategy where we aggregate preferences from these five models to improve the robustness of our synthetically generated preferences.

We thank the reviewer for taking the time to provide detailed feedback on our work. We also thank them for appreciating our ranking-based preference datasets, and our experiments. Below, we address the questions from the reviewer.

Results of DPO on Syn-Pic: While Tab. 1,2,3 compare our preference-tuned SDXL and SD3-Medium models with other popular public models, in Tab. 4, all the methods are trained on the same data (i.e., Syn-Pic except the entry DPO (Pick-a-Picv2) which uses the Pick-a-Picv2 dataset). We see consistent gains from a) higher-quality images compared to Pick-a-Picv2 dataset, b) stronger and more consistent labeling through our reward model ensemble, and c) from the RankDPO objectives (last two lines).

MaPO with Ranking based Preferences: An important aspect of our RankDPO formulation is that it is directly applicable to any pairwise preference-optimization loss (e.g., MaPO). Therefore, we can derive a RankMaPO formulation by applying the DCG and gain weights to the MaPO (or any other pairwise preference) objective:

$$\Delta_{i,j} = |G_i - G_j| \cdot \left| \frac{1}{D(\tau(i))} - \frac{1}{D(\tau(j))} \right|.$$ $$\mathcal{L}{\text{RankGeneralized}}(\boldsymbol{\theta}) = -\mathbb{E}{(\boldsymbol{c}, \boldsymbol{x}^1, \boldsymbol{x}^2, \dots, \boldsymbol{x}^k) \sim \mathcal{D}, t \sim [0,T]} \left[ \sum_{i > j} \Delta_{i,j} \cdot \mathcal{L}_{\text{pairwise}}\left(s(\boldsymbol{x}^i, \boldsymbol{c}, t, \boldsymbol{\theta}), s(\boldsymbol{x}^j, \boldsymbol{c}, t, \boldsymbol{\theta})\right) \right].$$

Such a formulation allows us to extend other preference objectives, such as MaPO, to include ranking-based cues to improve the optimization. In the case of SPO, it only takes prompts as input (similar to the other reward fine-tuning approaches such as AlignProp and TextCraftor). Therefore, it cannot utilize the images or labels from our Syn-Pic dataset. We elaborate on the RankMaPO formulation in our updated paper (Section A.3) and hope that our ranked preference dataset can improve various diffusion preference optimization methods.

RankDPO on binary preferences: RankDPO could be applied to the binary preference setting where we have the winning and losing pair with ranks 1 and 2, respectively. On simplifying the RankDPO objective, we see that the DCG weights would be constant for the binary case. We would only be left with the gain weighting term, which we also show provides some improvements over the standard DPO formulation in Tab. 5 (second last row). However, in the Pick-a-Pic dataset, where we don't have reward model scores to construct the gain weights, our RankDPO formulation would entirely simplify into the standard DPO objective. We have a discussion about this in the updated paper (Sec A.1).