Diffusion Preference Alignment via Relative Text-Image Contrast

We appreciate the comments from reviewer TQH8, we will address your concerns point by point:

• W1: We acknowledge the level of novelty may be perceived differently, yet, we believe generalizing RPO to Diffusion models is a significant contribution: as we suggest in Section 2, the multi-step sampling process of Diffusion Models is way more complicated than the next token prediction in LLM. We proposed an algorithm that aligns diffusion models at each step of the diffusion chain. Additionally, we designed contrastive weights for more complicated multi-modal data. It requires decent innovation to generalize a preference optimization algorithm from LLMs to T2I models.

• W2: We agree that the evaluation of style alignment was done on training data since there’s no existing metric to evaluate how well the model adapts to a certain style. In addition, our figures 4 and 8 utilizes out-of-domain prompts and the model performed well. We have considered picking an equal-sized test set to further validate the effectiveness, however, picking prompt-image pairs that fits the desired styles is too time consuming.

• Q1: Figure 1 is an example of our generation, please checkout Figure 3 for comparison with other algorithms

• Q2: Our preliminary experiments suggest that LoRA tuning also works, however, reproducing all experiments on SD1.5 and SDXL even with LoRA is very time consuming, we will leave this for future exploration.

• Q3: To evaluate the performance of an alignment algorithm, it is best to control all variables and focus on the algorithm itself. Stable Diffusion 1.5 and Stable Diffusion XL are the most popular open-source text-to-image models in medium and large scales, respectively. The experiments on these two models sufficiently demonstrate the effectiveness and generalizability of our algorithm. Adapting our algorithm to Flux1.0 and SD3 requires considerable effort, and we will explore this in future work.

• Q4: We have conducted quantitative evaluation in Table 1. The purpose of the example figures are for qualitative evaluation.

We thank reviewer 4f3P for your positive feedback and constructive suggestions, we will address your concerns point by point:

• W1: It is generally hard to train a reward model that fully capture the information of the styles, just like the human preference models may not fully reflect human preference. Evaluation with FID strikes a balance between effectiveness and computation costs.

• W2: Thanks for pointing out the typos, we will fix them in an updated version.

• For Q1 and Q2. We selected styles that maximize sample efficiency, as this process requires human evaluation of style transfer success. These three styles were chosen because they are relatively easy to transfer to most pick-a-pic subset images, have strong distribution shifts, and thus serve as appropriate tasks. The images were selected based on their correct style representation, as human preference is not the priority in this task.

• Q3: there is a distribution shift between the pre-trained data and our style-transferred data. It is standard practice to use SFT to close the distribution gap when the alignment goal is different from the pre-trained model, as mentioned in [1].

[1] Bram Wallace, Meihua Dang, Senthil Purushwalkam, Rafael Raifalov, Stefano Ermon, Linqi Zhou, Aaron Lou, Caiming Xiong, Nikhil Naik, Shafiq Joty, Diffusion Model Alignment Using Direct Preference Optimization, 2024

• Q4: Yes, the contrastive weighting scheme assigns higher weights to the pairs with higher similarity, the “prompt-image pairs with semantically related content mean” are prioritized in this scheme. Consequently, prompt-image pairs with identical prompts and those with semantically related content will be utilized.

We thank reviewer DP5H for your valuable feedbacks, we will address your concerns point by point:

• W1: We acknowledge that the level of novelty may be perceived differently; however, we believe that generalizing RPO to diffusion models is a significant contribution. As we discussed in Section 2, the multi-step sampling process of diffusion models is far more complicated than the next-token prediction in LLMs. We have proposed an algorithm that aligns diffusion models at each step of the diffusion chain. Additionally, we have designed contrastive weights for more complex multi-modal data. It requires significant innovation to generalize a preference optimization algorithm from LLMs to T2I models.

• W2: Yes, the style alignment task is motivated by and based on style transfer, which is mentioned in section 3. Our motivation is twofold: 1) to measure the performance of Direct Alignment Algorithms under preference distributions where winner and loser clusters have strong distribution shifts, and 2) to provide an economic metric for evaluating image preference learning methods that distinguishes between these methods more effectively. We believe that "re-branding" would twist our presentation, as we have already acknowledged our aim to achieve style transfer via alignment algorithms in line 321 and cited relevant literature. In this regard, the name "style alignment" articulates our goal clearly by specifying the methodological context.

• W3: We appreciate this constructive feedback, we agree that FID is an effective evaluation metric, we will consider adding the gram matrix into our next version due to time limit.

• W5: Again, We acknowledge that the level of novelty may be perceived differently. However, we will continue to improve the clarity of our presentation.

• Q1: The $f(x_{\text{prompt}}, y_{\text{image}})$ function is the clip encoder from openai/clip-vit-large-patch14