DSPO: Direct Score Preference Optimization for Diffusion Model Alignment

We sincerely thank the reviewer for the comments and suggestions. Please see our clarifications below:

Q1. The paper misses out on using MaPO [1] as a reasonable baseline even though it considers contemporary works like Diffusion KTO.

A1. Thanks for your suggestions! This is indeed a relevant paper, and we have cited it in our revised manuscript. We have included the baseline provided in the MaPO trained on the Pick-2-pic dataset (we report win rate by treating SDXL as the base model). Also, We fine-tune SDXL using our proposed DSPO method and get the following win rate (VS SDXL) results:

Note that we utilize a default guidance scale of 7.5 and 50 sampling steps for these two models with seed 0. The results above indicate that our models outperform MaPO across most reward models, with the exception of aesthetics. In the last column, we present the average results across different reward models, demonstrating that our model outperforms MaPO. This also demonstrates the effectiveness of our model.

Q2. Lack of experimental results on models like SDXL makes it unclear as to how scalable DSPO is and if it works for models other than SD v1.5.

A2. Thanks for your comments! We are keen to extend our methods to additional models. We have finished expanding our approach to SDXL, as you mentioned. Here are the corresponding results for win rate (SDXL as base model):

These results show that our models can still outperform baselines and verify the effectiveness of our model.

Q3. The ablations lack experiments on some of the design choices the authors make to arrive at the final objective of DSPO.

A3. Thanks for your suggestions! We conduct this ablation study on treating the pretrained diffusion model $p_{ref}$ and $p_{data}$ as the reference model to fine-tune SD15, which are denoted as DSPO-ref, DSPO. Here are the corresponding results for win rate (SD15 as the base model):

We observe that our method, which leverages the score function of the true data distribution, outperforms the approach that treats the original pretrained model as the reference model. This is because the quality of images generated by the original pretrained stable diffusion models is lower than that of the fine-tuning dataset. As a result, using the score function of the true data distribution can produce better outcomes.

Thank you very much for providing detailed responses to my comments. I appreciate it! I have updated my score as well.

Consider this comment as a consolidated response to all the comments you have made in response to my review.

I think the new results are very promising and definitely help to place DSPO as a more favorable alternative to other existing methods. Below are some suggestions in terms of editing your manuscript:

1. Since you have now gathered the comparative scores for MaPO, I suggest using them in the main results as well. IMO, this will help make DSPO an even stronger alternative.
2. Consider including the tables from ablation, deduplication experiments, and memory-wall-clock trade-off either in the main paper or the Appendix.
3. If you're using an existing codebase/library to perform your experiments, consider citing them appropriately.

I can notice that Diffusion DSPO is still memory intensive, which is where MaPO shines. To try to mitigate that problem, could you try to do LoRA fine-tuning experiment instead of full fine-tuning of the UNet? If this helps achieve results similar to full fine-tuning, then perhaps LoRA fine-tuning could be made the default choice. This might allow you to fine-tune even larger models like Stable Diffusion 3.

Thank you for your valuable suggestions and increasing your score! We sincerely appreciate the helpful feedback, which has been useful in further improving our paper. In response, we have made the following modifications in the revised paper:

• We put the table of SDXL compared with Mapo and also include Diffusion-DPO and SFT baselines in Table 2.
• The memory and wall-clock experiments are included in Table 3, with the corresponding analysis provided in Section 5.2. Additionally, the ablation study and deduplication experiments are detailed in Table 6 of Appendix E.1.
• Our code is based on the implementation of Diffusion-DPO and we have added the corresponding link in Appendix D.2.

For more details, please refer to the revised paper.

We completely agree with your suggestion that using LoRA can reduce memory usage and training time. To explore this, we conducted a quick experiment fine-tuning SD15 with LoRA and obtained the following results:

We have observed that fine-tuning with LoRA enhances performance, though it slightly falls short of achieving results similar to full parameter training. However, the performance gap between these two methods is minimal, making LoRA a viable approach for fine-tuning large models. Next, we plan to fine-tune DSPO on SD3 with LoRA to evaluate its performance as future work. Thank you again for your valuable advice!

We gratefully appreciate your time in reviewing our paper. We would like to clarify some misunderstandings regarding our approach.

Q1. In general, I find that many claims are too vague and ambiguous. When we examine the final loss of Diffusion-DPO and DSPO, how can we definitively say that one aligns with the pretrained loss of Stable Diffusion more clearly than the other?

A1. Thanks for your questions! The Diffusion-DPO loss differs from the pretraining loss used in Stable Diffusion models. Specifically, in the pretraining stage of diffusion models, the objective relies on denoising objectives or score matching objectives, as demonstrated in Eq (7) in [1], e.g. $\|\mathbf{s}\_{\theta}(\mathbf{x}(t), t)-\nabla_{\mathbf{x}(t)} \log p_{0 t}(\mathbf{x}(t) \mid \mathbf{x}(0))\|_2^2$. However, Diffusion-DPO takes a different approach by using a log-likelihood objective to train a BT-model classifier that distinguishes between preferred and dispreferred images as shown in Eq (7) of our paper. Therefore, the objective of a classification loss is different from score matching. Our proposed DSPO method, on the other hand, applies the score matching objective for fine-tuning, aligning with the pretraining loss approach. This method fine-tunes diffusion models to match human-preferred scores as shown in Eq (10), maintaining consistency with the original score matching objective in the pretraining stage.

Q2. Additionally, why is aligning the diffusion model with direct reward optimization or RL considered suboptimal due to a mismatch in pretraining and fine-tuning objectives? Is there any theoretical justification beyond the win rates?

A2. Thank you for your comments! We apologize for any confusion. In Figure 2, we empirically demonstrate the suboptimality of DPO-based algorithms for diffusion models. To further clarify this, we provide the following reasons why DPO-based algorithms for diffusion models are suboptimal compared to our method, DSPO:

• The actual loss of Diffusion-DPO, adapted from the LLM domain, acts as an upper bound on the direct preference optimization (DPO) loss, as shown in Eq. (12) of [1]. Consequently, optimizing this upper bound may lead to suboptimal results compared with the actual loss. And Diffusion-KTO builds upon Diffusion-DPO and retains similar drawbacks. In contrast, our DSPO is equivalent to optimizing the RL loss without relying on any such upper bound, as established in Theorem 1.

• DPO can be interpreted as the forward KL divergence between the learnable policy network and the optimal policy network, which may lead to suboptimal results, as shown in Figure 1, Theorem 3.2, and Theorem 3.3 in [2]. In contrast, DSPO optimizes the reverse KL divergence similar to RL objectives, as demonstrated in Eq. (15) and Eq. (16) in our paper. This approach encourages a mode-seeking policy, allowing it to capture key characteristics of the optimal policy more effectively and perform better than DPO, as discussed in [2].

• Empirical experiments demonstrate that DSPO also outperforms direct reward optimization methods on diffusion models.

Furthermore, DSPO introduces a novel perspective by designing a preference learning loss derived from the diffusion loss itself (score matching). Rather than simply adopting methods from the LLM domain, our approach aims to inspire fresh ideas and innovation within this field.

Q3. In my opinion, since the method is based on human preference, a human evaluation should be conducted to confirm whether it truly increases the reward aligned with human preference

A3. Thanks for your suggestions! Following the human evaluation methodology in Diffusion-DPO, we use prompt indices 200, 600, 1000, 1400, 1800, 2200, 2600, and 3000 from the HPSV2 dataset. Five labelers assess these prompts by selecting their preferred images. The results below compare our DSPO method with Diffusion-KTO, the state-of-the-art method (except from DSPO) as reported based on reward models. DSPO achieves a 67.5% win rate compared with Diffusion-KTO. This further verifies the effectiveness of our method.

Dear ICLR Reviewer wk6Q,

We greatly appreciate your time and the insightful comments provided during the review of our paper.

We have made extensive efforts to address all your questions, suggestions, and misunderstandings in the response and believe that they adequately address all your concerns. The reviewer's comments primarily focused on clarifying certain claims and experimental details. We have addressed these points in our response by providing detailed explanations, including clarifications on specific claims, experimental procedures, and the results of human evaluation experiments. We believe that the reviewer's insightful comments can be easily and effectively addressed in the final version.

With the public discussion phase ending soon, we would like to confirm whether there are any other clarifications they would like. We would be grateful if the reviewer could increase the score.

Many thanks for your time; we are extremely grateful.

Best regards,

The authors of “DSPO: Direct Score Preference Optimization for Diffusion Model Alignment”

I apologize for thelate reply. Thank you so much for your thorough rebuttal addressing my concerns.

Most of my concerns, especially regarding human evaluations and a few minor clarifications, have been resolved.

However, I still believe that if we focus solely on the final DPO loss, without considering the intermediate derivations, the process essentially predicts the noise better for the chosen sample compared to the reference model, and the opposite for the not-chosen sample. I don't see this as a critical mismatch with the pretraining loss. Additionally, why is it so important to have the same objective for pretraining and fine-tuning? But it is also true that incorporating score-matching loss is an interesting direction.

Taking these points into account, I will raise my score to 6. Once again, thank you for your hard work.

Thank you for the great summarization of our contributions, and we really appreciate your encouraging comments. Please see our responses below:

Q1. Figure 3 looks cramped; the equations are pitted against other visual elements, making them hard to read.

A1. Thank you for your valuable suggestions! We have updated the figure to enhance its clarity, and you can find the revised version in our updated paper.

Q2. In table 2, Diff.-DPO seems to have a higher score in the PickV2 dataset with the Pick Score Metric, but it’s not highlighted.

A2. Thank you for pointing this out! We have emphasized the Pick Score for PickV2 in the revised paper, and you can review the updates there.

Q3. The paper shows the equivalence between the proposed method and RLHF, but it seems like the evaluation does not contain a direct comparison with a model that has been fine-tuned with RLHF.

A3. Thanks for your insightful questions! Our paper focuses on preference learning for human alignment, utilizing only human preference data without reward models, such as pairs of preferred and non-preferred images [1]. Our model is fine-tuned on this offline dataset. In contrast, RLHF algorithms firstly require reward models and fine-tune their models by sampling from a policy network and utilizing reward models, operating under fundamentally different settings from our approach. This distinction makes it challenging to conduct a fair comparison between our DSPO and RLHF algorithms. Moreover, the optimization of the two stages in RLHF algorithms adds complexity to the process and results in significant training costs, even for larger datasets [1],[2],[3]. We have also included a state-of-the-art baseline for preference learning in diffusion models that shares the same setting as our algorithm. This has demonstrated the effectiveness of our proposed DSPO.

[1] Diffusion Model Alignment Using Direct Preference Optimization

[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model

[3] Aligning Diffusion Models by Optimizing Human Utility