A Tailored Framework for Aligning Diffusion Models with Human Preference

Thank you for your positive feedback and valuable comments. We will try our best to answer all your concerns. Please let us know if you still have further concerns, so that we can further update the response ASAP.

Q1: About details of user study. "The details of user study are not disclosed and results are not carefully analyzed."

A1: Thank you. We would like to provide more details about the user study, and we have followed your suggestion to add these details in Appendix D of the revised manuscript.

• Participant: We collect feedback from five annotators in the original manuscript, and we include five more annotators during the rebuttal phase. All annotators acknowledge that their efforts will be used to evaluate the performance of different methods in this paper.
• Task instruction: The human annotators are given several triplets of ($c, x^{(a)}_1, x^{(b)}_0$), where $c$ is the text prompt and $x^{(a)}_1$ and $x^{(b)}_0$ represent the image generated by the model finetuned by method $a$ and method $b$, respectively. Then, the annotator is asked to compare the two images from the perspective of alignment, aesthetics, and visual pleasantness. If both images in a pair look very similar or are both unappealing, then the annotator should label “draw” for them. Otherwise, they label the "win" and "lose" tag for each image. In this way, for each pair of comparing methods, we have 225 triplets of ($c, x^{(a)}_1, x^{(b)}_0$) and each annotator label 225 "win/lose" or "draw" tags.
• Mitigation: In order to avoid user bias, we hide the source of $x^{(a)}_1$ and $x^{(b)}_0$ and randomly place their order to annotators.

---

Q2: About comparison with Diffusion-DPO. "fails to compare against state-of-the art offline methods such as Diffusion-DPO."

A2: Thank you. We have followed your suggestion to conduct a new experiment to compare TailorPO with Diffusion-DPO (Wallace et al., 2024). Diffusion-DPO finetuned SD-v1.5 on the Pick-a-Pic dataset in an offline manner. Therefore, we also finetune SD-v1.5 using TailorPO on prompts in the Pick-a-Pic training set and evaluate the performance using prompts in the Pick-a-Pic validation set. We use aesthetic scorer and ImageReward as the reward model, respectively.

The results of Diffusion-DPO in the above table are from (Liang et al., 2024). The above table shows that our methods achieve higher reward values than Diffusion-DPO in both aesthetic score and ImageReward score.

---

Q3: About performance on in-distribution prompts. "What are the performance on in-distribution prompts? Table 3 shows that TailorPO-G loses to TailorPO on HPSv2 ... Is this because TailorPO-G overfits to the training prompts?"

A3: I would like to confirm whether the "in-distribution prompts" refer to prompts used in training. If so, then Table 2 reports the performance on in-distribution prompts. Furthermore, we have conducted new experiments to finetune and evaluate SD-v1.5 on complex prompts in the Pick-a-Pic dataset. In this case, prompts used for evaluations are also in-distribution prompts.

On the other hand, In Table 3, the model is finetuned on 45 simple prompts of animals and tested on 500 complex prompts in the Pick-a-Pic dataset. In this case, the testing prompts contain many descriptions of objects, scenes, light, and style, and these descriptions are unseen in finetuning. Therefore, TailorPO-G may only strengthen the ability of the model to generate high-quality animal-related objects but cannot cover all these complex scenes. Nevertheless, both TailorPO and TailorPO-G have outperformed previous methods in most cases.

Q8: About the baseline for comparison. "The baseline for comparison seems relatively weak ... share results using the typical choice of 50 timesteps for DDIM or 25 timesteps for other advanced schedulers"

A8: Thank you for the suggestion. We have followed your suggestion to conduct a new experiment to compare TailorPO with more baselines. We measure the reward values of images generated by the base model using 50 and 100 DDIM timesteps and 50 timesteps for PDNM and DPM++ schedulers, respectively. We also fine-tune and evaluate the model using DDPO and D3PO with 50 timesteps for DDIM, respectively. The following table reports the results of the base model using different settings, as well as the results of our methods using 20 timesteps for DDIM. Our methods still outperform the strengthened baselines.

Q6: "provide insights into the differences between TailorPO and TailorPO-G"

A6: The key difference between TailorPO and TailorPO-G is that the gradient guidance better aligns the optimization direction with the reward model. We will elaborate on this by analyzing the gradient in TailorPO and TailorPO-G. In Eq.(11) of the original manuscript, we have shown that the gradient of the TarilorPO loss function can be written as follows.

$$\nabla_\theta \mathcal{L}(\theta) = -\mathbb{E}\left[(f_t/{\sigma^2_{t}})\cdot\nabla^T_\theta\mu_\theta(x_{t})(x^w_{t-1} - x^l_{t-1})\right]$$

For TarlorPO-G, the term $x^w_{t-1}$ is modified by adding the gradient term $\nabla_{x^w_{t-1}}\log p(r_{\text{high}}|x^w_{t-1})$. Therefore, we can derive its gradient term as follows.

$$\begin{array}{ll} \nabla\_\theta \mathcal{L}\_{TailorPO-G}(\theta) & = -\mathbb{E}\left[(f\_t/{\sigma^2\_{t}})\cdot\nabla^T\_\theta\mu\_\theta(x\_{t})((x^w\_{t-1} + \nabla\_{x^w\_{t-1}}\log p(r\_\text{high} | x^w\_{t-1}))- x^l\_{t-1})\right] \\ &= -\mathbb{E}\left[(f\_t/{\sigma^2\_{t}})\cdot\nabla^T\_\theta\mu\_\theta(x\_{t})(\underbrace{\nabla\_{x^w\_{t-1}}\log p(r\_\text{high} | x^w\_{t-1})}\_{\text{pushing towards high reward values}} + (x^w\_{t-1} - x^l\_{t-1})\right] \end{array}$$

The gradient term pushes the model towards the high-reward regions in the reward models. Therefore, TarlorPO-G further improves the effectiveness of TailorPO. Thank you for your insightful comment and we have added this discussion in Appendix B of the revised manuscript.

---

Q7: "I am concerned about the method's performance in real-world applications. Why do TailorPO and TailorPO-G lead to over-saturation issues in Figures 5 and 6?"

A7: To address your concern, we first evaluate the method's performance in real-world prompts. We design and sample several prompts from [1], and generate images using the model fine-tuned by our methods. Please refer to Figure 11 of the revised manuscript for visual demonstrations. These results show that on real-world prompts, the generated images are natural and have good quality, not exhibiting the over-saturation issue.

Second, we would like to discuss the over-saturation issue in Figure 5/6. This issue is caused by that the generative model is overoptimized to improve the score of the reward model on a given training set of prompts, but a high reward score may cause the generation distribution shift. We provide some examples in Figure 12 of the revised manuscript to demonstrate this phenomenon. For example, when taking JPEG compressibility as the reward model, DDPO, D3PO, and our methods all generate images with a blank background. In Figure 5 and Figure 6, the reward models are ImageReward and aesthetic scorer, which are trained on human preference rankings and potentially prefer images with bright colors (as shown in Figure 1 of (Xu et al., 2023)). In contrast, if we use other reward models (Figure 10), or use real-world prompts not in the training set (Figure 11), the over-saturation issue does not appear.

Nevertheless, the over-saturation issue in Figure 5 and Figure 6 is in an acceptable range. These figures are colorful and contain more details, but have no distortion. The user study in Figure 7 also validates that our method generates more preferred images.

[1] https://openai.com/index/dall-e-3/

Q4: "The framework may not be data-efficient, as it seems unable to leverage existing dataset including preference pairs of candidates across different conditions."

A4: This is a good question. Although our method cannot directly leverage existing datasets in an offline manner, the online learning of our method has additional advantages, including good performance and good generalization ability.

First, many studies [1-5] have shown that online strategies of learning algorithms significantly outperform their offline counterparts, while offline strategies face the challenges of OOD samples and gradient issues [4]. The following table also shows that our methods outperform the state-of-the-art offline method, Diffusion-DPO  (Wallace et al., 2024). The results of Diffusion-DPO in the following table are from (Liang et al., 2024).

Second, our method has good generalization ability. The online TailorPO is applicable to open-vocabulary scenarios, not limited by prompts in the dataset. In addition, Table 3 and Figure 8 have shown that our methods exhibit good generalization ability over different prompts. Besides, TailorPO-G incorporates the gradient guidance of reward models in the framework, which supports the injection of different conditions in training.

Finally, if we want to leverage an existing dataset for TailorPO, the most straightforward approach is to first train a reward model on the given dataset and then align the diffusion model towards the reward model. In fact, the training of the reward model is also effective. For example, [6] has found that a simple linear model on the top of CLIP ViT/14 is sufficient to produce satisfying results.

[1] Liu et al., Statistical Rejection Sampling Improves Preference Optimization. ICLR 2024.

[2] Xu et al., Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024.

[3] Xiong et al., Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint. arXiv: 2312.11456.

[4] Feng et al., Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective. arXiv: 2404.04626.

[5] Dong et al., RLHF Workflow: From Reward Modeling to Online RLHF. TMLR 2024.

[6] Schuhmann et al., LAION-5B: an open large-scale dataset for training next generation image-text models. NeurIPS 2022.

---

Q5: "reflect the preference selection procedure in Eq.10"

A5: Thank you for the kind suggestion. We did not explicitly include the preference selection procedure in Eq. (10) because Eq. (10) followed the classic formulation of DPO (Eq. (3)) and we thought it was easy to be understood by readers. On the other hand, the suggestion of "reflecting the preference selection procedure in Eq.10" is also insightful, and we rewrite the loss function in Eq. (10) as follows.

$\mathcal{L}(\theta) = -\mathbb{E}\_{(c,x\_{t},x^{(0)}\_{t-1}, x^{(1)}\_{t-1})} \left[\log \sigma \left((-1)^{\mathbb{1}(r\_t(c,x^{(0)}\_{t-1})<r\_t(c,x^{(1)}\_{t-1}))} \cdot \left[\beta\log \frac{\pi\_\theta(x^{(0)}\_{t-1}|x\_{t}, c)}{\pi_\text{ref}(x^{(0)}\_{t-1}|x\_{t}, c)} - \beta\log \frac{\pi\_\theta(x^{(1)}\_{t-1}|x\_{t}, c)}{\pi\_\text{ref}(x^{(1)}\_{t-1}|x\_{t}, c)} \right]\right)\right]$

where $\mathbb{1}(\cdot)$ is the indicator function. The term $(-1)^{\mathbb{1}(r\_t(c,x^{(0)}\_{t-1})<r\_t(c,x^{(0)}\_{t-1})}$ represents the step-level preference ranking procedure. We have added this form of the loss function in Appendix B of the revised manuscript for clarification.

Thank you for your valuable comments. We will try our best to answer all your concerns. Please let us know if you still have further concerns, so that we can further update the response ASAP.

Q1: About ablation study. "ablation study on the contribution of each component in TailorPO to its effectiveness."

A1: Thank you. We have followed your suggestion to conduct a new experiment on the contribution of each component in TailorPO and TailorPO-G. There are three key components: (1) step-level preference ranking, (2) the same input condition at each step, and (3) gradient guidance of reward models. Therefore, we fine-tune SD-v1.5 based on the aesthetic scorer using (1), (1)+(2), (1)+(2)+(3). Here we set the same random seed for a fair comparison, so the results of (1)+(2) and (1)+(2)+(3) are slightly different from Table 2 (where we averaged the results of three runs under different random seeds). The following table shows that all these components improve the aligning effectiveness. We have added this experiment to Appendix F of the revised manuscript.

---

Q2: About generalization on different fine-tuning methods and base models. "verification of generalization on fine-tuning methods, such as LoRA and full fine-tuning ... expanding evaluation to include a broader range of base models would strengthen the results."

A2: (1) Generalization on different fine-tuning methods. In this study, we use LoRA because almost all fine-tuning methods for aligning diffusion models used LoRA, including DPOK, DDPO, D3PO, SPO, and DenseReward. For a fair comparison, we also fine-tuned the model using LoRA. On the other hand, due to the limited resources, we are not able to fully fine-tune a diffusion model in an acceptable period of time. Nevertheless, we are positive about the generalization ability of our method on different fine-tuning methods, given that it has demonstrated effectiveness on LoRA.

(2) Generalization on different base models. We have followed your suggestion to conduct a new experiment on Stable Diffusion-v2.1-base (SD-v2.1-base, https://huggingface.co/stabilityai/stable-diffusion-2-1-base). We fine-tune SD-v2.1 on the set of animal-related prompts by taking the aesthetic scorer as the reward model, and then evaluate the model using the same prompts. After fine-tuning with TailorPO, we improve the aesthetic score of generated images from 5.95 to 6.21. In comparison, DDPO only reaches 6.02.

---

Q3: About user study. "Figure 5 indicates an evident over-saturation issue ... Conducting a user study could better substantiate this claim."

A3: Thank you. We have conducted a user study on these generation results as you requested, as stated in Lines 469-473 of the original manuscript (Lines 469-475 of the revised manuscript). Results in Figure 7 show that TailorPO and TailorPO-G receive higher preference than previous methods. Moreover, in the revised manuscript, we extend the user study and collect feedback from a total of ten human annotators. The result in Figure 7 shows that our method indeed generated human-preferred images.

On the other hand, the over-saturation issue in Figure 5 is caused by the overoptimization of the model towards the preference bias of the reward model. We provide a detailed discussion about this problem in the answer to your Q7. When we used other reward models, Figure 10 in Appendix E.4 of the revised manuscript demonstrates that the over-saturation issue does not appear.

Hi Authors,

Thanks for your response. I would like to point out that Figure 1(b), which is a main contribution of the submission, is almost identical to Fig. 3 (c) in SPO (Liang et al. 2024).

It is indicated the submission that "In contrast, we generate noisy samples from the same input xt and directly rank their preference order for optimization." This is the exactly the same as SPO.

This paradigm has been proposed in SPO. Indeed you cited SPO, and you know SPO very well. But you didn't discuss how pipeline drawn in Fig. 3(c) differs from SPO (In fact there is no difference). To me, this difference has been intentionally ignored.

I still hold strong reject and am skeptical about this paper's academic integrity.

Regards, Reviewer

Thank you for your review. Your overall reviews lie in the similarity between our method and SPO, as well as the doubts about the experimental results regarding SPO. We will now provide detailed responses to these questions.

First, the similarity between our study and SPO (Liang et al., 2024) only lies in that both of us observe the inconsistency in the preference order between the intermediate-step outputs and final generations, and both of us compare the preference of intermediate-step outputs to address this inconsistency, although we have different motivations and propose different solutions. To this end, we have clearly and respectfully cited SPO (Liang et al., 2024) and discussed our differences in Lines 126-130 and Lines 204-208, and we have compared our method with SPO in experiments, as shown in Table 2, Table 3, and Figure 7. For example, we have mentioned that "SPO (Liang et al., 2024) also pointed out the problematic assumption ..." "Liang et al. (2024) demonstrated that ..." Therefore, we have never deliberately ignored the contribution of SPO.

Second, beyond the above similarity, there are many differences between our study and SPO.

• The motivation is different. Our study is motivated by the theoretical discovery of the mismatch between the trajectory-level ranking and step-level optimization. In comparison, SPO observed the inconsistency between intermediate-step outputs and final outputs from visual demonstrations. Specifically, we conduct detailed theoretical analysis and identify the following two issues of existing training framework: (1) inaccurate preference order and (2) disturbed gradient direction. These theoretical analysis motivates us to propose a new training framework of DPO tailored for diffusion models.
• We use totally different method to tackle the inaccurate preference order issue. In SPO, the authors trained a step-wise reward model based on another assumption of the consistency between $x_t$ and images. In comparison, we do not train a new model for reward evaluation. Instead, we formulate the denoising process as MDP and utilize the value function as the measurement of step-level reward.
• We propose different method and implementation to address the gradient issue. First, we look back to the original formulation of DPO and ensure the same conditional input accordingly, which is very straightforward based on our theoretical analysis. Second, we notice that this operation potentially causes pairwise samples to be similar, so we introduce the gradient guidance of reward model in the training framework. This is one of the major contributions of this study, and it significantly improves the performance. In comparison, SPO chose to sample multiple outputs at each step for comparison.

Therefore, we are the first to explicitly derive the theoretical flaws of previous DPO implementations in diffusion models based on distinct characteristics of diffusion models. Moreover, we are the first to leverage the gradient guidance technique of diffusion models in preference aligning to enhance the performance.

Third, we obtain the results of SPO in Table 2 by running the officially released training code in https://github.com/RockeyCoss/SPO using 45 animal-related prompts and evaluating the model on these prompts, following DDPO and D3PO for a fair comparison. The difference in prompts causes the difference in the reported results. We have introduced the detailed experimental settings in Lines 404-414 of the original manuscript and we will emphasis this difference for clarification.

Furthermore, we also conduct a new experiment to compare the performance of TailorPO and SPO using the same prompts with SPO (Liang et al., 2024), i.e., 4k prompts in the pick-a-pick dataset. We finetune SD-v1.5 using 4k prompts in the Pick-a-Pic training set, and evaluate the performance on 500 prompts in the Pick-a-Pic validation set. Results in the following table show that our methods still outperform SPO.

Besides the difference in prompts, the implementation of SPO in our study has slightly difference from (Liang et al., 2024). Due to the limit of resources, we finetuned SD-v1.5 with a small batch size of 2 on one V100 GPU for 10k samples, while Liang et al. (2024) finetuned SD-v1.5 with a large batch size of 40 on 4$\times$ A100 GPUs for 40k samples.

Q7: About user study. "the inclusion of a brief statement on ethical considerations or participant consent would enhance transparency."

A7: Thank you. We would like to provide more details about the user study, and we have followed your suggestion to add these details and discuss the ethical considerations in Appendix D of the revised manuscript.

• Participant: We collect feedback from five annotators in the original manuscript, and we include five more annotators during the rebuttal phase. All annotators acknowledge that their efforts will be used to evaluate the performance of different methods in this paper.

• Task instruction: The human annotators are given several triplets of ($c, x^{(a)}_1, x^{(b)}_0$), where $c$ is the text prompt and $x^{(a)}_1$ and $x^{(b)}_0$ represent the image generated by the model finetuned by method $a$ and method $b$, respectively. Then, the annotator is asked to compare two images from the perspective of alignment, aesthetics, and visual pleasantness. If both images in a pair look very similar or are both unappealing, then they should label “draw” for them. Otherwise, they label the image with the "win" and "lose" tags. In this way, for each pair of comparing methods, we have 225 triplets of ($c, x^{(a)}_1, x^{(b)}_0$) and each annotator label 225 "win/lose" or "draw" tags.

• Mitigation: In order to avoid user bias, we hide the source of $x^{(a)}_1$ and $x^{(b)}_0$ and randomly place their order to annotators.

Q6: "would consider raising the rating (even above the upper bound) if the paper includes a more thorough formulation."

A6: Thank you for the insightful suggestion. We would like to provide a more thorough formulation of the value function and the diffusion framework. We have added the formulation of the value function in Section 3.2 of the revised manuscript. For the formulation of diffusion models based on EDM, due to the page limit and considering the readability of the paper, we add it to Appendix C of the revised manuscript.

Value function. By formulating the denoising process of the diffusion model as MDP in Eq. (6), we aim to maximize the action value function at time $t$, i.e., $Q(s,a)=\mathbb{E}[G_t|S_t=s,A_t=a]$, where $G_t$ represents the cumulative return at step $t$. We define  $G_t$ in the general form of $TD(\lambda)$, $G_t^{\lambda}=(1-\lambda)\sum_{n=1}^{T-t-1}\lambda^{n-1}G_t^{(n)}+\lambda^{T-t-1}G_t^{(T-t)}$, where $G_t^{(n)}=\sum_{i=1}^n \gamma^{i-1}R_{t+i}+\gamma^nV(S_{t+i})$ denotes the estimated return at step $t$ based on $n$ subsequent steps. Here, we simplify the analysis to $TD(1)$ and it degrades to Monte Carlo method. In other words, we have $G_t^\lambda=G_t^{(T-t)}=\sum_{i=1}^{T-t} \gamma^{i-1}R_{t+i}+\gamma^nV(S_{t+i})$. In the scenario of diffusion models, there is no intermediate feedback $R_t$ for intermediate steps and we assume $R_t=0$ for $t<T$. Therefore, the cumulative return can be further simplified as $\gamma^n V(S_{T})$. By setting $\gamma=1$ and $V(S_T)=R_T=r(c, x_0)$, which is the reward value of generated images, we have $Q(s,a)=\mathbb{E}[r(c,x_0)|S_t=(c,x_{T-t}, A_t=x_{T-t-1})]=\mathbb{E}[r(c,x_0)|c, x_{T-t-1}]$.

Diffusion framework. Diffusion models contain a forward process and a reverse denoising process. Given an input $x_0$ sampled from the real distribution $p_\text{data}$, the forward process can be formulated as follows (EDM, [1]), which is a uniform formulation for DDPM, DDIM, and other methods.

$$x_t=s_tx_0+s_t\sigma_t\epsilon$$

where $x_t$ is the noisy sample at timestep $t$. $s\_t$ represents a scale schedule coefficient and $\sigma_t$ represents a noise schedule coefficient. At timestep $t$, we have $p(x\_t|x\_0)\sim\mathcal{N}(s\_t x\_0, s\_t^2\sigma\_t^2 I)$. $s\_t$ and $\sigma\_t$ are usually selected to ensure the final output $x\_T$ follows a certain Gaussian distribution.

The reverse process aims to recover the distribution of original inputs $x\_0$ from a Gaussian noise $x\_T$. According to [1], the reverse ODE process is given as follows.

$$dx=[\frac{\dot{s}\_t}{s\_t}x-s\_t^2\dot{\sigma}\_t\sigma\_t\nabla\_x\log p(\frac{x}{s\_t};\sigma\_t)]dt$$

where $\dot{s\_t}$ and $\dot{\sigma}\_t$ denote the time derivative. $\nabla_x\log p(\frac{x}{s_t};\sigma_t)$ is the score function, which is usually approximated by a neural network, denoted by $s_\theta(\cdot)$. Replacing this term in the above equation, we can solve the reverse process. For a set of discrete timesteps, we can obtain a sequence $[x_T, x_{T-1}, \ldots, x_t, \ldots, x_0]$, and our study focuses on the optimization of $s_\theta(\cdot)$ at each timestep to generate $x_0$ with better image quality.

Subsequently, the predicted $\hat{x}\_0$ at the step $t$ can be represented $\hat{x}\_0(x\_t)=\frac{1}{s\_t}(x\_t+s\_t^2\sigma\_t^2 s\_{\theta}(x\_t))$. Then, the step-wise reward value of $x\_t$ can be estimated based on $\hat{x}\_0(x\_t)$. Similarly, the conditional score function used in our gradient guidance can be rewritten as $\nabla_x\log p (\frac{x}{s_t}|r_\text{high};\sigma_t)=\nabla_x\log p (\frac{x}{s_t};\sigma_t)+\nabla_x\log p (r_\text{high}|\frac{x}{s_t};\sigma_t)$. The first term is estimated by the neural network $s_\theta(\cdot)$, and the second term can be approximated following Eq.(13) of our paper.

[1] Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2022.

Q3: About novelty and soundness.

A3: Thank you for the comment. We would like to discuss the novelty issue here and answer your concerns about soundness in the corresponding questions (Q1-Q2 and Q4-Q7).

First, this study is not a simple A+B-styled work. Although we use a similar method with GAE [1] to introduce the value function for reward evaluation in aligning diffusion models, we are not simply combining these works. Instead, we have a complete analysis framework tailored for aligning diffusion models. Specifically, we start by rethinking the existing aligning framework of diffusion models and identifying the mismatch between the trajectory-level preference ranking and step-level optimization. This issue is substantiated by our theoretical analysis of the (1) inaccurate preference order and (2) disturbed gradient direction. Then, we propose methods to address these issues based on the theoretical analysis. Beyond introducing the value function for reward evaluation to address the inaccurate preference order, we revise the training framework of D3PO to align the gradient direction in optimization, and this is also a major contribution of this study. Moreover, we notice the potential impact of TailorPO on the similarity of paired samples, and novelly design TailorPO-G tailored for diffusion models to further improve the effectiveness. Finally, we conduct various experiments to demonstrate the effectiveness and generalization ability of our methods.

Second, this study has a distinctive contribution to the community of generative models, especially the alignment of generative models. We identify potential issues in the existing DPO-styled aligning framework and provide a new framework tailored for the diffusion pipeline. Our framework can also be extended to various scenarios including aligning the generation of videos and 3D objects based on diffusion models.

---

Q4: Typos and the descriptions. "The DPO method is NOT first proposed to fine-tune large language models to align with human preferences unless the "preferences" refer to preference datasets."

A4: Thank you for your careful review. We have followed your suggestions to correct the typo and clarify the description of DPO in Line 155: "The DPO method is originally proposed to fine-tune large language models to align with human preferences based on paired datasets."

---

Q5: "Can we use a trained policy to generate deterministically like probability flow ODE?"

A5: Thank you, but I do not fully understand this question, so I would like to first confirm it with you. You mentioned that "DDPO/D3PO-related methods cannot retain the Fokker-Planck equation of the original model," and I am not sure what component of the FK equation is broken by DDPO/D3PO-related methods. Furthermore, in my understanding, although DDPO and D3PO change the model parameter, the model is still learned to approximate the score function $\nabla_x \log p_t(x)$ in the probability flow ODE, constrained by a KL regularization term. Therefore, I am confused about which operation prevents the model from deterministically generating. Could you kindly explain more about this concern? I hope we can discuss more on this question together during the rebuttal period, so as to help us understand and address your concern.

Thank you for your insightful comments. We will try our best to answer all your concerns. Please let us know if you still have further concerns, so that we can further update the response ASAP.

Q1: About the estimation of $r_t$ during training. "whether $r_t$ can still be accurately estimated using either $\epsilon_\theta$ or $\epsilon_{\theta'}$ after the model parameter is updated from $\theta$ to $\theta'$... whether the optimization method remains effective after the parameter update, where $\theta'$ is significantly different from $\theta$ ..."

A1: This is a good question. During training, $r_t$ can still be estimated by Eq. (12). This is because the approximation of $r_t(c, x_t) \triangleq \mathbb{E}[r(c,x_0)|c, x_t] \approx r(c, \hat{x}_0(x_t))$ is derived based on (1) the proof of $\mathbb{E}[x_0|c,x_t]=\hat{x}_0(x_t)$ and (2) the estimation of $\mathbb{E}[r(c,x_0)|c, x_t]\approx r(c, \mathbb{E}[x_0|c,x_t])$ in Proposition 1. Both proofs are not limited to a specific parameter $\theta$ and can be extended to $\theta'$ after training.

First, the proof of $\mathbb{E}[x\_0|c,x\_t]=\hat{x}\_0(x\_t)$ is based on Tweedie’s formula, and the formula itself has no dependence on the model parameters. Specifically, Chung et al., (2023) provided the following proof in Appendix A of their paper. Given a noisy latent representation $x_t$, the conditional probability of $x_0$ can be written as $p(x\_0|x\_t)=p_0(x_t)\exp(x_0^TT(x_t)-\varphi(x_0))$, where $p\_0(x\_t)=\frac{1}{(2\pi (1-\bar{\alpha}\_t))^{d/2}} \exp(-\frac{\Vert x\_t \Vert^2}{2(1-\bar{\alpha}\_t)}), T(x\_t)=\frac{\sqrt{\bar{\alpha}\_t}}{1-\bar{\alpha}\_t}x\_t$, and $\varphi(x\_0)=\frac{\bar{\alpha}\_t \Vert x\_0 \Vert ^2}{2(1-\bar{\alpha}\_t)}$. According to this equation and Tweedie’s formula, we have $\mathbb{E}[x\_0|c,x\_t]=\frac{1}{\sqrt{\bar{\alpha}\_t}}(x\_t+(1-\bar{\alpha}\_t) \nabla\_{x\_t} \log p\_t (x\_t))$. To this end, both $\epsilon_{\theta}$ and $\epsilon_{\theta'}$ in the diffusion model represent an estimation for the term $\nabla_{x_t}p_t(x_t)$. Therefore, we can still estimate $\mathbb{E}[x_0|c,x_t]$ in the current model.

Second, the estimation of $\mathbb{E}[r(c,x_0)|c, x_t]\approx r(c, \mathbb{E}[x_0|c,x_t])$ in Proposition 1 is proven based on the Jensen gap upper bound [1] given the reward function $r(\cdot)$, and it is agnostic to the model parameter of the diffusion model.

[1] Gao et al., Bounds on the Jensen Gap, and Implications for Mean-Concentrated Distributions. arXiv:1712.05267.

---

Q2: About best quality. "... whether it can practically achieve this score. The authors should demonstrate how their method can achieve the best quality of DDPO/D3PO on ANY of the rewards."

A2: Thank you for the helpful suggestion. First, Figure 4 has shown that given the JPEG compressibility as the reward, out method can achieve the best quality of DDPO (>-10) within 4k paired samples, while DDPO needs more than 20k paired samples (according to Figure 4 of DDPO (Black et al., 2024)).

Second, we have followed your suggestion to conduct a new experiment, where we take the aesthetic scorer as the reward model to finetune SD-v1.5 using DDPO, D3PO, and TailorPO on 40k paired samples. We report the change of the reward scorer during the training process in Figure 9 of the revised manuscript (the fine-tuning of D3PO is still in progress and we will update the result as soon as possible). We observe three phenomena from this figure.

(1) TailorPO increases the aesthetic scorer the most effectively with less than 20k paired samples. This means that we can use fewer samples than other methods to achieve a good performance.

(2) Although DDPO reaches the highest aesthetic score at 40k samples, we observe the severe reward hacking problem with the generated images. We provide some examples in Figure 9.  All these images are unnatural with the same color, same style, and similar background (yellow leaves). Therefore, instead of fine-tuning the diffusion model with too many samples to achieve an extremely high reward score, we would suggest controlling the number of samples to strike a balance between the good image quality and a high reward score.

(3) D3PO is less effective than both DDPO and TailorPO, and this conclusion is consistent with Figure 3 of D3PO's original paper. This phenomenon also supports our discovery of its inherent issues about preference order and gradient direction.

Context:

• Regardless of whether it is implemented in DDIM, DDPM, or other schedulers, given a dataset $\mathcal{D}$, a trained diffusion model $\epsilon_\theta$ is trained to fit $\nabla_{x_t}\log p_t(x_t)$ of the dataset under some rescheduling (SDE by Song et al., EDM by Karras et al.).
• Theoretical assumption: I assume the base model $\epsilon_\theta$ perfectly implements $\nabla_{x_t}\log p_t(x_t)$.
• The evolution of $p_t(x_t)$ can be derived from the Fokker-Planck equation with the forward SDE. The form is easier if formulated in EDM, i.e., as a mixture of Gaussians.

I acknowledge the author's effort in the additional formulation of EDM and TD($\lambda$). However, I was expecting the author to fundamentally formulate within this framework (starting from Equation 1). The current presentation in Appendix E appears to be a redundant add-on. Since I understand the required work is formidable, I will not decrease my rating without including them.

My main concern remains an unsound technical flaw:

1. I believe that $r_t(c, x_t) \approx r(c, \hat{x}(x_t))$ if you implement $\hat{x}(x_t)$ using $\epsilon_\theta$.
2. I'm not convinced that you use $\epsilon_{\theta'}$ to implement $r_t(c, x_t) \approx r(c, \hat{x}(x_t))$. Specifically, "To this end, both $\epsilon_{\theta}$ and $\epsilon_{\theta'}$ in the diffusion model represent an estimation for the term $\nabla_{x_t}\log p_t(x_t)$" (I assume $\nabla_{x_t}p_t(x_t) \to \nabla_{x_t}\log p_t(x_t)$ is a typo). Once $\theta$ is modified to $\theta'$, it does not perfectly implement $\nabla_{x_t}\log p_t(x_t)$.
3. As explained by my theoretical argument, I've already predicted that this method will be effective only at the beginning of the training. Figure 9 further fortifies my confidence in this technical flaw.

Q3: About limitations. "could provide a more detailed discussion on the limitations of TailorPO," "could the authors discuss these limitations and potential avenues for improvement?"

A3: Like other methods based on an explicit pre-trained reward model, including DDPO, D3PO, and SPO, TailorPO has the potential of being prone to reward hacking [1], if we fine-tune the model on very simple prompts for too many iterations. It means that the generative model is overoptimized to improve the score of the reward model but fails to maintain the original output distribution of natural images. We provide some examples in Figure 12 of the revised manuscript to demonstrate this phenomenon. For example, when we take JPEG compressibility as the reward, DDPO, D3PO, and our methods all generate images with a blank background.

The problem of reward hacking is related to the quality of reward models. Given the fact that these pre-trained reward models are usually trained on a finite training set, they cannot perfectly fit the human preference for natural and visually pleasing images. Therefore, the optimization of generative models towards these reward models may lead to an unnatural distribution of images.

In order to alleviate the reward hacking problem, TailorPO can be further improved from the following perspectives.

• Using a better reward model that well captures the distribution of natural and visually pleasing images. A better reward model can avoid guiding model optimization towards unnatural images.
• Utilizing the ensemble of multiple reward models to alleviate the bias of a single reward model. While each single reward model has its own preference bias, considering multiple reward models altogether may be able to alleviate the risk of falling into a single model. To this end, [2] has shown that the reward model ensembles can effectively address the reward hacking in RLHF-based fine-tuning of language models. Therefore, we are hopeful that the reward model ensembles are also effective for diffusion models.
• Searching for a better setting of the hyperparameter $\beta$ in the loss function to strike a balance between natural images and high reward scores. In DPO-style methods, the coefficient $\beta$ controls the deviation from the original generative distribution (the KL regularization). In this way, we can search for a better value of $\beta$ to avoid the model being fine-tuned far away from the original base model. For example, [3] has provided a method to dynamically adjust the value of $\beta$.

We have followed your suggestion to add all these discussions in Appendix G of the revised manuscript.

[1] Skalse et al., Defining and Characterizing Reward Hacking. NeurIPS 2022.

[2] Coste et al., Reward Model Ensembles Help Mitigate Overoptimization. ICLR 2024.

[3] Wu et al., $\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$. NeurIPS 2024.

Thank you for your valuable comments. We will try our best to answer all your concerns. Please let us know if you still have further concerns, so that we can further update the response ASAP.

Q1: About user study. "A larger and more diverse user base would provide more robust evidence of the framework's effectiveness."

A1: Thank you for the valuable comments. We have followed your suggestion to conduct a broader user study by including more users. Beyond five users included in our original manuscript, we ask another five users to compare the preference of images generated by different methods. Following the settings in Lines 469-475 and Appendix D, we collect feedback from five more users and report the results obtained from a total of 10 users as follows.

The result shows that TailorPO and TailorPO-G better align the model generations with human preference. We have accordingly revised the description and the result of the user study in Figure 7 in the revised manuscript.

---

Q2: About comparison with SOTA methods. "While the paper compares TailorPO with other DPO-style methods, it would be strengthened by including comparisons with the current state-of-the-art methods." "How does TailorPO perform compared to the current state-of-the-art methods ..."

A2: Thank you. First, this study aims to align a base generative model with human preference via fine-tuning, instead of training a new base model, so we compared our methods with existing fine-tuning methods. Second, we have actually included both PPO-style method (DDPO) and DPO-style methods (D3PO and SPO) for comparison. These methods are all proposed in 2023 and 2024, and they represent the current state-of-the-art methods in this direction. Table 2 and Figure 3 have shown that our methods achieved higher reward values than these methods.

Third, we have followed your suggestion to conduct a new experiment to compare TailorPO with the state-of-the-art offline method, Diffusion-DPO (Wallace et al., 2024). Diffusion-DPO finetuned SD-v1.5 on the Pick-a-Pic dataset in an offline manner. Therefore, we also finetune SD-v1.5 using TailorPO on prompts in the Pick-a-Pic training set, taking the aesthetic scorer and ImageReward as the reward model, respectively. Then, we evaluate the performance using 500 prompts in the Pick-a-Pic validation set, as done by (Liang et al., 2024). The results of Diffusion-DPO in the following table are from (Liang et al., 2024).

The above table shows that our methods achieve higher reward values than Diffusion-DPO in both aesthetic score and ImageReward score. We have added this experiment in Appendix E.1 of the revised manuscript.

Dear Reviewer,

We sincerely appreciate your efforts and insightful comments, which has helped us to improve our paper. We have followed your suggestions to conduct new experiments and to discuss the limitations in our response. We would like to confirm whether our response has addressed all your concerns. Please let us know if you have any further questions, and we will respond as soon as possible.

Best regards, Authors