Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback

We sincerely thank the reviewer for the valuable feedback. Below, we address each of the questions in detail. As the concerns raised in the weaknesses section are directly reflected in these questions, we focus our response on answering the questions provided.

• Q1

  As mentioned in Strengths And Weaknesses section, could the authors provide preliminary results on the generalizability of the DRO on larger size models, say SDXL if time permits.

  A1

  In response to a suggestion from reviewer NjRJ, we extended our experiments to a larger model and dataset, specifically SDXL and MJHQ-30k [A]. We also adopted the GenEval benchmark [B], as proposed by NjRJ, to more comprehensively assess model performance.

  We ranked MJHQ-30k samples using PickScore and selected the top 500 image-text pairs as expert demonstrations. Both SDXL + SFT and Diffusion-DRO were fine-tuned on this dataset. For reference, we also report results from the original SDXL model and Diffusion-DPO, the latter of which was fine-tuned on Pick-a-Pic v2. Following the GenEval protocol, each prompt was used to generate four images to ensure reliable evaluation.

  Method	Colors	Color Attr.	Single Object	Two Object	Counting	Position	Overall
  SDXL	87.23	23.00	99.69	78.03	43.75	11.75	57.24
  Diffusion-DPO	86.17	25.00	98.75	85.25	48.25	13.60	59.50
  SDXL + SFT	88.94	27.00	99.75	87.68	46.50	13.20	60.51
  Diffusion-DRO	90.43	27.00	100.00	87.88	50.00	12.00	61.22

  This demonstrates the method’s strong generalization ability to both a larger backbone (SDXL) and a higher-quality dataset (MJHQ-30k). Notably, even when fine-tuning on expert demonstrations in the form of image-text pairs alone, without explicit preference labels, Diffusion-DRO is able to outperform SFT. For example, it achieves a 1.49 percent gain in the "Colors" category and a 3.5 percent improvement in "Counting."

  [A] "Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation", arXiv.

  [B] "GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment." NeurIPS 2023.

  ---

• Q2

  It appears to me that the threshold is an crucial factor in preventing overfitting, could the authors provide some ablation studies on the behavior of Diffusion-DRO w.r.t different choices of m?

  A3

  We fully agree that the threshold $𝑚$ can play a critical role in preventing the model from "pushing too far." Below, we provide an ablation study evaluating Diffusion-DRO with different values of $m$ on the Pick-a-Pic v2 test set, keeping all other settings consistent with those in Section 4.

  Method	PickScore	Aesthetic	CLIP	IR
  Diffusion-DPO	21.31	6.35	0.3334	0.7912
  Diffusion-KTO	21.25	6.34	0.3296	0.7636
  SPIN-Diffusion	21.40	6.26	0.3305	0.5619
  Diffusion-DRO ($m=10^{0}$)	21.37	6.38	0.3290	0.7279
  Diffusion-DRO ($m=10^{-1}$)	21.39	6.40	0.3267	0.7076
  Diffusion-DRO ($m=10^{-2}$)	21.49	6.47	0.3268	0.7399
  Diffusion-DRO ($m=10^{-3}$ used in paper)	21.54	28.48	6.42	0.3390
  Diffusion-DRO ($m=10^{-4}$)	21.29	6.25	0.3431	0.7104
  Diffusion-DRO ($m=0$)	21.13	6.18	0.3426	0.5955

  From the results, we observe that when $m$ becomes extremely small, such as $10^{-4}$ or $0$, performance drops due to the "push too far" effect. The best performance is achieved at $m = 10^{-3}$, and while increasing $m$ further slightly reduces the scores, Diffusion-DRO still consistently outperforms both Diffusion-DPO and Diffusion-KTO across most metrics.

  We hope this ablation provides a clearer understanding of how the threshold affects model performance.

  ---

• Q3

  It seems a bit unnatural that the aesthetics focused scores are dropping as the training set size increases with both SFT and DRO, given that in table 1 SFT improves such scores for SD1.5. could the authors elaborate a bit more on this?

  A3

  Thank you for raising this point. We believe you are referring to the trend shown in Figure 3, but please feel free to correct us if we misunderstood.

  In this figure, the so-called training set is not a random subset of the full dataset. Instead, it is constructed by ranking the training data using the HPSv2 preference model and selecting the top K samples. As K increases, the average quality of the selected expert demonstrations gradually decreases. This leads to a drop in performance for both SFT and DRO, especially on metrics related to aesthetics.

  To avoid confusion, we will revise the x-axis label in the subfigures of Figure 3 from "Train Size" to "Top K" to better reflect that it refers to the size of the expert demonstration subset, ranked by preference quality.

We sincerely thank the reviewer for the careful and thoughtful review. Below, we respond to the concerns raised in both the weaknesses and questions sections.

• Q1

  Now, looking at how Diffusion-DRO achieves these two goals: equations (1) to (6) explain the basics of RL and reward functions, but equations (7) to (12) are quite obscure. My understanding is that the authors want there to exist an oracle image from the distribution $D(c)$, and the model-generated images should be as close as possible in quality to those from $D(c)$. However, if this were the whole story, wouldn’t simple supervised fine-tuning (SFT) suffice?

  A1

  From a reinforcement learning perspective, inverse reinforcement learning (IRL) offers a principled alternative to supervised fine-tuning (SFT), which is also known as behavior cloning in RL terminology. We summarize the key differences in the following table:

  Aspect	SFT	Inverse RL
  Objective	Imitate expert policy	Recover reward function
  Required Data	Expert text-image pairs	Expert text-image pairs
  Advantages	Simple to implement; no interaction required	Interpretable objectives; better generalization
  Disadvantages	Susceptible to compounding errors; limited extrapolation	Harder to optimize; sample inefficient

  In summary, we choose inverse RL over direct supervised fine-tuning in order to improve generalization. Rather than directly imitating expert demonstrations, our method learns a reward model that implicitly evaluates generation quality. This enables stronger learning signals and allows the policy to generalize beyond the specific modes captured in the demonstration data.

  ---

• Q2

  According to my shallow understanding, the SFT term encourages the model’s current noise prediction to match the ground truth noise, while the second term encourages the current model’s noise prediction to differ from that of its previous version, which helps ensure diversity. I think I understand how this addresses the limited diversity of offline datasets. However, I am still unsure whether this fully solves the problem of accurately estimating image probabilities. This seems to relate to the ranking formulation, but I am struggling to grasp how ranking is used here.

  A2

  The first term (SFT-style loss) indeed encourages accurate noise prediction, while the second term pushes the reward model to assign lower scores to policy-generated samples, helping to diversify and improve generation quality.

  The core of our contribution lies in the ranking-based formulation introduced in Eq. 7, which specifies that expert demonstrations should receive higher rewards than policy samples. This ranking constraint is enforced via a max-margin objective, allowing the reward model to learn preference-consistent scores without requiring exact image likelihoods. Unlike Diffusion-DPO, which uses a non-linear transformation $\log \sigma(\cdot)$ to convert rewards into classification logits (Eq. 14), our method maintains a linear relationship between expert and policy scores. This avoids the use of Jensen’s inequality, which introduces a relaxation gap in the optimization. As a result, our approach can more precisely optimize the reward margin, leading to stronger alignment with expert demonstrations.

  While we do not attempt to estimate exact image probabilities, which is known to be difficult in diffusion models, our ranking loss offers a more faithful training signal for learning sample quality preferences. This directly addresses the challenge of aligning generation quality with human-like judgments without relying on brittle probabilistic modeling.

  We hope this clarifies how ranking is directly utilized in our formulation to improve the fidelity of reward modeling.

  ---

• Q3

  Although the article says that the reward model can be removed, Algorithm 1 still needs to initialize a reward model.

  A3

  In conventional reinforcement learning from human feedback (RLHF), a reward model is typically trained on preference data first, and then used to fine-tune the generative model. This results in a two-stage training pipeline.

  In contrast, our method does not require pretraining a separate reward model. As shown in Eq. 10, our reward model and policy model are essentially equivalent. Therefore, we only need to train the policy model once, and the reward signal is implicitly defined through this joint formulation. This allows us to bypass the need for a standalone reward model and simplifies the training process into a single stage.

  ---

• Q4

  The ICML 2024 paper Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models treats the SFT dataset as oracle data, which is equivalent to the expert demonstrations in this context. In their approach, any content generated by the model is regarded as a negative example, while the SFT examples are treated as positive examples—similar to the second term in equation (13).

  A4

  While both SPIN [A] and SPIN-Diffusion [B] make use of oracle data, the notion of expert demonstrations in our work differs in a fundamental way. Specifically, our expert demonstrations are required to satisfy the max-margin condition in Eq. 7, meaning that their quality must be optimal with respect to the reward model.

  In contrast, SPIN and SPIN-Diffusion train a discriminator to distinguish whether a sample originates from the dataset or from the model. This is particularly evident in SPIN-Diffusion, where the discriminator must be updated frequently to maintain its effectiveness. As a result, their training procedure follows a multi-stage process.

  Although our final formulations may appear similar, the role and interpretation of expert demonstrations in our method are conceptually distinct from the discriminators used in SPIN-based approaches.

  [A] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, ICML 2024.

  [B] Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation, NeurIPS 2024.

We sincerely appreciate your thoughtful review. Below, we address each of your questions in detail. Lastly, we provide clarifications for several concerns mentioned in the weaknesses section that were not explicitly formulated as questions.

Reply to Questions

• Q1

  For line 102, the paper says that they make an assumption formulated in Eq. 4. I am curious when this assumption holds. It seems that Eq. 4 says that the reward of the image $x\_0$ given prompt $c$, depends on the entire diffusion process $x\_{1:T}$ that generates $x\_0$. This is a bit counterintuitive for me because I think whether an image $x\_0$ is good or bad only depends on itself, not its generation process. Could you help me ground this assumption a bit more?

  A1

  It is correct that human judgment typically depends only on the final image $\mathbf{x}\_0$ and not on the specific trajectory that produced it. However, in Eq. 4, the trajectory-level reward $R(\mathbf{x}\_{0:T}, \mathbf{c})$ is introduced as a modeling choice to enable more efficient and stable optimization. The key requirement is that the expected value of this trajectory reward under the diffusion process aligns with the scalar reward $r(\mathbf{x}\_0, \mathbf{c})$:

  $$r(\mathbf{x}\_0, \mathbf{c}) = \mathbb{E}\_{\mathbf{x}\_{1:T} \sim p(\cdot \mid \mathbf{x}\_0, \mathbf{c})} \left[ R(\mathbf{x}\_{0:T}, \mathbf{c}) \right].$$

  Importantly, this does not assume that human preferences depend on $\mathbf{x}\_{1:T}$; instead, it enables us to decompose the reward across the diffusion steps. This decomposition is beneficial for computational reasons, as it enables gradients to be computed at each denoising step, making the optimization more tractable than backpropagating through the entire diffusion process.

  Moreover, the formulation is flexible. If the learned $R$ chooses to ignore $\mathbf{x}\_{1:T}$ and focus solely on $\mathbf{x}\_0$, it naturally collapses to a standard reward function:

  $$\begin{aligned} r(\mathbf{x}\_0, \mathbf{c}) &= \mathbb{E}\_{\mathbf{x}\_{1:T}} \left[ R(\mathbf{x}\_{0:T}, \mathbf{c}) \right] \\\\ &= \mathbb{E}\_{\mathbf{x}\_{1:T}} \left[ R(\mathbf{x}\_0, \mathbf{c}) \right] \\\\ &= R(\mathbf{x}\_0, \mathbf{c}). \end{aligned}$$

  This formulation allows flexibility and lets the learning process determine whether information from the full trajectory is beneficial for modeling the reward.

• Q2

  For line 127, the authors say that they parameterize the trajectory reward as similar to Eq. 3. I wonder whether this is a choice made by the authors for math convenience, or perhaps there is some ground truth supporting this parameterization?

  A2

  The parameterization in Eq. 9 is inspired by the formulation used in DPO [A], where the reward is expressed as a log-ratio of policy and reference distributions. As shown in Theorem 1 of DPO, this parameterization can represent any reward function without loss of generality. Therefore, the reward model does not lose any modeling capability with this parameterization. Moreover, this parameterization closely resembles Eq. 3 in our paper, which enables a seamless transition to the closed-form expression for the optimal policy in Eq. 10.

  Thus, the parameterization serves both as a principled modeling choice and a practical tool that simplifies the derivation while preserving generality.

  [A] "Direct preference optimization: Your language model is secretly a reward model." NeurIPS 2023.

• Q3

  For lines 226-231, I think the text tries to explain why the proposed method, Diffusion-DRO, is outperformed by baselines regarding the CLIP scores. But I don't quite understand the explanations from lines 226-231. Is the reason that Diffusion-DRO only uses the expert demonstrations, a small part of the preference dataset, resulting in a bad text encoder?

  A3

  We would like to clarify that the text encoder used in SD 1.5 shares the same pretrained weights as the one used in CLIP. Since SD 1.5 was trained from scratch on a large dataset, the images it generates naturally align well with the CLIP text encoder, resulting in strong CLIP scores.

  When SD 1.5 is fine-tuned using only a small dataset and a relatively small batch size, the model may slightly deviate from the original distribution. This shift can lead to a mild drop in CLIP score, as the generated images become less aligned with the CLIP encoder.

  This phenomenon can also be observed in models fine-tuned from SD 1.5 w/ SFT. For example, in Table 3, Diffusion-DPO w/ SFT shows a lower CLIP score compared to the version using the official weights (Diffusion-DPO), even though other metrics improve.

Clarification of Weakness

• Weakness 1

  "One key concern of the proposed method is that it seems to lose the benefit of DPO methods that are designed to avoid first fitting a reward function and then finding a policy to optimize the reward. Instead, the proposed method in Alg.1 has to first do IRL (Alg.1 line 14) and then do policy optimization (Alg.1 line 16). It is very helpful that Sec.3.4 discusses the frequency of the two alternative steps for IRL and policy optimization. But I wonder whether the alternation between reward and policy learning results in increased computation overhead? Maybe some empirical evaluation of computation could be helpful to address this.

  Also, it is not quite clear to me how the $p\_\phi$ in Alg.1 line 16 is computed. It would be great if the authors could clarify this."

  Response to Weakness 1

  We appreciate the reviewer’s concern regarding the potential overhead of alternating between reward and policy optimization. We would like to clarify that in our framework, the policy model $p\_\theta$ is always identical to the reward model $p\_\phi$, as formally derived in Eq. 10. This means that every update to the reward model simultaneously updates the policy—no additional optimization step is required for the policy. Line 16 in Algorithm 1 simply copies the parameters ($p\_\theta \leftarrow p\_\phi$) and introduces no computational overhead, regardless of the value of $M$. In all of our experiments, we set $M = 1$ so that the policy and reward model remain synchronized at all times. We will add this detail explicitly in Appendix A.2 of the revised version.

  Please note that this design closely mirrors the efficiency advantage of DPO: by training only a single model (the reward model), the generation policy is optimized implicitly. Our proposed Diffusion-DRO thus retains the same single-model training benefit as DPO, despite being derived from an IRL perspective. We will revise the paper to make this explanation more explicit and appreciate the reviewer’s feedback on this point.

• Weakness 2

  Another concern is to find demonstration data. Preference data are largely available and easy for humans to provide. But expert demonstrations are hard to obtain. This paper proposes to use the top K images as the expert demonstrations, from a given preference dataset ranked by the preference labels. This makes sense, but introduces a hyperparameter K to tune.

  Response to Weakness 2

  The table below shows the original data used to produce Figure 3 in our paper. As noted in the main text, expert demonstrations are selected based on HPSv2, so we omit HPSv2 scores here.

  As shown, when $K$ ranges from 100 to 1000, Diffusion-DRO consistently outperforms the baselines across most metrics and performs comparably in the remaining cases. While our method introduces a hyperparameter $K$, we find that it performs robustly across a reasonably range of values. It is not necessary to fine-tune $K$ precisely to obtain strong performance.

  Method	PickScore	Aesthetic	CLIP	ImageReward
  Diffusion-DPO	21.31	6.35	0.3334	0.7912
  Diffusion-KTO	21.25	6.34	0.3296	0.7636
  SPIN-Diffusion	21.40	6.26	0.3305	0.5619
  Diffusion-DRO($K=100$)	21.55	6.46	0.3373	0.7876
  Diffusion-DRO($K=500$)	21.54	6.42	0.3390	0.8452
  Diffusion-DRO($K=1000$)	21.46	6.34	0.3409	0.8188
  Diffusion-DRO($K=5000$)	21.25	6.30	0.3439	0.7686

Thank you for your clarification and extra results! I really appreciate them!

I am still a bit concerned about the requirement of finding high-quality demonstration. On the one hand, it requires some metrics to find good demonstration data. On the other hand, depending on the selection metrics, we might need different amounts of demo data, making it hard to tune the K parameter. Moreoever, from the table provided in the author response, it seems that the improvement of Diffusion-DRO over Diffusion-DPO is quite small. It is unclear how significant that it in reality. I plan to keep the score now.

We sincerely appreciate your thoughtful review. Below, we address each of your questions in detail. Lastly, we provide clarifications for several concerns mentioned in the weaknesses section that were not explicitly formulated as questions.

Reply to Questions

• Q1

  What is the intuition behind Eq. 7? In line 122, the paper states that "the reward model aims to maximize the margin between the expert and the policy," but this seems inconsistent. If the expert trajectory (left-hand side) is fixed and the reward model is optimized using samples from the policy (right-hand side), wouldn’t this minimize the margin? Could the authors clarify this apparent contradiction?

  A1

  Thank you for the insightful question. We would like to clarify the underlying optimization dynamics. Eq. 7 is given as:

  $$\mathbb{E}\_{\mathbf{c}\sim\mathcal{C},\bar{\mathbf{x}}\_0\sim\mathcal{D}(\mathbf{c})}\Big[r\_{\phi}(\bar{\mathbf{x}}\_0,\mathbf{c})\Big]\ge\mathbb{E}\_{\mathbf{c}\sim\mathcal{C},\mathbf{x}\_0\sim p\_{\mathbf{\theta}}(\mathbf{x}\_0|\mathbf{c})}\Big[r\_{\phi}(\mathbf{x}\_0,\mathbf{c})\Big].$$

  The reward model $r\_\phi$ is trained to assign higher scores to expert demonstrations than to policy samples, thereby maximizing the margin. In contrast, the policy (updated via Eq. 10) implicitly attempts to close the gap by generating higher-reward samples. This creates a minimax dynamic similar to IRL.

  In short:

  • $r\_\phi$ maximizes the margin: $r(\bar{x}\_0) > r(x\_0)$,
  • $p\_\theta$ reduces the margin by improving sample quality,
  • The contradiction is resolved by understanding the alternating optimization roles.

  We will revise the text to make this dynamic more explicit.

  ---

• Q2

  Eq. 9 introduces a new parameterization of the trajectory reward model. What is the intuition behind this formulation? In prior work, similar parameterizations are usually derived from a definition of and training objective.

  A2

  We adopt the parameterization in Eq. 9 following the approach used in DPO [A]. As shown in Theorem 1 of DPO, this formulation can represent any reward function (up to an additive constant), ensuring no loss of generality. Importantly, our Eq. 9 mirrors the structure of Eq. 3, enabling a smooth derivation of the closed-form policy in Eq. 10. This consistency simplifies the optimization pipeline by making the reward model and policy model interchangeable under KL-regularized objectives. We believe this formulation balances theoretical soundness and practical convenience, facilitating both analysis and implementation.

  [A] "Direct preference optimization: Your language model is secretly a reward model." NeurIPS 2023.

  ---

• Q3

  In line 8 of Algorithm 1, how is the sample $x\_t \sim p\_\theta$ efficiently obtained? Does this require performing the full denoising process from step T to t? If so, how is this efficient during training?

  A3

  In our implementation, we explored two strategies to accelerate the sampling of $x\_t \sim p\_\theta$.

  We first experimented with maintaining a replay buffer that stores intermediate samples $x\_t$ for reuse. In each iteration, only a small portion of the buffer is updated, and full training batches are sampled from the buffer. However, we observed that when the policy is updated every step (i.e., $M = 1$ in line 15 of Algorithm 1), this strategy led to suboptimal performance because the buffer tends to accumulate outdated samples and negatively affects training. As a result, we did not adopt the replay buffer approach in our final implementation.

  Afterward, we considered using more efficient samplers, such as DDIM [B] and DPM-Solver [C]. As mentioned in Section 4 of the paper, we ultimately adopted DPM-Solver++ for faster sampling. Below, we present a comparison between training with the DDPM sampler and DPM-Solver++:

  Sampler	PickScore	Aesthetic	CLIP	ImageReward	Training Time
  DDPM(steps=50)	21.50	6.48	0.3259	0.7501	46.264 hours
  DPMSolver++(steps=20)	21.54	6.42	0.3390	0.8452	25.708 hours

  As shown in the table, DPM-Solver++ reduces training time by 44 percent while slightly improving performance across several metrics. We note that this setup was adopted without extensive hyperparameter tuning, suggesting that there is still considerable room for further improving training efficiency.

  [B] "Denoising diffusion implicit models." ICLR 2021.

  [C] "DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps." NeurIPS 2022.

Clarification of Weakness

• Weakness 1

  The experiments are limited to SD 1.5, while it is now standard to also evaluate on larger models like SDXL, as done in Diffusion-DPO/SPO. This is important as gains may diminish on stronger base models.

  Reply to Weakness 1.

  Thank you for pointing out this important limitation. In response to a suggestion from reviewer NjRJ, we extended our experiments to a larger model and dataset, specifically SDXL and MJHQ-30k [D]. We also adopted the GenEval benchmark [E], as proposed by NjRJ, to more comprehensively assess model performance.

  We ranked MJHQ-30k samples using PickScore and selected the top 500 image-text pairs as expert demonstrations. Both SDXL + SFT and Diffusion-DRO were fine-tuned on this dataset. For reference, we also report results from the original SDXL model and Diffusion-DPO, the latter of which was fine-tuned on Pick-a-Pic v2. Following the GenEval protocol, each prompt was used to generate four images to ensure reliable evaluation.

  Method	Colors	Color Attr.	Single Object	Two Object	Counting	Position	Overall
  SDXL	87.23	23.00	99.69	78.03	43.75	11.75	57.24
  Diffusion-DPO	86.17	25.00	98.75	85.25	48.25	13.60	59.50
  SDXL + SFT	88.94	27.00	99.75	87.68	46.50	13.20	60.51
  Diffusion-DRO	90.43	27.00	100.00	87.88	50.00	12.00	61.22

  This demonstrates the method’s strong generalization ability to both a larger backbone (SDXL) and a higher-quality dataset (MJHQ-30k). Notably, even when fine-tuning on expert demonstrations in the form of image-text pairs alone, without explicit preference labels, Diffusion-DRO is able to outperform SFT. For example, it achieves a 1.49 percent gain in the "Colors" category and a 3.5 percent improvement in "Counting."

  We hope this additional experiment directly addresses your concern and further demonstrates the scalability and effectiveness of our method in more advanced settings.

  [D] "Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation", arXiv.

  [E] "GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment." NeurIPS 2023.

Thank you for the detailed response and for providing new empirical results. I do appreciate the effort.

However, I still have concerns regarding the theoretical novelty and the underlying intuitions.

• In particular, W1 was not addressed: although the response refers to “W1,” it actually discusses W2 from my review.

• Regarding Q2, the statement that “our Eq. 9 mirrors the structure of Eq. 3” does not, in my view, constitute a strong theoretical contribution. For comparison, while Diffusion-DPO/SPO may share a similar training loss with DPO, the derivations are from the RLHF objective; therefore, simply mirroring an objective’s structure is not sufficient.

I encourage the authors to clarify the derivations and motivations more explicitly. It is still unclear which parts are assumed, which are derived, and which are structural analogies. Addressing this would likely require substantial revision, so at this stage, I lean toward rejection.

Nevertheless, I do appreciate the authors’ additional experiments and the effort put into the rebuttal.

Thank you for the detailed response.

I believe Eq. 1,2,3,a are all standard results from prior work.

As for Eq. (9), how do you verify this parameterization? Starting from Eq. 5 and following an argument analogous to Eq. (3), one has $R_\phi=\log \frac{p_\theta}{p_{ref}}$. However, you write $R_\phi=\log \frac{p_\phi}{p_{ref}}$ in Eq. 9 instead and subsequently conclude $p_\theta=p_\phi$ in the end? $p_\phi$ does not seem to arise naturally in this derivation.

Overall, I appreciate the authors' effort and the empirical results demonstrated in the rebuttal. However, I'm not convinced by the derivation and insights. I will keep my current rating.

Thank you for the clarification. My point is that, starting from Eq. 9 (which is the parameterization of $p_\phi$), $p_\phi=p_\theta$. Breaking this into multiple gradient steps does not change anything. Otherwise, why not use any other parameterization?

Nevertheless, I appreciate the authors' response and the empirical efforts of this work.

We sincerely thank you for your review. Below, we address each of your questions in detail. Several of these questions also overlap with the concerns raised in the weaknesses section, which we address jointly in our responses.

Reply to Questions

• Q1

  While the paper does an excellent job of distinguishing the proposed method from Diffusion-DPO, it might also be a good idea to explicitly have a conceptual comparison with SPIN-Diffusion (at least in the appendix) since it seems closely related as a method.

  A1

  We sincerely appreciate the reviewer’s suggestion. While our main comparison focuses on Diffusion-DPO, we agree that SPIN-Diffusion shares conceptual similarities and merits explicit discussion.

  The key distinction lies in the formulation and training paradigm:

  • Our method is grounded in a max-margin inverse reinforcement learning (IRL) objective (Eq. 7), which directly compares expert demonstrations against policy samples. This formulation leads to a closed-form expression (Eq. 10), where the optimal policy coincides with the reward model. As a result, the min-max optimization reduces to a single-stage minimization over the reward model.

  • SPIN-Diffusion, by contrast, adopts a discriminative training approach. It frames reward modeling as a binary classification task, i.e., distinguishing between expert and generated samples, and relies on alternating updates of the reward and policy models. This requires multi-stage training and careful synchronization to ensure stable optimization.

  We believe this conceptual distinction, i.e., IRL-based ranking vs. discriminative classification, captures the core difference between the two approaches. We will include a discussion of this comparison in the appendix to help clarify the conceptual differences more explicitly.

  ---

• Q2

  Lines 205-208 mention that there are 5 generations per prompt and the image with the median Pickscore value is selected. This seems like an interesting choice for evaluation: while seeds have a huge role to play, wouldn't it be a better idea to simply average the results over all generations (and then report std dev)?

  A2

  We follow the evaluation protocol adopted by Diffusion-KTO [A], which selects the image with the median Pickscore for pairwise comparisons in automated win rate evaluation (Appendix A.2 of [A]). This also allows us to use the selected images consistently in both automated evaluation and user studies. As a result, the automated win rates can be directly compared to user study results, which improves their reliability.

  It is worth noting that we have provided the mean and standard deviation of preference scores over all generated samples in Table 3 and Table 4 of Appendix C to address the concern about variability across generations. These results show that the propopsed Diffusion-DRO achieves the highest average preference scores across most metrics, with the only exception being a slight gap in CLIP score on the HPSv2 dataset compared to baseline methods.

  [A] "Aligning Diffusion Models by Optimizing Human Utility." NeurIPS 2024.

  ---

• Q3

  Lines 70-73 mentions that while KTO also decouples the positives and negatives, this could introduce semantic biases. However, even after reading the paper, it's not entirely clear why this cannot happen at all for the proposed method either. Could the authors clarify a bit more about this aspect?

  A3

  The concern about semantic bias is valid and worth clarifying further. In Diffusion-KTO, positive and negative samples are explicitly decoupled based on a fixed dataset. If the negative set disproportionately contains certain semantic categories (e.g., images of "cats"), the model may inadvertently learn to avoid generating those concepts, even if they are not inherently undesirable. This fixed split makes the model vulnerable to dataset-induced semantic skew.

  In contrast, our method generates negative samples online from the evolving policy. This has two advantages: (1) The negative samples naturally reflect the current distribution of the policy, which tends to be more diverse and balanced than a manually curated negative set. (2) The dynamic nature of online sampling reduces the chance of reinforcing static biases, since what constitutes a "negative" evolves over training.

  While no method can entirely eliminate semantic bias, our approach is specifically designed to reduce its impact by avoiding hard-coded negative sets. We will clarify this point in the revised version of the paper.

  ---

• Q4 + Weakness 1 + Weakness 2

  The choice of the expert demonstrations is also quite interesting: while Fig. 3 provides an analysis of the number of samples from the Pick-a-Pic dataset, I'm also curious about would having higher-quality images (e.g. the MJHQ-30k images) or captions (e.g. through recaptioning) as a tool to improve the results of the proposed method.

  A4

  We sincerely thank the reviewer for the thoughtful suggestions and for pointing us to valuable resources. Following your comments in both this question and the weaknesses section, we conducted additional experiments using the larger SDXL model fine-tuned on the higher-quality MJHQ-30k [B] dataset. We also adopted the GenEval benchmark [C] as you suggested, to more comprehensively evaluate the model's performance.

  We ranked MJHQ-30k samples using PickScore and selected the top 500 as expert demonstrations. Both SDXL + SFT and Diffusion-DRO were fine-tuned on this dataset. For reference, we also include the results of the original SDXL and Diffusion-DPO, the latter of which was fine-tuned on Pick-a-Pic v2. Following the GenEval protocol, each prompt was used to generate four images to ensure stable evaluation.

  Method	Colors	Color Attr.	Single Object	Two Object	Counting	Position	Overall
  SDXL	87.23	23.00	99.69	78.03	43.75	11.75	57.24
  Diffusion-DPO	86.17	25.00	98.75	85.25	48.25	13.60	59.50
  SDXL + SFT	88.94	27.00	99.75	87.68	46.50	13.20	60.51
  Diffusion-DRO	90.43	27.00	100.00	87.88	50.00	12.00	61.22

  This demonstrates the method’s strong generalization ability to both a larger backbone (SDXL) and a higher-quality dataset (MJHQ-30k). Notably, even when fine-tuning on expert demonstrations in the form of image-text pairs alone, without explicit preference labels, Diffusion-DRO is able to outperform SFT. For example, it achieves a 1.49 percent gain in the "Colors" category and a 3.5 percent improvement in "Counting."

  [B] "Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation", arXiv.

  [C] "GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment." NeurIPS 2023.

  ---

• Q5

  The paper might also want to acknowledge other works on ranking based DPO e.g. LiPO[a]

  [a] Liu et al. "LiPO: Listwise Preference Optimization through Learning-to-Rank", NAACL 2025

  A5

  We sincerely thank the reviewer for pointing out this relevant work. In our current setting, we treat expert demonstrations as binary preferences (i.e., preferred vs. not preferred) without modeling any internal ordering or relative ranking among them. Therefore, our formulation is more aligned with pairwise or margin-based optimization rather than listwise ranking, which is the focus of LiPO [a].

  That said, we agree that incorporating richer ranking information (such as listwise preferences or soft rankings) could offer valuable additional supervision for fine-tuning generative models. We will cite LiPO in the revised version and highlight it as a promising direction for future work that complements our current approach.