A Gradient Guidance Perspective on Stepwise Preference Optimization for Diffusion Models

Thank you for taking the time to review our submission and provide detailed feedback. We’ve addressed your comments below in the order they were raised.

---

Weaknesses (W)

[W-Q1.1] Lack of empirical justification for timestep bias problem raised

[W-A1.1] To clarify the effect of timestep bias, we conducted an additional empirical study. Specifically, we plotted the change in preference scores of the $𝑥_0$ predictions at each timestep before and after preference fine-tuning, for both SPO and GradSPO, using noisy images from the Win subset of the Pick-a-Pic v2 dataset on Stable Diffusion 1.5.

The results show that SPO significantly improves preference scores in early timesteps, but the improvements sharply decline in later steps, indicating an exponential bias toward early timesteps. In contrast, GradSPO exhibits a more gradual, linear decline, suggesting it distributes learning improvements more evenly across timesteps.

This supports the presence of timestep bias in SPO and highlights GradSPO's advantage. The corresponding plot and discussions will be included in the main paper.

[W-Q1.2] Lack of empirical results supporting improved training stability claims of GradSPO

[W-A1.2] We thank the reviewer for highlighting this point. To support our claim regarding GradSPO’s improved training stability, we will include a training reward convergence curve in the camera-ready version. This curve demonstrates that GradSPO consistently achieves higher training rewards compared to SPO, which we attribute to reduced gradient noise and more stable optimization. We will also revise the introduction and abstract to clarify our definition of 'training stability' and better align the narrative with the empirical evidence presented.

[W-Q2.1] Lack of ablation experiments isolating the weighting mechanism

[W-A2.1] In Section 4.4 (Figure 5) for our main paper, we provide an ablation study to isolate the weighting mechanism of GradSPO. Specifically, we introduce a simplified objective that incorporates only the new weighting mechanism to standard SPO (denoted as SPO+Simple), to better assess the individual contribution of the weighting mechanism. Our results show that the weighting mechanism contributes positively, leading to improved win rates over the baseline SPO on the SD 1.5 model.

[W-Q2.2] Lack of ablation experiments for gradient correction

[W-A2.2] An ablation study analyzing the effect of the gradient correction term is also provided in Section 4.4 (Figure 5) (denoted as SPO+Simple+Maxmin). It shows a clear performance gain over the simplified objective alone (SPO+Simple), demonstrating the contribution of each component to the overall effectiveness of GradSPO*.

*(note that GradSPO would be equivalent to SPO+Simple+Maxmin+EMA).

[W-Q3.1] The theoretical benefits of GradSPO are not matched by empirical gains.

[W-A3.1] We respectfully disagree with this assessment. GradSPO shows consistent improvements over SPO across all benchmarks. While the absolute gains (~0.2 across preference metrics) may seem modest, they are in fact meaningful, especially when compared to absolute gains of DPO to SPO on both SD 1.5 and SDXL.

These results indicate that GradSPO not only offers theoretical advantages but also translates them into empirical gains, reinforcing its effectiveness as a practical preference learning approach.

[W-Q3.2] The paper does not explore failure cases

[W-A3.2] To better understand the failure cases in terms of training of GradSPO, we vary the $\gamma$ parameter on SD1.5 on Pick-a-Pic v2 while keeping the rest fixed to the setting used in the main results.

Table A summarizes the effect of varying $\gamma$ on key metrics such as HSPv2, Pick Score, Aesthetic Score, and Image Reward. The results suggest that at higher guidance scales, increasing $\gamma$ introduces more variance due to noise, leading to worse convergence. Conversely, lower $\gamma$ values result in a weaker reward signal, preventing the model from optimizing effectively. We find that $\gamma = 0.5$ offers a good trade-off between stability and reward strength, and we adopt this setting in both SDXL and SD1.5 experiments.

Table A. Ablation study on reward guidance scale $\gamma$ for GradSPO on SD 1.5 for Pick-a-Pic v2.

Moreoever, one potential failure area worth exploring in future work is the reduction in diversity after alignment. Specifically, we observe that stronger alignment to the preference signal tends to reduce output diversity. To investigate this, we measured diversity scores across 10 randomly sampled prompts with 10 different seeds each, following the setup in [1].

The results show a consistent drop in diversity: SDXL scores 12.54, SPO drops to 10.32, and GradSPO further to 9.40. This mirrors patterns seen in preference learning for large language models [2], where optimization toward high-reward outputs often comes at the cost of diversity. Understanding and mitigating this trade-off remains an open research challenge.

[1] Ibarrola et al., Measuring Diversity in Co-creative Image Generation

[2] Cui et al., The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

We thank the reviewer for the insightful comments and careful evaluation. Our responses are organized according to the sequence of points raised in the review.

Weaknesses (W)

[W-Q1] addressed in [Q-Q2] below

[W-Q2] No visual side-by-side comparison with SPO.

[W-A2] Side-by-side visual comparisons with SPO are already included in Figure 4 of the main paper, as well as Figures 6–7 in the Appendix. These figures present qualitative comparisons between GradSPO, SPO, and other widely used preference learning methods to highlight their differences more clearly.

[W-Q3] addressed in [Q-Q6] and [Q-Q7] below

[W-Q4] addressed in [Q-Q3] below

---

Questions (Q)

[Q-Q1] The fact that the author did not have to implement "multi-step" SPO to make SDXL work is very interesting but under-explored in the paper.

[Q-A1] 'Multi-step' SPO was originally introduced to strengthen reward signals by simulating longer trajectories. However, in GradSPO, the use of noise reduction stabilizes gradient estimates, effectively reducing the need for such rollouts. This allows SDXL to achieve strong performance without the added complexity of multi-step simulation, highlighting noise reduction as a more effective alternative in this setting.

[Q-Q2] Lack of ablation on key hyperparameters.

[Q-A2] In Table A, we present ablation results of our GradSPO on SPO-specific hyperparameters, including guidance scale and timestep count for SD 1.5 on Pick-a-Pic v2 with all other hyperparameters fixed to the ones used in the main results.

We observe that increasing the guidance scale leads to a consistent drop in performance, while reducing it improves results. We hypothesize this is due to high CFG scales introducing oversaturation artifacts during sampling [1]. While high CFG can yield visually appealing images during inference, training with these settings amplifies artifacts, ultimately degrading generation quality.

We also examine the impact of increasing the number of sampling steps from 20 to 50. As shown in the table, this change yields no meaningful performance improvement, with results remaining comparable to those obtained at 20 steps. Additionally, it nearly doubles the training time due to the increased sampling overhead.

Table A. Ablation study on classifier-free guidance scale and number of timesteps in GradSPO on SD 1.5.

Additionally, in Table B, we conduct ablations on GradSPO specific hyperparameters, $\gamma$ and $\mu$, with all other settings fixed for SD 1.5 with results reported on Pick-a-Pic v2 (we set $\gamma=0.5$ when varying $\mu$ and $\mu=0.9$ when varying $\gamma$).

Table B. Ablation results on reward guidance scale $\gamma$ and EMA momentum $\mu$ for GradSPO on SD 1.5.

We find that the $\mu$ parameter is relatively robust, with only a slight drop in performance at lower values. This may be attributed to oversmoothing, where the EMA model updates too slowly. In contrast, modifying the $\gamma$ parameter in either direction results in noticeable performance degradation.

At higher guidance scales, increasing $\gamma$ amplifies noise, leading to higher gradient variance and poorer convergence. Conversely, lowering $\gamma$ weakens the reward signal, preventing the model from optimizing effectively. Overall, we find that $\gamma = 0.5$ provides a strong balance between reward strength and signal stability, and we adopt this value for both SDXL and SD 1.5 experiments.

[1] Sadat et al., Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models

[Q-Q3] Adding specific failure cases would be very interesting.

[W-A3] One potential failure area worth exploring in future work is the reduction in diversity after alignment. Specifically, we observe that stronger alignment to the preference signal tends to reduce output diversity. To investigate this, we measured diversity scores across 10 randomly sampled prompts with 10 different seeds each, following the setup in [1].

The results show a consistent drop in diversity: SDXL scores 12.54, SPO drops to 10.32, and GradSPO further to 9.40. This mirrors patterns seen in preference learning for large language models [2], where optimization toward high-reward outputs often comes at the cost of diversity. Understanding and mitigating this trade-off remains an open research challenge.

[1] Ibarrola et al., Measuring Diversity in Co-creative Image Generation

[2] Cui et al., The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

[Q-Q4] The authors should also consider evaluating on GenEval results.

[Q-A4] Table C. below presents the GenEval results for SDXL. From these results, we observe that GradSPO achieves notable improvements over SPO, not only in aesthetic quality but also in compositional generation, resulting in a higher overall score. However, similar to what was reported in the SPO paper, GradSPO still falls short of Diff-DPO in overall GenEval performance. This suggests that, similar to SPO, GradSPO tends to prioritize optimizing for aesthetic reward signals over learning fine-grained compositional alignment, despite outperforming Diff-DPO on all individual image reward metrics.

Table C. GenEval results for SDXL models.

[Q-Q5] A figure comparing GradSPO and SPO would be helpful as a quick reference in the introduction.

[Q-A5] We recognize that including such a comparison earlier in the paper could improve clarity and accessibility. For the camera-ready version, we will revise the introduction to feature a clearer visual reference as a quick overview of the improvements introduced by GradSPO.

[Q-Q6] Curation details for Figure 4

[Q-A6] We appreciate the reviewer’s careful analysis and thoughtful suggestion. We acknowledge that without a clearly defined sampling or scoring procedure, the presentation may give the impression of preferential selection.

To address this, we will revise Figure 4 in the camera-ready version to follow a more rigorous and transparent protocol. Specifically, we will sample multiple generations per model for each prompt, score them using consistent preference criteria (e.g., reward model scores), and display the top result per method. This should provide a more credible and balanced qualitative comparison. We will also clarify this selection process in the caption and main text.

[Q-Q7] Concerns regarding user study

[Q-A7] In our original user study, participants were shown image pairs (Image A and Image B) generated using the same prompt and initial noise, each corresponding to a different method. Participants were asked to choose the image they preferred. To mitigate presentation order bias, the assignment of methods to Image A/B was randomized for each comparison. The high win rate in the user study is due to the lack of a "tie" option, which may have inflated the win rate of GradSPO over SDXL by forcing a decision even when the quality was comparable.

To address this, we conducted an additional fine-grained user study with 100 prompt samples per method and 5 independent raters. In this follow-up, participants were allowed to select one of three options: Image A better, Image B better, or Tie. Additionally, similar to SPO, the preference was done over General Preference, Visual Appeal and Prompt Alignment (Table D and Table E).

Table D. User preference results: GradSPO vs SDXL.

Table E. User preference results: GradSPO vs SPO.

These results confirm that GradSPO maintains a strong preference margin over SDXL and SPO, even with the tie option included, further validating the perceptual improvements introduced by our method.

[Q-Q8] Non-transitivity of win rates and need for direct comparison with SPO

[Q-A8] To address the concern regarding the non-transitivity of win rates, we provide a direct comparison of each ablation variant against SPO using standard preference metrics in Table F. The results show a consistent upward trend as each component is added, reinforcing the value of the proposed design choices in GradSPO.

Table F. Preference metrics comparing ablation variants of GradSPO against SPO on SDXL. Gradual inclusion of key components (e.g., max-min guidance) leads to consistent improvements across all metrics.

---

Thank you for taking the time to review our paper and for offering detailed feedback. We address them below, following the order in which they appear across the review’s sections.

---

Weaknesses (W)

[W-Q1] The proposed function a(t) in GradSPO appears to be overly simplistic. It would be beneficial to explore alternative formulations, such as non-linear functions.

[W-A1] Table A compares uniform timestep weighting with alternative schemes, including the widely used min-SNR weighting [1], and P2 weighting [2] for SD 1.5 on Pick-a-Pic v2. Unlike in standard diffusion training, where schemes like min-SNR or P2 often outperform uniform weighting, we find that uniform weighting performs slightly better in most metrics.

While non-uniform weighting schemes [1,2] have seen wide use in diffusion model training, their use in preference learning remains limited; we find that most preference learning for text-to-image diffusion models uses uniform weighting. Although the two objectives are related, they differ in nature: standard diffusion training aims to match a target distribution, whereas diffusion preference learning focuses on maximizing the margin between winning and losing samples. This leads to different optimization dynamics, where uniform weighting may offer a better inductive bias for margin-based learning.

Table A. GradSPO with different weighting schemes for SD 1.5 on Pick-a-Pic v2.

[1] Hang et al., Efficient Diffusion Training via Min-SNR Weighting Strategy

[2] Choi et al., Perception Prioritized Training of Diffusion Models

[W-Q2] While Equations (10)–(12) offer a solid theoretical argument, the paper would benefit from additional empirical evidence or illustrative visualizations to more concretely demonstrate the time-step weighting bias issue inherent in SPO.

[W-A2] We will add the plot in the main text to provide clear empirical evidence for the timestep bias issue. To visualize its impact, we measure the difference in preference scores of $\mathbf{x}_0$ predictions at each timestep before and after preference finetuning, using noisy images from the Pick-a-Pic v2 win set (SD 1.5).

We observe that in SPO, the difference in preference scores is high at early timesteps but drops off sharply in later timesteps. This exponential decrease reflects a timestep bias, where the model allocates more capacity to optimizing preference scores at earlier denoising stages, while neglecting later ones. In contrast, GradSPO exhibits a more linear decrease, indicating more balanced improvements across all timesteps, including the later ones.

[W-Q3] It lacks evaluations on recent backbone models.

[W-A3] Thank you for the suggestion. We agree that evaluating on more recent architectures would provide a stronger justification on the generality of our method. However, due to the limited time and computational constraints during the rebuttal period, we were unable to conduct additional experiments on these more resource-intensive models.

That said, our experiments on SD 1.5 and SDXL, two widely adopted diffusion backbones, demonstrate consistent performance gains across multiple datasets and metrics, providing strong evidence of the effectiveness of GradSPO over prior preference learning methods.

---

Questions (Q)

[Q-Q1] addressed in [W-Q1]

[Q-Q2] Robustness checks under different settings.

[Q-A2] To properly analyze the performance and stability of GradSPO, we conduct ablations on key hyperparameters with all other settings fixed for SD 1.5 with results reported on Pick-a-Pic v2 (we set $\gamma=0.5$ when varying $\mu$ and $\mu=0.9$ when varying $\gamma$). Results are shown in Table B.

We find that the method is relatively robust to variations in the EMA momentum parameter $\mu$, with only a slight drop in performance at higher values. We attribute this to an oversmoothing effect, where the EMA model lags behind the current model and adapts too slowly.

In contrast, the reward guidance scale $\gamma$ has a stronger impact. Setting $\gamma$ too high increases the variance of the reward signal, especially in noisy regions, which makes convergence less stable. On the other hand, a very low $\gamma$ weakens the signal and prevents the model from learning meaningful preferences. We find $\gamma = 0.5$ to be a good balance between signal strength and noise, and use this value across both SDXL and SD 1.5 experiments.

Table B. Ablation study on EMA momentum $\mu$ and reward guidance scale $\gamma$ for GradSPO on SD 1.5 for Pick-a-Pic v2.

Thank you for your thoughtful review. We address each of your comments below, following their order in the original review.

---

Weaknesses (W)

[W-Q1] Weighting schemes other than the uniform timestep weighting

[W-A1] Table A compares uniform timestep weighting with alternative schemes, including the widely used min-SNR weighting [1], and P2 weighting [2] for SD 1.5 on Pick-a-Pic v2. Unlike in standard diffusion training, where schemes like min-SNR or P2 often outperform uniform weighting, we find that uniform weighting performs slightly better in most metrics.

While non-uniform weighting schemes [1,2] have seen wide use in diffusion model training, their use in preference learning remains limited; we find that most preference learning for text-to-image diffusion models uses uniform weighting. Although the two objectives are related, they differ in nature: standard diffusion training aims to match a target distribution, whereas diffusion preference learning focuses on maximizing the margin between winning and losing samples. This leads to different optimization dynamics, where uniform weighting may offer a better inductive bias for margin-based learning.

Table A. GradSPO with different weighting schemes for SD 1.5 on Pick-a-Pic v2.

[1] Hang et al., Efficient Diffusion Training via Min-SNR Weighting Strategy

[2] Choi et al., Perception Prioritized Training of Diffusion Models

[W-Q2] Impact of using components like EMA and noise reduction

[W-A2] We will include the training curve in the main paper to illustrate the impact of EMA and noise reduction. These components lead to ~2 times faster convergence and a higher plateau in average preference score (GradSPO: ~17.8 vs. SPO: ~17.4) during training on SD 1.5. We attribute the faster convergence and higher training reward to reduced noise, which makes it easier for the diffusion model to optimize toward higher rewards.

[W-Q3] Issue of memory and convergence speed issue for a step-aware model

[W-A3] Stepwise preference methods (SPO and our GradSPO) can introduce two sources of computational overhead compared to standard Diffusion DPO [1].

1. They require a separate step-aware preference model. In our case, we reuse the publicly available pretrained checkpoint from SPO for fairness, rather than training a new model.

2. Both SPO and GradSPO perform online sampling and labeling during training using the step-aware preference model, whereas standard Diffusion DPO relies on offline-labeled data (e.g., from human or AI annotators).

Despite these costs, stepwise methods converges significantly faster. Our full finetuning on SD1.5 completes in 24 hours on 4×A6000 GPUs, compared to 768 hours reported in the original Diffusion DPO paper on the same hardware.

[1] Wallace et al., Diffusion Model Alignment Using Direct Preference Optimization

[W-Q4] Dataset used for DPO&MaPO vs GradSPO

[W-A4] Pick-a-Pic v2 is an extended version of v1, preserving the same prompt quality while expanding the dataset size. Following SPO, GradSPO uses only the prompt dataset (no image supervision), selecting 4,000 prompts from Pick-a-Pic v1 and generates the images via online sampling. To ensure fairness, we compare against baselines using the same underlying prompt distribution. Although DPO and MaPO are trained on the larger v2 dataset, the prompt quality is equivalent to v1. Since our method, like SPO, depends only on prompt-level supervision, the comparison remains valid despite the size difference.

SPO uses online sampling and relies only on the prompt dataset (no image required), selecting a subset of 4,000 prompts from Pick-a-Pic v1.

For a fair comparison with SPO, we use the same prompt dataset that SPO was built on.

Since our method also depends solely on a small prompt dataset, the extended size of Pick-a-Pic v2 is still a valid comparison as they share the same prompt quality.

[W-Q5] Effect of stop-gradient (Eq. 10) is not clear

[W-A5] In Eq. 10, the training objective updates the current score model $s_\theta(x_t, c, t)$ by pulling it closer to the winning score $s^w_\theta(x_t, c, t)$, and pushing it away from the losing score $s^l_\theta(x_t, c, t)$. To ensure a stable learning signal, the targets are detached using stop-gradient: $sg(s^w_{\theta_0})$ and $sg(s^l_{\theta_0})$.

If the stop-gradient is not applied, the winning score would be updated during training to move toward the current score, collapsing $s^w_\theta \rightarrow s_\theta$, thereby weakening the supervision signal and destabilizing training.

---

Questions (Q)

[Q-Q1] Results on flow-based model

[Q-Q1] Thank you for the suggestion. We agree that evaluating on recent flow-based models such as Lumina2 [1] and SANA [2] would be useful. However, due to the limited time of the rebuttal period, we were unable to conduct these new expriments on additional architectures. We note, however, that our current results, based on widely adopted benchmarks and models in diffusion preference learning literature consistently show performance improvements, demonstrating the effectiveness of GradSPO within the established preference learning setting.

[1] Qin et al., Lumina-Image 2.0: A Unified and Efficient Image Generative Framework

[2] Xie et al., SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer

[Q-Q2] Mention of MaPO in the introduction

[Q-Q2] Thank you for the suggestion. We will include a mention of MaPO in the introduction in the camera-ready version.

[Q-Q3] Clarity of symbols throughout the text

[Q-A3] We acknowledge that some symbols, such as $x$ (image), $c$ (conditioning input, e.g., text prompt), and $T$ (total number of diffusion timesteps), are used without explicit definitions when first introduced. We agree that providing clear definitions would improve clarity for a broader audience. We will revise the paper to explicitly define all key symbols when first mentioned, especially in the Notation and Preliminaries section.

[Q-Q4] In equation 3 and elsewhere, the authors mention the use of a timestep-dependent weighting factor. Even though it was there in DDPM, it's not used much in practice. Could the authors comment on that?

[Q-A4] In DDPM, the timestep weighting function corresponds to the ELBO. This timestep weighting is useful when modeling likelihoods with VDM [1] and DDPM [2] achieving better likelihood estimation with an ELBO timestep weighting scheme in contrast to the uniform weighting scheme, however this weighting scheme has been shown to hurt image quality achieving lower FID on standard image benchmarks due to a timestep bias issue during training. As most image generation as well as our research is more concerned with image perceptual quality than accurate likelihood estimation, we opt for using uniform weighting instead of a timestep dependent weighting factor.

[1] Kingma et al., Variational Diffusion Models

[2] Ho et al., Denoising Diffusion Probabilistic Models

[Q-Q5] The existing reward models (like PickScore) are not typically timestep-aware. This work, however, relies on timestep-aware reward models. Could the authors comment on it a bit more?

[Q-A5] Our approach requires a timestep-aware reward model primarily because both SPO and GradSPO rely on dense supervision across intermediate denoising steps, rather than sparse supervision only at the final output. This is analogous to findings in LLM RL training, where dense token-level rewards have been shown to lead to better alignment than sparse response-level rewards [1,2,3].

In the diffusion setting, this means scoring not just the final (clean) image but also the intermediate, noisy ones during denoising. Existing reward models like PickScore are trained only on fully denoised images and therefore produce unreliable or uninformative outputs when applied to intermediate blurry samples. To enable effective preference learning across the full denoising trajectory, we use timestep-aware reward models specifically trained on such intermediate images.

[1] Yoon et al., TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback

[2] Zeng et al., Token-level Direct Preference Optimization

[3] Chan et al., Dense Reward for Free in Reinforcement Learning from Human Feedback

[Q-Q6] Equation 13 - can't we scale with the variance of z?

[Q-A6] While scaling with the variance of $z$ can reduce the variance of the estimator, it also unintentionally scales down its mean, $\sqrt{T} \, \beta_t \, \nabla_{x_t} r(x_t, t)$. This weakens the strength of the guidance signal. Our goal with the variance-reduced estimator in Eq. 13 is to reduce variance without altering the mean, so that the average reward signal strength remains consistent while the noise is suppressed. This ensures that the model receives a stable yet strong training signal. We will include this rationale in the main paper.

[Limitations Section Presentation]
We acknowledge the concern and agree that limitations should be clearly presented in the main paper. We will move the limitations section to the main paper to ensure better visibility.