Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences

Thank you for highly recognizing the value of our study and helpful feedback!

---

Q1: Why do reward models outperform large human preference datasets despite their simplicity?

A1: This is a great question! We believe reward models have the following advantages:

1、We note that PickScore (reward model) feedback is interpretable as pseudo-labeling the Pick-a-Pic dataset—a form of data cleaning [1] [2]. Specifically, the PickScore reward model serves not only as a proxy for human preferences but also as a mechanism for data refinement.

2、Granular Feedback vs. Binary Human Labels

Human preference datasets inherently provide binary labels (e.g., "A > B"), limiting their ability to represent nuanced preference distributions. In contrast, reward models assign continuous scores, enabling two key advantages:

• Smooth Supervision: Loss functions can adaptively weight samples based on reward differences,avoiding the brittleness of binary thresholds.

• Relative Calibration: Fine-grained rewards better capture perceptual similarity (e.g., distinguishing "slightly better" from "significantly better" cases), which is critical for guiding iterative model updates.

This explains why reward-guided training can outperform raw human datasets: it amplifies signal-to-noise ratios in human preferences while retaining their semantic intent. We emphasize that our approach complements (not replaces) human feedback, as reward models are trained on human-annotated data.

[1] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet. CVPR 2020

[2] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pretraining and self-training. NeurIPS 2020

---

Q2: The paper uses DDIM inversion in DPO training but lacks theoretical justification of its advantage over basic noise addition.

A2: Thank you for the valuable feedback! We first derive the minimal proof. Note that for Equ (5), Diffusion-DPO utilizes $q(x_{1:T}|x_{0})$ to approximate $p_{\theta}(x_{1:T}|x_{0})$. For each step, Diffusion-DPO used $q(x_{t-1,t}|x_{0})$ to approximate $p_{\theta}(x_{t-1,t}|x_{0})$. Suppose we replace standard noise injection at $x_{t-1}$ with a single-step DDIM inversion, when given $x_{0}$. Formally, we propose to use $q(x_{t-1}|x_{0})p_{\theta}(x_{t}|x_{t-1})$ for approximation. And this approximation yields lower error because $D_{KL}(q(x_{t-1}|x_{0})p_{\theta}(x_{t}|x_{t-1})||p_{\theta}(x_{t-1,t}|x_{0})) = D_{KL}(q(x_{t-1}|x_{0})||p_{\theta}(x_{t-1}|x_{0})) <D_{KL}(q(x_{t-1,t}|x_{0})||p_{\theta}(x_{t-1,t}|x_{0})).$

The complete derivation employing full inversion will be detailed in the paper.

---

Q3: Could DDIM inversion be generalized and applied as an standard alternative to noise injection in diffusion training?

A3: Yes, generalizable! We greatly appreciate the opportunity to discuss this point!

Currently, the training of large-scale diffusion models/flow matching primarily consists of two stages: pre-training and post-training. The pre-training stage focuses on establishing the trajectory from a noise distribution to a data distribution, while we argue that the post-training stage should specifically fine-tune variables that are highly correlated with the target images.

This idea shares some similarities with the Reflow method in Rectified Flow [3], which trains specialized (noise, image) pairs. However, a key distinction is that most real-world datasets contain only images (without paired noise). Thus, our motivation is to adaptively adjust image-dependent variables during post-training. We believe this approach could become a dominant paradigm in future post-training strategies, as it preserves the model’s inherited capabilities while enhancing its task-specific performance.

[3] Liu, Xingchao, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR2023

---

Q4: Please provide a clearer introduction and context for the compared baselines.

A4: Thank you for the suggestion! We fully agree that providing clearer motivation for method selection is critical for readability. Below, we outline our planned revisions to address this concern:

These baselines share a common focus on human-aligned image generation but differ fundamentally in their technical mechanisms: Diffusion-KTO adopts the Kahneman-Tversky [4] model to represent human utility instead of maximizing the log-likelihood of preferences. Margin-aware Preference Optimization (MaPO) jointly maximizes the likelihood margin between the preferred and dispreferred datasets and the likelihood of the preferred sets, simultaneously learning general stylistic features and preference without reference model.

[4] Tversky, A. and Kahneman, D. Advances in prospect theory: Cumulative representation of uncertainty.

We're truly grateful for your enthusiastic reception of our manuscript and your insightful feedback!

---

Q1： Table 7 caption can be improved.

A1： We sincerely appreciate your constructive feedback! We have implemented the following improvements:

• Caption Revision: We have revised the table caption to explicitly state: "Table 7: Impact of weight-to-sensitivity ratio using different reward models. We report the performance comparison where rows represent different training signals and columns denote evaluation metrics. Upward arrows (↑) in column headers indicate higher values are preferable."
• Visual Reinforcement: To further enhance clarity: We will include a brief structural description in the Results section (Section 5.4) when first referencing the table in "Choice of Reward Model" subsection.

---

Q2：Discuss the statistic significance of the gap in table 7, 8, 9.

A2: Thank you for this helpful suggestion!

To provide a comprehensive statistical evaluation of our approach, we present both win-rate analysis and effect size measurements.

• The win-rate results in Supp Table 1-2 demonstrate how frequently evaluators prefer SmPO generations over baseline methods, with values exceeding 50% (indicating statistical majority preference) highlighted in bold.

Supp Table 1: Win-rate comparison between SmPO-SDXL and baselines on Pick-a-Pic v2 test set.

Supp Table 2: Win-rate comparison between SmPO-SD1.5 and baselines on Pick-a-Pic v2 test set.

• Additionally, we compute Cohen's d to quantify the effect sizes between SmPO and baselines in Supp Table 3-6, following conventional interpretations: |d|<0.2 (small), 0.2≤|d|<0.5 (medium), 0.5≤|d|<0.8 (large), and |d|≥0.8 (very large). Cohen's d measures the standardized mean difference in standard deviation units, making it unit-free.

Supp Table 3: Cohen's d for comparison between SmPO-SDXL and baselines on Pick-a-Pic v2 test set.

Supp Table 4: Cohen's d for comparison between SmPO-SD1.5 and baselines on Pick-a-Pic v2 test set.

Supp Table 5: Cohen's d for comparison between SmPO-SDXL and baselines on HPD v2 test set.

Supp Table 6: Cohen's d for comparison between SmPO-SD1.5 and baselines on HPD v2 test set.

Thank you for your feedback and we'll do our utmost to resolve your concerns.

---

Q1: For evaluation, Image Reward, and Aesthetic score could also be considered.

A1: We have incorporated both metrics: Image Reward and Aesthetic Score are reported in Table 2,8 and 9 (Quantitive comparison), Table 3-7 (Ablation studies) and discussed in Line 262-263 of Section 5.1.

---

Q2: There is no formal proof in this paper.

A2: The detailed derivation of SmPO loss is provided in Appendix B.

---

Q3: The ablation study on hyperparameters are not comprehensive.

A3: Our ablation studies on hyperparameters were systematically designed along five key dimensions to validate our method’s robustness:

• Component Effectiveness: Table 3 shows each module's contribution through progressive ablation.
• DDIM Inversion Steps: Table 4 shows step selection's impact on output quality and training efficiency.
• $(\gamma,\beta)$ combinations : Table 5 shows hyperparameter interaction effects.
• CFG in Inversion: Table 6 shows CFG's influence during DDIM inversion.
• Weight-to-Sensitivity Ratio $\frac{\alpha}{\gamma}$: Table 7 shows reward-model derived calculations.

---

Q4: Lack of comparison and discussion with many Diffusion DPO variants.

A4: Thank you for your feedback. We've compared our method with key baselines: SFT, Diffusion-DPO, its variants (Diffusion-KTO and MaPO). Additional SPO comparison are included (see response A3 to Reviewer UNNR). To our knowledge, these are the most relevant and publicly available baselines. We welcome suggestions for additional baselines with references.

---

Q5: Label smoothing has been proposed by the DPO authors in 2023, Conservative DPO (cDPO).

A5: Thank you for your feedback. In fact, cDPO differs fundamentally from our proposed SmPO.

1. cDPO focuses on noisy labels (where labels may be flipped with some probability) and applies a linear weighting of swapped DPO losses: $L_{cDPO}(x_0^{w},x_0^{l})=(1-\epsilon)L_{DPO}(x_0^{w},x_0^{l})+\epsilon L_{DPO}(x_0^{l},x_0^{w})$. Our SmPO assigns fine-grained labels to each image pair and incorporates them into the DPO loss function through distribution averaging, yielding Equ (9).
2. In cDPO, $\epsilon$ is a is a fixed manually-tuned hyperparameter (refer to Line 50 and 84 of [1]), whereas our SmPO introduces reward-adaptive smoothing - zeroing loss for similar pairs, amplifying gradients for dissimilar ones. In other words, distinct image pairs are assigned different weights. As such, in Equ (12), $\frac{\alpha}{\gamma}$ could be more precisely represented as $\frac{\alpha}{\gamma}(x_{0}^{w},x_{0}^{l})$.

[1] https://github.com/eric-mitchell/direct-preference-optimization/blob/main/trainers.py

---

Q6: I cannot understand what the contribution is.

A6: We appreciate your feedback and are happy to clarify our key contributions:

1. Smoothed Preference: We propose a novel smoothed preference distribution replacing DPO's binary modeling. Unlike fixed-weight approaches, our (2α-γ) scaling factor acts as a pair-dependent dynamic regulator, automatically driving loss to zero for similar pairs while amplifying gradients for dissimilar ones with each image pair's reward signal.

2. Precision Optimization: Unlike Diffusion-DPO's random noise injection, our Renoise Inversion technique enables direct optimization of image-correlated variables. This contribution provides more stable training and higher efficiency.

---

Q7: It seems that $\gamma$ can be absorb in $\beta$ by $(2\alpha-\gamma)\beta=(\frac{2\alpha}{\gamma}-1)\gamma\beta$ is calculated by equation (12).

A7: We appreciate this observation.

The term $(2\alpha-\gamma)$ is image-pair-dependent, automatically adjusted based on reward differences. The core idea of $(2\alpha-\gamma)$ is to ensure that "the loss decreases when preferences are more similar, and increases otherwise." This can be implemented via different parameterization strategies. For example, $\gamma$ can be step- or epoch-aware, adjusting dynamically to avoid being absorbed into $\beta$. We adopt a simple formulation: $(2\alpha-\gamma)$ is implemented as $(\frac{2\alpha}{\gamma}-1)\gamma$, balancing adaptability and numerical stability. Ablations (Table 5) validate the impact of different $(\gamma,\beta)$ pair configurations.

---

Q8: Optimization via Renoise Inversion. There is no optimality analysis.

A8: Thank you for your feedback. Optimizing Equ (10) requires sampling $x_{1:T}\sim p_{\theta}^{c}(x_{1:T}|x_{0})$. While Diffusion-DPO approximates this with $q(x_{1:T}|x_{0})$, we argue precise $x_{0}$ reconstruction needs more accurate latents. This motivates our use of diffusion reconstruction methods (DDIM Inversion), supported by additional theoretical analysis (refer to response A2 to Reviewer 5qKt).

We are honored by your favorable evaluation and have carefully considered your suggestions!

---

Q1: The rationale for designing the weight-to-sensitivity ratio as given in Equ (12) is unclear.

A1: Thank you for your feedback! According to Equ (8), $\tilde{p}(x_{0}^{w}|c) = \frac{p(x_{0}^{w}|c)^{\alpha}p(x_{0}^{l}|c)^{\gamma-\alpha}}{Z_{p}^{w}(c)}=\frac{(p(x_{0}^{w}|c)^{\frac{\alpha}{\gamma}}p(x_{0}^{l}|c)^{1-\frac{\alpha}{\gamma}})^{\gamma}}{Z_{p}^{w}(c)}$

where weight-to-sensitivity ratio could be regarded as the pairwise preference probability $p(x_{0}^{w} \succ x_{0}^{l}|c)$. Since a pairwise preference is hard to model directly, we adopt the well-established Bradley-Terry framework through the reward model as Equ (12).

---

Q2: The GPU hours for SmPO should include the training time of the reward model.

A2: Thank you for the feedback! Training PickScore requires 8×A800 GPUs for ~40 minutes [1]. We have updated Table 1 to include this cost.

Supp Table 1: Computational cost comparison

[1] https://github.com/yuvalkirstain/PickScore

---

Q3: PRDP, SPO and RankDPO should be included.

A3: We appreciate the suggestion! We'll update our related work with these papers and include the SmPO-SPO comparison.

Supp Table 2: Median score comparison of SD1.5 on HPD v2.

Supp Table 3: Win-rate comparison on Parti-Prompts.

However, Comparisons with PRDP (different SD-v1.4 base) and RankDPO (different training data) are limited by the unavailability of checkpoints.

---

Q4: There are no experiments validating the design of the weight-to-sensitivity ratio.

A4: Thank you for the feedback. We have designed two experiments to verify this aspect:

1. Ablation Study (Table 3): The results demonstrate that incorporating our PickScore-based weight-to-sensitivity ratio (+Smoothed) leads to significant performance gains, validating its effectiveness.
2. Reward Model Analysis (Table 7): We have further compared the weight-to-sensitivity ratio with different reward models, providing additional empirical support for our design choice.

---

Q5: Could you provide quantitative results on the GenEval benchmark?

A5: Yes! Thank you for highlighting this important aspect. We have conducted quantitative results on the GenEval.

Supp Table 4: GenEval scores over SD1.5 baselines.

Supp Table 5: GenEval scores over SDXL baselines.

Experimental results indicate that SmPO consistently enhances image-text alignment performance.