Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models

Q1 "Could you compare the results of this method to those of more intensive DPO training (e.g., running DPO for a longer period)"

Please refer to W2.2.

Q2 "If the original pretrained model is used instead of the NPO-trained model to obtain unconditional/negative outputs, is there a significant performance drop?"

Yes, it will suffer from significant performance degradation. Please refer to Q1 of Reviewer 5Mi5. Your mentioned baseline is equivalent to the baseline of AutoGuidance [1].

Further clarity on the application scope of Diffusion-NPO:

In addition to the above replies to proposed questions, we hope to clarify that our method and validation are not solely limited to DPO (Direct Preference Optimization) which requires paired data for tuning. Please refer to lines 66-81 and lines 354-367. Specifically, current diffusion-based preference optimization methods can be generally categorized into three types: 1) Differentiable Reward (DR) 2) Reinforcement Learning (RL) 3) Direct Preference Optimization (DPO). We validate diffusion-NPO on three baselines: VADER (DR-based method, online generation, and evaluation, data-free), Diffusion-SPO (RL-based method, online generation, and evaluation, data-free), Diffusion-DPO (DPO-based method). This is to say, we implement and validate the effectiveness of NPO on all three types of methods for diffusion-based preference optimization instead of being limited to DPO.

[1] Guiding a Diffusion Model with a Bad Version of Itself. NeurIPS Oral.

W2.2 "It does not clearly demonstrate the difference from scaling up the existing DPO method. "

The default training iteration of DPO and NPO are both 2,000 iterations. To show the effectiveness of NPO compared to solely scaling up existing DPO method. We adopt the official code of DPO and extend the training iteration of DPO to 20,000 iterations (i.e., 10x training cost). In the following table, we use the 1x, 2x, ... to denote the training iterations of each model.

We summarize the quantitative evaluation results in the following table.

From the table, we can observe the following:

1. Regardless of how long DPO is trained, the NPO weight offset, trained with only 1x training cost, consistently enhances overall performance.

2. The combination of DPO (1x) + NPO (1x) achieves significantly better results:

   • Aesthetic Score: 5.7667
   • HPS Score: 27.46
   • PickScore: 21.47

   This outperforms DPO (10x), which requires 5 times longer training cost (10/(1+1)) but only achieves:

   • Aesthetic Score: 5.6264
   • HPS Score: 27.10
   • PickScore: 21.05

We believe this is very strong evidence of the effectiveness of diffusion-NPO.

Thank you for your in-depth review and question. We value the opportunity to further showcase the effectiveness of Diffusion-NPO.

W1 "The paper seems underdeveloped in certain areas." "Table 2 displays scores but lacks discussion in the main text."

Please refer to lines 411-414 where we provided a discussion of Table 2. Following your suggestion, we are willing to enrich the discussion here.

"Figure 8’s caption does not specify the methods compared."

Figure 8 is to show user study comparison of samples generated with NPO and samples generated without NPO. We adopted the legends with different colors to show the difference. The legends are "w/ NPO better" (samples generated with NPO are better), "w/o NPO better" (samples generated without NPO are better), and "Tie" (No preference).

Additionally, please refer to lines 470-473, where we provided a discussion on Figure 8 and specified the models adopted including Stable Diffusion XL, Stable Diffusion v1-5, and Dreamshaper.

W2.1 "In the motivating example of Section 2. Subtracting the resulting model's output seems equivalent to scaling η in the DPO model’s unconditional parameter."

Please kindly note that the $((1 − \gamma)(− \boldsymbol \eta))$ is just an approximate of the negative preference optimized $\boldsymbol \delta$, therefore we can not expect simpling scaling $\eta$ to achieve the same effect of NPO.

In the paper, we showed that the weight adopted for unconditional/negative-conditional prediction can be denoted as

$$\boldsymbol \theta_{NPO} = \boldsymbol \theta + \alpha\boldsymbol \eta + \beta\boldsymbol \delta,$$

where $\boldsymbol \theta$ is the pretrained model weight, $\boldsymbol \eta$ is the weight offset obtained through preference optimization, and $\boldsymbol \delta$ is the weight offset obtained through negative preference optimization. In the motivation example, we replace $\boldsymbol \delta$ with $(1-\alpha) (-\boldsymbol \eta)$ to give a straightforward thinking and illustration of core idea of negative preference optimization. However, please note that it is just a simplified case that assumes the parallel of $\boldsymbol \eta$ and $\boldsymbol \delta$.

In general, we can decompose the components of $\boldsymbol \delta$ into

$$\boldsymbol \delta = \boldsymbol \delta_{\parallel} + \boldsymbol \delta_{\perp}.$$

The component $\boldsymbol \delta_{\perp}$ can never be obtained with the oversimplified case.

Dear Reviewer 5t3C,

As the discussion deadline approaches, we kindly ask if our response has addressed your concerns. Please feel free to share any additional questions or feedback, and we’ll be happy to provide further clarification.

Best regards,

The Authors

Thank you for your encouraging review!

Q1

Thank you for proposing the insightful question. We conduct additional experiments to show the effectiveness of NPO compared to previous proposed training-free CFG-strengthening techniques including Autoguidance [1] and SEG [2] as you mentioned.

Specifically, we test with the following comparisons:

1. SEG on Stable Diffusion XL vs Naive CFG on Stable Diffusion XL.
2. DPO-optimized Stable Diffusion XL as conditional predictors and original Stable Diffusion XL as the unconditional predictors vs Naive CFG on Stable Diffusion XL.
3. DPO-optimized Stable Diffusion XL as conditional predictors and NPO-optimized Stable Diffusion XL as unconditional predictors vs Naive CFG on Stable Diffusion XL.

Note that the original Stable Diffusion XL can be viewed as an earlier version of DPO-optimized Stable Diffusion XL. Therefore, we adopt 2 as the baseline of Autoguidance.

We generate all the images with the same seed and the same prompts from the test_unique split as the text prompts. We score generated images with the HPSv2, PickScore, ImageReward, and Laion Aesthetic and compare their average scores and win-rates.

The results are shown in the following table. All the win-rate are compared with original SDXL.

We can see that SEG only achieves minor improvement on Laion-Aesthetic compared to the original baseline of SDXL. All the other metrics show that SEG significantly underperforms the naive CFG strategy with SDXL. This can be attributed to that SEG was originally proposed to tackle the circumstances where the prompt is not used. When applying AutoGuidance, we find the outputs tend to be blurry. Our quantitative evaluation also proves our claim. This can be attributed to the prediction difference between original SDXL and DPO-optimized SDXL being too large (as analyzed in lines 340-344)

We also test on the Stable Diffusion v1-5 for further comparison:

1. DPO-optimized Stable Diffusion v1-5 as conditional predictors and original Stable Diffusion v1-5 as the unconditional predictors vs Naive CFG on DPO-optimized Stable Diffusion v1-5.

2. DPO-optimized Stable Diffusion v1-5 as conditional predictors and NPO-optimized Stable Diffusion v1-5 as the unconditional predictors vs Naive CFG on DPO-optimized Stable Diffusion v1-5.

We did not test SEG on Stable Diffusion v1-5 since we did not find open-source implementation of SEG on Stable Diffusion v1-5.

The results are shown in the following table. All the win-rate are compared with dpo-optimized Stable Diffusion v1-5.

NPO still consistently outperforms the compared baselines.

Q7 "Ethical guidelines and limitations on NPO's use to ensure responsible application"

Thank you for the suggestion. Negative Preference Optimization (NPO) is designed to enhance model outputs by explicitly training the diffusion models to recognize and avoid generating results that are not preferred by users. By leveraging "undesirable knowledge" in a controlled manner, NPO provides a proactive mechanism for steering generative models away from harmful, inappropriate, or irrelevant outputs, which directly aligns with the principles of responsible AI.

To ensure ethical and responsible deployment, we propose the following guidelines and limitations for NPO's application:

Prevention of Malicious Use: NPO should strictly be restricted to contexts where the objective is to enhance user satisfaction or reduce harm. Clear safeguards must be in place to prevent its misuse for generating harmful, biased, or deceptive content, such as misinformation or discriminatory artifacts.

Transparent Disclosure: The incorporation of NPO should be transparently documented, allowing stakeholders to understand its role in mitigating undesirable outputs. This is critical to build trust in its application, especially in sensitive domains.

Ethical Oversight: The application of NPO must be guided by rigorous ethical review processes to evaluate potential downstream impacts, particularly when undesirable knowledge might inadvertently influence adjacent areas. Regular audits and public engagement can help ensure accountability.

Focused Scope: NPO should only be applied in areas where controlling and minimizing negative outputs is essential—such as user-facing systems or domains with stringent safety requirements—to avoid overfitting to narrowly defined "undesirability" that may limit broader applicability.

By embedding these practices into its development and deployment, NPO has the potential to act as a robust framework for aligning generative models with user preferences and societal values. This structured approach ensures that the technique is applied with care, precision, and an unwavering commitment to ethical AI.

[1] Diffusion model alignment using direct preference optimization. CVPR 2024.

[2] Aligning preference with denoising performance at each step.

[3] Video Diffusion Alignment via Reward Gradients.

[4] Training Diffusion Models with Reinforcement Learning.

[5] Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model. CVPR 2024.

[6] Large-scale Reinforcement Learning for Diffusion Models. ECCV 2024.

[7] Directly Fine-Tuning Diffusion Models on Differentiable Rewards. ICLR 2024.

[8] DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models. NeurIPS 2023.

[9] Improving sample quality of diffusion models using self-attention guidance. CVPR 2023.

[10] Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance. ECCV 2024.

[11] Guiding a Diffusion Model with a Bad Version of Itself. NeurIPS 2024 Oral.

[12] Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention. NeurIPS 2024.

Q1 "How does NPO distinguish between truly negative attributes and those that are simply neutral or contextual?"

Please refer to our reply to W1.

Q2 "What are the computational costs of NPO, especially in terms of memory and processing time for real-time applications?"

Training: Since NPO requires minor modifications to the original preference optimization methods, it requires the same computation costs.

Inference: Since all the models used in the paper adopt CFG for inference, NPO requires the same computation costs for inference. On the other hand, as illustrated in our limitations, Diffusion-NPO requires the storage of two different weight offsets for inference. However, since preference optimization can be achieved with parameter-efficient tuning techniques such as LoRA (LoRA is a low-rank approximated weight residual), it typically requires less than 1% additional parameters. For instance, we adopt the LoRA rank 8 for training, which only takes 0.17% additional parameters.

Again, please kindly note that this paper focuses on improving model alignment with human preferences, not real-time applications. Due to the inherently slow iterative denoising process, diffusion models are unsuitable for real-time use under standard settings, and none of the prior works [1-8] have achieved real-time performance. Thus, real-time application is not addressed here.

Q3 "How sensitive is NPO's performance to parameter settings, and what are the recommended heuristics for tuning them effectively"

We follow the default training configurations of original preference optimization methods [1-3] and find those default hyperparameters work well for NPO. We also conduct a hyperparameter sensitivity analysis in lines 476-485, Fig. 9 (visual-based), and Fig. 10 (metric-based). We tuned the parameters with a simple heuristic grid search. Specifically, we first choose parameters from 10^{k}, k= ..., -2, -1, 0, 1, 2,... to find the optimal k. Then we conduct simple grid search to find better hyperparameters.

Q4 "How well does NPO perform on domain-specific datasets like medical imaging or abstract art?"

Please refer to our reply to W4.

Q5 "Beyond quantitative metrics, what is the perceived quality improvement from NPO based on qualitative user evaluations"

Please refer to our reply to w2.

Q6 "How does NPO mitigate the risk of reinforcing biases present in preference datasets"

As illustrated in the paper, we adopt the pickapic dataset proposed in PickScore when training the DPO-based NPO model.

On one hand, PickScore has already made efforts to minimize bias to the greatest extent possible. While it is true that some degree of bias still exists in the dataset, as acknowledged in the PickScore paper, the dataset was constructed through a comprehensive evaluation by a large and diverse population. This broad aggregation of assessments helps mitigate biases inherent in the data, ensuring it is as objective as possible within practical limits.

On the other hand, our method emphasizes making effective use of the dataset, rather than aiming to completely eliminate its biases. While addressing bias is an important and ambitious goal, we believe it requires dedicated efforts in future specialized research, as it extends beyond the primary focus of our current work.

In addition, note that our method is based on Stable Diffusion, which includes an important safety checker capable of detecting and filtering generated content. This allows our method to leverage it for safety checks on the generated results, further reducing potential negative biases, such as sexuality.

W4 "NPO for Large-scale and real-time applications are not fully explored."

Large-Sacle:

We are already using the large-scale models to validate our paper. In total, we validate our methods on four large-scale models including Stable Diffusion v1-5 (0.9 B), DreamShaper (0.9 B), Stable Diffusion XL (3 B), VideoCrafter v2 (1.5 B), which are all considerable large models in diffusion fields. For example, the largest model in the widely recognized DiT (SoTA performance on the diffusion-based image generation) termed as DiT-XL/2 has only 0.6 B parameters.

For a direct comparison, most previous works [4-8] on diffusion-based preference optimization only validate their methods on one model including Stable Diffusion v1-4, v1-5, or v2, which are all 0.9 B parameters.

Real-Time:
This paper focuses on improving model alignment with human preferences, rather than developing real-time applications.

Diffusion models are inherently slow due to their iterative denoising process, making them unsuitable for real-time applications under standard settings. Additionally, none of the prior works [1-8] have achieved real-time performance. Therefore, real-time application is not explored in this paper.

W5 "Focus on CFG-based models limits the exploration of NPO's compatibility with other diffusion architectures. NPO's compatibility with non-CFG diffusion models remains unclear."

Diffusion-NPO is independent of model architectures and diffusion forms. It works at the prediction level instead of the model architecture level.

CFG is the motivation and basis of diffusion-NPO. We cannot perform diffusion-NPO if CFG is not supported. However, please note that CFG has become almost a necessity in the diffusion field for improved generation quality.

Previous works [9–12] show that even for unconditional diffusion models, we can still adopt CFG to significantly improve generation performance.

Thank you for your detailed comments! We are delighted that you find our paper novel, simple and effective, clear and well-organized, and with future impact!

W1 "While the paper is innovative, the simple reversal of preference pair rankings may oversimplify the complexity of human aesthetics."

Thank you for your thoughtful comment!

On one hand, the reversal of preference rankings naturally selects the "less-preferred" option in a pair, a concept inherently implied when individuals choose their "preferred" option. This makes it a straightforward and reasonable approach to construct negative data for training.

On the other hand, our approach is intentionally designed as a deliberate simplification to tackle a specific challenge: steering the model away from generating outputs that misalign with user preferences. By reversing the rankings, we effectively penalize undesired outputs, reducing their likelihood without significantly increasing computational complexity.

We view our method as a foundational step rather than a comprehensive solution. It highlights the potential of leveraging preference data in innovative ways while acknowledging that more sophisticated methods could better capture the nuances of human aesthetics. Future research could explore incorporating richer datasets, utilizing advanced ranking models, or developing mechanisms to account for context-dependent aesthetic judgments. These directions could further enhance the model's ability to generate outputs that more accurately align with the complexity and subjectivity of human preferences.

W2 "Metrics-driven approach lacking user perspective."

We already conducted a comprehensive user study that evaluates key aspects such as high-frequency details, color and lighting, and low-level compositional structures in our original submission. Please refer to Lines 430-475 and Figure 8 in the paper.

The user study highlights that NPO significantly enhances high-frequency details in the generated outputs, produces colors and lighting that better reflect human preferences, and improves the overall compositional structure of the images. These findings underscore the effectiveness of NPO in aligning diffusion models with human aesthetics.

Additionally, the reward models used in our paper, HPS v2 and PickScore, are specifically designed to incorporate the user's perspective. By leveraging these models, our goal is to ensure the generated outputs align more closely with human preferences.

W3 "The paper validates NPO primarily on general-purpose datasets, While these are commonly used in text-to-image generation, they may not represent the variety of domains where preference alignment is crucial, such as medical imaging, scientific visualization, or highly specialized art styles."

Mainstream prior works [1-8] on diffusion-based preference optimization predominantly focus on general-purpose text-to-image or text-to-video generation tasks, using these as standard benchmarks for validation.

Consistent with this established practice, our study evaluates performance on these widely adopted tasks, validating our effectiveness on both text-to-image and text-to-video. Our primary aim is to improve human preference alignment within the context of general-purpose text-to-image and text-to-video generation following the common practice of prior works.

While extending preference alignment to specialized domains such as medical imaging, scientific visualization, or highly specialized art styles may hold potential value, such explorations might lie beyond the scope of our current work. Specifically, to the best of our knowledge, we do not find relevant datasets, reward models, or methods for preference optimization in the field of medical imaging. The concept of preference optimization in the field of medical imaging is also somewhat ambiguous, as there are no established definitions or frameworks specifically addressing what constitutes "preference" in the context of medical imaging. This raises questions about how preference can be meaningfully defined and evaluated in such a domain.

Additionally, as none of the compared works have conducted experiments in these small specific domains, focusing on general-purpose benchmarks allows for a fair and meaningful comparison.

We acknowledge that future research could explore these specialized areas to further broaden the applicability of diffusion-based preference optimization.

Dear Reviewer Fyov,

As the discussion deadline approaches, we kindly ask if our response has addressed your concerns. Please feel free to share any additional questions or feedback, and we’ll be happy to provide further clarification.

Best regards,

The Authors

Thank you for your insightful and detailed review! We are delighted that you found our work to be simple, intuitive, and effective! Below, we address your questions in detail.

Q1: it is optimal to share the same training data for PO and NPO?

Thank you for your insightful and thought-provoking question!

In our original implementation, to ensure a fair comparison, we utilized the default training data pairs, data processing methods, and configurations from the original preference optimization techniques to train NPO (as described in Ln 372). This decision was made to isolate the impact of training data distribution, thereby highlighting the simplicity and effectiveness of NPO.

In response to your question regarding the impacts of better designed negative dataset, we conducted additional experiments to analyze the influence of training data. These experiments were performed using Diffusion-DPO-based NPO, as Diffusion-DPO is the only setting in our framework that requires training data.

To explore the impact of training data, we examined two specific scenarios:

1. Perturbed Text Prompts: In this scenario, we randomly shuffled the sequence of text prompts within batches to create irrelevant image-text pairs. This led to consistent performance degradation, suggesting that even for negative preference training, maintaining proper image-text pairings is crucial.
2. Random Image Corruptions: Here, we applied random corruptions, such as Gaussian blur, to the negative images. We observed that mild corruptions improved performance. However, excessively strong corruptions caused the unconditional outputs to deviate significantly from the conditional outputs, which could lead to inference failures.

These findings indicate that the image quality and image-text consistency of training data remain essential, even in the context of negative preference optimization.

The quantitative evaluation is shown in the following table. All the win-rate are compared with the original NPO.

Our experiments demonstrate that NPO with appropriately applied data corruption enhances overall performance. The results highlight that effective data usage and processing strategies can further boost the effectiveness of NPO, showcasing its great potential and extendability.

Thank you once again for your thought-provoking question! We believe this topic warrants deeper exploration in future work.

Difference and comparison with scaling up the existing DPO method. (Reviewer 5t3C)

The default training iteration of DPO and NPO are both 2,000 iterations. To show the effectiveness of NPO compared to solely scaling up existing DPO method. We adopt the official code of DPO and extend the training iteration of DPO to 20,000 iterations (i.e., 10x training cost). In the following table, we use the 1x, 2x, ... to denote the training iterations of each model.

We summarize the quantitative evaluation results in the following table.

We can see from the table that:

1. No matter how long DPO is trained. The NPO weight offset trained with only 1x training cost can consistently and significantly improve the overall performance.
2. DPO (1x) + NPO (1x) achieves Aesthetic Score (5.7667), HPS Score (27.46), PickScore (21.47). The performance significantly outperforms the DPO (10x), which requires 10/(1+1) = 5 times longer training cost with Aesthetic Score (5.6264), HPS Score (27.10), PickScore (21.05)

We believe this is very strong evidence of the effectiveness of diffusion-NPO.