Self-Supervised Direct Preference Optimization for Text-to-Image Diffusion Models

W1 & Q1:The authors make use of the preference image in the Pick-a-Pic dataset. However, the claim is that the algorithm does not need annotation. To support the claim the authors should use random images from the Pick-a-Pic dataset, rather than the preference image; Will the approach work using random images from pick-a-pic?

Yes, the method performs effectively when using randomly selected images from the pick-a-pic dataset. As shown follows:

The results demonstrate that the random selection approach achieves comparable performance to the standard Self-DPO method across most metrics. We appreciate your feedback and will incorporate these findings into the revision.

Q2:Will the algorithm work with real images rather than generated images?

Yes, the method also works with real images. Specifically, we conduct experiments on a subset of the LAION-COCO dataset containing ~400k text-image pairs. Unlike carefully curated datasets, we do not manually design the data distribution or employ re-captioning techniques to refine prompts.

While naive SFT degrades performance, our Self-DPO method achieves consistent improvements over the baseline across four out of five metrics. Please note that we cannot provide a DPO baseline as it requires additional paired images and annotations for the same prompts, which are not available in regular text-image dataset.

Q3:Will the approach work for SOTA models such as FLUX? If it is unclear how to perform such an experiment, it can be useful to add this to the limitations section.

Yes, Self-DPO works for SOTA models such as FLUX beacuase it is model-agnosic. Ideally it can be employed in most diffusion models. As an initial attempt, we performs this experiment on FLUX using the Pick-a-Pic V2 synthetic dataset. Notably, this dataset comprises image generations from SD2.1 and SDXL, which have lower quality compared to FLUX outputs. Despite this, our experiments demonstrate that the method remains effective when applied to modern architectures.

Q4:For different downgrading operations, are you able to analyze and identify different improvement axis? this can also be answered qualitatively.

Thank you for your feedback. Each image downgrading operation involves distinct optimization dimensions. For random grid downgradtion, it varies in grid configurations and selection numbers. In such scenarios, a enhancement could involve using instance masks to relocate objects within the image, generating more feasible results with minimal fidelity loss. For blur downgradtion, it can be changed by different blur kernels and area. We appreciate your suggestion and will incorporate this discussion into the revision.

W1:While the paper reports final metrics, it omits learning-curve plots comparing Self-DPO with vanilla DPO. Without these curves, readers cannot assess convergence speed and optimization stability. Such information is important to judge whether the observed gains come from faster learning, better local minima, or merely longer training.

All baseline models are trained for the same number of iterations. We observe that the learning curves can demonstrate consistent improvement over training. Unfortunately, due to the rebuttal policy, we are unable to include visualizations in this rebuttal. The final figures and detailed descriptions will be provided in the revision. We appreciate your feedback.

W2:It is crucial to demonstrate effectiveness on a standard text–image dataset. A comparison against conventional supervised fine-tuning (SFT) on such datasets would reveal whether the method’s advantages persist in real-world scenarios.

Specifically, we conduct experiments on a subset of the LAION-COCO dataset containing ~400k text-image pairs. Unlike carefully curated datasets, we do not manually design the data distribution or employ re-captioning techniques to refine prompts.

While naive SFT degrades performance, our Self-DPO method achieves consistent improvements over the baseline across four out of five metrics. Please note that we cannot provide a DPO baseline as it requires additional paired images and annotations for the same prompts, which are not available in regular text-image dataset.

Minor1:Typo.

We will fix it in the revision, thank you.

Minor2: How were these hyperparameters ($\lambda_1$ and $\lambda_2$ in Equ.9) selected, and how sensitive are the results to them?

We present an initial ablation study on the impact of removing the MSE loss in the main text. Here, we provide an expanded analysis comparing different weight ratios between the Self-DPO and MSE components:

We have two obervations: First, when Self-DPO is entirely removed (0.0:1.0), the method reduces to standard SFT, yielding the lowest performance across all metrics. Second, introducing Self-DPO (ratios > 0.0:1.0) consistently improves results, with performance remaining stable across varying weight distributions. This suggests the Self-DPO loss plays the main role during the training process. These findings confirm that our method is insensitive to loss weight tuning, as the Self-DPO component primarily delievers performance gains.

Minor3: Blur and random-grid corruptions can hurt performance, whereas replacing the winner with a random image helps dramatically. Could you analyze why semantic mismatches outperform perceptual degradations?

This phenomenon can be attributed to the model's greater ease in learning blur and random-grid corruption patterns. In contrast, identifying random images proves more challenging for the model due to their similar quality, which necessitates heightened sensitivity to prompts, thereby enhancing performance. Future work could explore more diverse and reliable image degradation strategies to further optimize model capabilities.

Q1:The results show that Self-DPO is even better than the DPO with human annotation datasets. Could the authors explain this?

This can be attributed to the fact that conventional DPO relies on pre-generated offline data with limited diversity, whereas our method dynamically generates preference pairs in an online fashion. Online generation introduces greater data diversity by continuously sampling from the model's current policy distribution, addressing the static nature of offline datasets.

Q2:Random replacement often yields completely unrelated images, could the model over-weight low-level aesthetics at the expense of fidelity? A qualitative failure analysis would be valuable.

The model does not sacrifice fidelity during preference optimization. A detailed qualitative failure analysis will be included in the revision. Thanks for your advice.

Q3:When you draw a random loser from the same minibatch, its prompt is different from the winner’s. Do you ensure different seeds each epoch to avoid memorization?

Yes, each epoch generates different image pairs within the same minibatch.

W1 & Q1:The paper lacks a fair comparison with Diffusion-KTO, which is an important baseline as it also does not require paired data; The paper lacks a fair comparison with Diffusion-KTO, which is an important baseline as it also does not require paired data.

We apologize for the omission and provide qualitative comparisons below. The experiments were conducted using the official released model for fair comparison:

Our method demonstrates superior performance over Diffusion-KTO. Notably, while Diffusion-KTO avoids pairwise ranking, it still requires binary annotations (e.g., "like/dislike") for training, which incurs annotation costs similar to traditional preference-based methods. Thank you for your valuable feedback.

W2 & Q2:More ablation studies on fine-tuning methods used and its interaction effect with MSE losses are needed to clarify the effectiveness of each component; The paper does not clearly specify the fine-tuning method used.

In the main text, all experiments are conducted under full-parameter fine-tuning. Here, we present results using LoRA fine-tuning with rank 64:

We observe that both DPO and Self-DPO under LoRA show lower performance than their full-parameter counterparts but outperform the baseline. This can be attributed to the fact that LoRA reduces trainable parameters significantly compared to full fine-tuning. Notably, Self-DPO consistently surpasses DPO in all metrics, demonstrating its robustness even with parameter constraints.

Q3:This paper's method is simple and clean, which is good. The effectiveness of the proposed method needs more evidence.

We further validate our method’s effectiveness by introducing a new real-world T2I evaluation setting. Specifically, we conduct experiments on a subset of the LAION-COCO dataset containing ~400k text-image pairs. Unlike carefully curated datasets, we do not manually design the data distribution or employ re-captioning techniques to refine prompts.

While naive SFT degrades performance, our Self-DPO method achieves consistent improvements over the baseline across four out of five metrics. Please note that we cannot provide a DPO baseline as it requires additional paired images and annotations for the same prompts, which are not available in regular text-image dataset.

Formatting Concerns: Rows 168 and 169 are too close.

We will fix it in the revision. Thank you for your advice.

W1: The proposed algorithm seems simply borrows existing negative sample crafting techniques from SSL, the technical contribution is limited.

We appreciate your feedback and acknowledge that our implementation adopts a straightforward approach. However, we emphasize two key distinctions from SSL. First, our work introduces a novel perspective to direct preference learning by exploring whether preference data pairs can be generated directly from unpaired data. We affirmatively answer this question through a simple yet effective framework, demonstrating that this approach not only achieves good performance but also significantly reduces the annotation cost inherent in traditional DPO methods. Second, task-relevant image degradation plays a critical role, as demonstrated by our ablation study in Table 3 of the main text. Different degradation strategies yield divergent performance outcomes, and we propose a simple yet effective degradation method. Crucially, we show that excessive image quality degradation negatively impacts results, underscoring the need for balanced degradation design.

W2: The experiments are limited to SD1.5 and SDXL, limiting the robustness of Self-DPO.

Prior works commonly adopt two widely-used baselines (SD1.5 and SDXL), and we follow this fashion. Our method is model-agnosic and can be employed in most diffusion models. As an initial attempt, we extend our evaluation to the state-of-the-art FLUX model using the Pick-a-Pic V2 synthetic dataset. Notably, this dataset comprises image generations from SD2.1 and SDXL, which have lower quality compared to FLUX outputs. Despite this, our experiments demonstrate that the method remains effective when applied to modern architectures.

W3 & Q4: The authors only evaluated CLIP-based reward model suites, additional human/ large VLM evaluation could further solidate the results; Could the authors potentially include VLM evaluation on general human preference to complement the reward models?

We employ UnifiedReward [1] as the VLM evaluation metric, a state-of-the-art reward model designed for multimodal understanding and generation tasks. Built upon Qwen2.5-VL-Instruct, it supports pointwise scoring to align model outputs with human preferences. The experimental results are summarized below:

SD1.5 resluts:

SDXL resluts:

FLUX resluts:

The results demonstrate that Self-DPO shows better results over baseline methods under most comparisons. It highlights the effectiveness of our proposed method in aligning model outputs with human preferences.

[1] Unified Reward Model for Multimodal Understanding and Generation. arXiv 2025.

Q1: It seems that the MSE term is doing SFT and did not impact the performance significantly, it would be interesting to see how tuning the weights of MSE and Self-DPO affects the performance.

We present an initial ablation study on the impact of removing the MSE loss in the main text. Here, we provide an expanded analysis comparing different weight ratios between the Self-DPO and MSE components:

We have two obervations. First, when Self-DPO is entirely removed (0.0:1.0), the method reduces to standard SFT, yielding the lowest performance across most metrics. Second, introducing Self-DPO (ratios > 0.0:1.0) consistently improves results, with performance remaining stable across varying weight distributions. This suggests the Self-DPO loss plays the main role during the training process. These findings confirm that our method is insensitive to loss weight tuning, as the Self-DPO component primarily delievers performance gains.

Q2: On SDXL, given that the MSE term is essentially doing SFT and could degrade performance, has the authors attempted to remove the MSE term and keep only the Self-DPO loss?

Thanks for your advice! We conducted this experiment, and the results are presented below:

As anticipated, eliminating the MSE loss for the SDXL baseline yields marginal improvements. This aligns with our hypothesis that the Self-DPO loss plays the main role, and its standalone application achieves the best overall performance. Thank you for your valuable feedback!

Q3: The scale of reported Image Reward seems different from the raw model output, did the authors scale it by 100?

Yes. As shown in Line 203-204 in the main text, we scale ImageReward by a factor of 100. It is beacause we aim to adjust the value range approximately 0-100.