Iterative DPO with An Improvement Model for Fine-tuning Diffusion Models

Validation

1. The performance improvement is very minor. From Table 4, we can see that the win-rate improvement is at most 0.9% compared to SPIN. I also think the presented paired visual comparisons in the paper (those figures) are not impressive.

2. The validation is not sufficient. It only validates performance on stable diffusion 1.5. Many works including Diffusion-DPO report their performance on both stable diffusion 1.5 and stable diffusion XL. The generation quality of stable diffusion 1.5 is not good and therefore improving generation quality from it is much easier than stable diffusion XL. It makes me suspicious about its effectiveness on better and larger models.

3. Validation of the effectiveness of the improvement model. I didn't find any validation on how much the improvement model can improve the given images. (I am sorry if I missed them.) The only available evaluation on the improvement seems to be the Table 3, which can not reflect the ability of the improvement model. The improvement model even performs worse than SPIN on test set. How can we expect it to improve the quality of generated images.

4. Many experimental details are not clearly stated. How is the win-rate calculated in Table 4?

5. Lack of user study including comparison and details.

Reasonability

1. Since all good images are generated with the improvement model, I think the generation quality might be upper-bounded by the improvement model. I am suspicious about the reasonability of the proposed method.

2. Generating new samples during training takes a lot of computation and time. I noticed in the paper that the authors train the improvement model 200K steps with batch size 2048, which is a very costly training compared to standard diffusion-DPO (They train their model with only 2K iterations with a similar batch size 2048). The very high training cost might also be the reason that they do not train their model on SDXL or even larger models.

Presentation

1. Most presented images are not visually appealing and their prompts are generally very short. It can not reflect the text-image alignment.

Due to the above reason, I think the paper still has a relatively large gap for publication and might not meet the bar of ICLR.

1. The authors should provide precise definitions and annotations for each concept, such as $\epsilon^w$ and $\epsilon^l$.
2. The proposed method introduces a semi-supervised learning framework to utilize a preference dataset for training a "high-quality" model, followed by fine-tuning the diffusion model using generated high- and low-quality data pairs. However, this approach appears to complicate the alignment process for diffusion models. A more straightforward solution would be to train a reward model directly, enabling reinforcement learning without the need to construct a high-quality model and apply DPO in such a rudimentary way.
3. Additionally, the performance gains achieved by the proposed method appear limited. As shown in Table 1, the results are only comparable to those of SPIN.
4. The authors should refine the contributions section, as the current version lacks clarity and specificity.
5. The authors should consider including a comparison with alignment methods based explicitly on reward model [1]. [1] Zhang, Y., Tzeng, E., Du, Y., & Kislyuk, D. (2024). Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401.12244, 4.

1. The main idea of this work is to provide online preference positive samples through constructing the preference improvement model that maps the outputs from the original diffusion model to high-quality ones. It is still under the static Diffusion-DPO framework without dynamic adaption, which is somewhat less convincing.
2. It is wondered whether the preference improvement model needs to be evolved in iteration, as the fixed improvement model can be hard to adapt to the shifted distribution of the evolved diffusion model, and the positive samples provided by consequent preference improvement are doubted to bias from the original user preference. Additionally, the fixed improvement model is still setting the upper bound of the optimized diffusion model, with only enlarged optimization samples.
3. It is recommended to provide the per-iteration result to monitor the diffusion model behavior.
4. The role of guidance weights in yielding positive samples is not well illustrated with corresponding qualitative or quantitative results. Basically, we have no idea about the behavior of the improvement model in terms of its multi-objective learning. The validation experiments should be provided.

Idea is unreasonable: Using the generated images from the improved models as winning samples is not a reasonable approach. Due to the instability of the generative model, even with the same seed, it is difficult to ensure that the images generated by the improved models are consistently better in quality than those from the base model. This introduces a significant amount of noisy data, which can severely affect the quality of model training. Furthermore, the method involves multiple iterations of training, which exacerbates this issue.

Lack of novelty: The paper merely proposes an iterative DPO method without providing a well-thought-out solution to the core problem, making its novelty and contributions quite limited.

Unreasonable experiments: Firstly, a considerable number of the visual examples presented in the paper do not demonstrate the advantages of the proposed algorithm. The generated images do not align well with the prompts, and the improvements over the baseline model seem quite limited. Does this suggest that the method’s effectiveness is rather underwhelming? To verify the validity of this approach, I think it is necessary to conduct experimental analysis on the number of iterations.