Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation

Q1: The paper lacks a comparison with traditional fine-tuning methods for diffusion models, e.g., LoRA

A1: While LoRA is a parameter-efficient fine-tuning method that focuses on reducing trainable parameters under resource constraints, it is orthogonal to SPIN-Diffusion, which utilizes a self-play mechanism for fine-tuning. According to your suggestion, we have provided SFT (LoRA) fine-tuning results as follows. We can see that full fine-tuning generally surpasses the performance of LoRA fine-tuning.

---

Q2: The computational overhead of the self-play mechanism is high

A2: As this is a common concern raised by all reviewers, we have provided a general response regarding the sampling overhead. In summary, we have implemented some of the “advanced sampling acceleration techniques” and reduced the sampling overhead by 83%.

---

Q3: The motivation of this method is unclear to me

A3: Thank you for your comments. Here is the motivation for our work. The standard SFT method for diffusion models suffers from low alignment with human preferences and low data efficiency due to two main reasons: (1) it does not directly optimize for alignment with human preferences, and (2) only one round of training can be performed. To address this limitation, prior works (Fan et al., 2023; Black et al., 2023; Wallace et al., 2023) have proposed to use RL fine-tuning (RLHF) to directly align the diffusion model with human preferences. However, RL fine-tuning methods also have limitations: they either require an external reward function trained on additional data (Fan et al., 2023; Black et al., 2023) or rely on expensive human-annotated winner/loser paired images for each prompt (Wallace et al.). In order to overcome the limitations of existing fine-tuning methods, we propose SPIN-Diffusion, which overcomes the drawbacks of both SFT and RLHF through a self-play mechanism. Compared with SFT, our method is more data-efficient, by repeatedly using the prompts from the SFT dataset to improve the model through self-play. Compared with RLHF methods, our method does not need external reward models or expensive human-annotated winner/loser pairs. We will highlight the motivation of our work in the revision.

---

Q4: The paper assumes that the data distribution can be adequately represented by the parameterized family, which may not hold in all practical scenarios

A4: Thank you for your suggestion. It is indeed a common assumption in diffusion model algorithms that the parameterized family is expressive enough.For example, DDPO (Black et al., 2023), DPOK (Fan et al., 2023) and Diffusion-DPO are all built upon the assumption of an expressive reward model to capture data preferences accurately. To our knowledge, our work provides the first convergence guarantee along this line of research. While this assumption is solely introduced for theoretical analysis, we believe that the expressiveness of large neural networks generally satisfies this assumption in practice.

---

Q5: The evaluation is primarily focused on a single dataset

A5: We believe this is a misunderstanding. Our evaluation is in fact performed on three datasets. In Section 5.1 “We use the Pick-a-Pic test set, PartiPrompts (Yu et al., 2022) and HPSv2 (Wu et al.,2023) as our evaluation benchmarks.”. Due to space limit of the main text, additional results for PartiPrompts and HPSv2 are provided in Section B.3 “Evaluation on Other Benchmarks” of our appendix.

---

Q6: The font size of Figure 1 is too small

A6: Thank you for your feedback. We have increased the font size in Figure 1 for better readability in our revision, which is now available in the uploaded PDF.

---

Q7: Could you please discuss the advantages of fine-tuning DMs with RL compared to traditional methods?

A7: We do not view our method as an RL fine-tuning method. Here we will highlight the advantage of self-play fine-tuning for diffusion models over both SFT and RL fine-tuning. Standard SFT of diffusion models maximizes the log-likelihood of the training data. However, it cannot directly optimize the diffusion model’s performance in terms of quality indicators, as log-likelihood is not directly related to any human-perceived quality indicators. RL fine-tuning, on the other hand, overcomes this limitation by utilizing a reward function to maximize or by using human-annotated winner/loser image pairs as feedback, thereby optimizing the model to generate images with higher rewards (e.g., aesthetic score, image quality, etc.). In contrast, self-play fine-tuning uses an implicitly defined reward function integrated within the diffusion model's training process, allowing it to fully utilize the SFT dataset without requiring an additional reward model or human feedback.

---

Q8: I do not understand why this approach is considered infeasible

A8: We would like to clarify that “RL fine-tuning is infeasible” because of the lack of reward information or human feedback. As you mentioned, the prompt and the image can be considered as observations, but they do not directly provide a reward. As previously explained, RL fine-tuning methods rely on a reward function to optimize performance or on the human-annotated winner/loser image pairs as feedback. In contrast, SPIN/SPIN-Diffusion uses a self-play mechanism, which only requires a high-quality SFT dataset, removing the need to train a reward model that accurately reflects human preferences or human-label winner/loser pairs. Theoretically, we prove that SPIN-Diffusion performs distribution-level matching to the target data distribution.

Q1: The main contribution of this work is an approximate SPIN loss compared to the previously proposed exact SPIN loss. The major modification is moving the average over sampling steps outside the loss function, resulting in an upper bound. I kindly argue this improvement is straightforward when transferring SPIN from LLM to diffusion model, without much insights into diffusion model itself.

A1: We appreciate your feedback, but we would like to emphasize that the derivation of SPIN loss for diffusion models is highly non-trivial. One of the novelty of our approach lies in adapting the SPIN objective for diffusion models by considering the main player (reward function) across the full trajectory $x_{0:T}$ (as in Equation 3.2), rather than focusing solely on the final state $x_0$ as done in prior work (e.g., Fan et al., 2023; Black et al., 2023; Wallace et al., 2023). This modification allows us to formulate an exact objective function up to Equation (3.8), which departs from Wallace et al., 2023 that only considers the end state for rewards or preferences.

Moreover, there seems to be a misunderstanding about our approximate training objective. The exact computation of Equation (3.8) is impractical due to two primary constraints: the large trajectory length $T$, which would require an impractical amount of GPU memory when the loss is summed over $T$. Additionally, the required samples from a reverse process are not readily accessible. Consequently, our approximation strategy includes adopting an upper bound and using samples from the forward process as practical surrogates for the unavailable backward process samples.

This approximation is directly motivated by the nature of diffusion models, which inherently decouple operations on a per-time-step basis. We provide a theoretical justification for our approximation method in Section 4, ensuring that our approximations are both practical and theoretically sound.

Thank you again for your suggestion. In the revision, we will highlight the insights from our derivation related to diffusion models.

---

Q2: Since the approximate SPIN loss is proposed for data/memory/time efficiency, I didn't see any computing efficiency comparisons of the approximate and exact SPIN loss in the main paper or supplements. As noted by the authors, "it requires additional sampling overhead, approximately 10 times of the training time when using traditional DDIM sampling." This additional computing cost significantly limits the method in practice.

A2: In terms of training time, our approximate SPIN loss is approximately 2 times the training time of SFT loss. The additional sampling overhead, as in our discussion of limitations, primarily arises from the generation of synthetic data. This inference process, apart from training, is flexible, parallelizable, and distributable across various computing engines, from large GPU clusters to standard home laptops. To address the sampling overhead, we have explored advanced sampling algorithms and optimizations at both the software and hardware levels. These efforts have successfully reduced the sampling time by 83%. Further details can be found in the general responses to all reviewers.

---

Q3: Assumption 4.1 assumes the loss being monotonically decreasing and convex. It seems like the SPIN loss proposed in Equations. 3.8 and 3.9 do not meet this strong assumption. It makes the analysis results less convincing

A3: We believe this is a misunderstanding. Assumption 4.1 is made on $\ell$, rather than the SPIN-Diffusion loss in Equations 3.8 and 3.9. More specifically, we choose $\ell$ to be the logistic loss (in SPIN-Diffusion experiments), which is monotonically decreasing and convex. Other losses such as correlation loss and hinge loss also satisfy this assumption. So our theoretical analysis indeed holds for SPIN-Diffusion. We will clarify it in the revision.

---

Q4: Quantitative performance improvement (Table 1) is mild.

A4: There might be a misinterpretation regarding the performance. The SPIN-Diffusion model in fact significantly outperforms baselines SD-1.5 and Diffusion-DPO, with aesthetic scores improving from 5.7691 to 6.2481 and PickScores reaching 22.0024, even exceeding winner images from the Pick-a-Pic test set, which have an aesthetic score of 5.985 and a PickScore of 21.87.

The improvement is more obvious in Figure 1 and Tables 3, 4, where the winning rate over SD-1.5 reaches 91.6%.

---

Q5: It's hard to tell which method generates the best pictures in qualitative comparison (Figure 2).

A5: Images generated by SPIN-Diffusion iterations are generally more aesthetically pleasing. To offer a more objective comparison, we assessed the aesthetic scores of the 3 images in Figure 2. The results indicate that SPIN-Diffusion consistently outperforms SD-1.5, SFT, and Diffusion-DPO in visual quality:

For additional qualitative comparisons, please refer to Figures 8, 9, 10, and 11 in the appendix.

---

Q6: The "(ours)" notations in the tables are a bit confusing, may misleading the audience considering them as a method proposed in this work.

A6: Thank you for your suggestion. We've replaced "(ours)" in tables and figures with “(reproduced)” to clearly distinguish between existing checkpoints and our reproductions.

Q1: The sampling overhead is significant, requiring 5-10 times more training time.

A1: Thank you for highlighting this concern. Since this is a common concern among all reviewers, during the rebuttal period, we have worked on practical solutions addressing the sampling overhead problem, and listed the results in the general response. In summary, the current sampling time can be reduced to 7 hours vs. originally 43 hours per iteration. Further improvements are possible with more software/hardware/algorithm level designs.

---

Q2: There is insufficient explanation regarding the assignment of the hyperparameter $\beta_t$ and its variation across iterations.

A2: Thank you for your insightful comment regarding the hyperparameter $\beta_t$. Due to space constraints, we deferred the detailed discussion of hyperparameters to Section B.1 of the appendix. To summarize:

• $\beta_t$ values: First iteration: 2000, Second and third iterations: 5000

• Selection process: We conducted a grid search over {2000, 5000, 10000} to determine these values.

• Rationale: Our experiments revealed that later iterations typically benefit from more conservative updates, hence the larger $\beta_t$ value (5000) for the second and third iterations.

---

Q3: If a stronger approximation is applied and the sampling overhead is reduced by focusing on trajectories rather than all time steps, it might be possible to compare improvements with significantly reduced training time. Showing improvements within a setup that grants no more than twice the training time compared to SFT could have demonstrated the efficacy of SPIN-Diffusion more clearly.

A3: Thank you for your suggestion. We would like to clarify that the current training time of our method is already no more than twice that of SFT. In Section 3.4, in addition to formulating the problem with respect to uniformly sampled $t$, we further approximate the reverse timesteps in the expectation to the forward process timesteps. This allows the application of the same training mechanism of SFT, i.e., the inputs of the model are two images (real vs. generated), a batch of data consisting of noisy samples $x_t$’s from randomly sampled $t$’s. In addition, in (3.1) and (3.2) when we formulate the problem, we have already used a trajectory-wise characterization of the IPM and reward function. The trajectory-wise derivation and the approximation in Section 3.4 together ensure that the training time of our algorithm is about twice that of SFT.

In addition, as we mentioned before, we have managed to reduce the sampling time by 83% during our rebuttal, and now the sampling time is less than the training time.

---

Q4: I would like to see a practical comparison of computation time in GPU hours between SPIN-Diffusion and other methods.

A4: Following your suggestion, we have provided detailed statistics on the overhead and dataset requirements during each stage of training/sampling. The results were all obtained using 8 A100 GPUs (80 GB memory). From the table below, we can see that by applying fast sampling acceleration techniques, as detailed in our general response, we can control the total overhead of our algorithm within a reasonable amount.

---

Note: The training times listed depend on the number of steps trained during each iteration, which are selected by the validation results.