Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting

We sincerely thank the reviewer for the efforts in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

---

W1: Concern about the novelty and difference from [1,2].

Thank you for your insightful thoughts regarding the distinction from existing works. We recognize there may have been some misunderstanding due to the weighted forms in Eq(3) and Eq(4). Our work is distinctively focused on the transferability of pre-trained diffusion models which is significantly different from the approaches taken in [1,2], both in analytical and technical aspects:

• Differences in Analysis: Our work discusses model transferrability from both theoretical and empirical analysis, as detailed in Section 3.1, which is never explored in previous works. [1,2], though also concentrate on varying timesteps $t$, focus on general learning of diffusion models and address the issues of gradient conflicts in multi-task learning.
• Differences in Technique Details: Technically, our proposed Diff-Tuning aims to trade-off not varying timesteps $t$ but the retention and adaptation. As stated in Line 194-196, we ensure $\psi(t) + \xi(t) = 1$ in our Diff-Tuning, which maintains the overall loss in Eq(5) with the default DDPM weight (uniform weight across timesteps). And in our experimental implementation, we uniformly sample timesteps and then sample training data from either the memory buffer or target dataset according to $\psi(t)$ and $\xi(t)$, respectively. This contrasts sharply with [1,2], which assign carefully designed weights to different diffusion timesteps $t$ to manage gradient conflicts. From the gradient conflict perspective, we do not alter the weights of individual timesteps; the improvements arise from the transfer preferences we introduce.

Additionally, we present supplementary experiments combining Diff-Tuning with MIN-SNR-$lambda$ ($\lambda = 1,5$) [1].

Table A': Comparison with MIN-SNR

The concerns raised by Reviewer WW5v discuss other existing works, we hope Common Concern #1 in Author rebuttal also helps to address your concerns.

From these results, we observe that MIN-SNR [1] does not perform well in transfer learning, since MIN-SNR lacks consideration on transferrability, and its weigting strategy is completely incompatible to the chain of forgetting principle.

---

W2: Robustness to sampling algorithms and additional diffusion models

We agree that evaluating our method with more sampling algorithms and diffusion models would strengthen our findings. To point out, our selected pre-trained models, DiT [5] and Stable Diffusion [6], are the largest and most representative foundation models in the diffusion model family, which ensures the effectiveness of our methods. Meanwhile, we always follow their default sampling strategies to ensure the validity of our experiments. As complement, we further conduct the following experiments to demonstrate the robustness of our method:

• In the original class-conditional experiments, the main results were obtained using 50 uniform DDIM[3] steps. We have extended these experiments to include 100, 250, and 500 DDIM steps, as well as DPM-solver[4]. The results are summarized below:

Table F: Analysis on the different sampling algorithms

*Due to the time limit, we do not carefully tune the hyperparameters.

• In the original manuscript, we have fine-tuned the DiT-XL-2 model (pre-trained using a VP-SDE approach [7]) and Stable Diffusion. We employed the same diffusion training strategy used for the pre-trained models. Additionally, we have evaluated EDM on publicly available repositories [9]. Since EDM incoporates continuous $\sigma$ instead of discrete $t$ in the training state, we extend to use $\psi(\sigma)=\text{cdf}(\sigma)$ and $\xi(\sigma)=1-\psi(\sigma)$, aligning our standard Diff-Tuning. Results are shown below:

Table G: Results of fine-tuning EDM

---

W3: Need to keep the same schedule in the Diff-Tuning?

Diff-Tuning does not assume any specific scheduling strategy. However, when fine-tuning a model from a pre-trained checkpoint, it is generally preferable to maintain the same schedule as the pre-trained model to avoid potential mismatches. In our experiments, we adhered to this principle by fine-tuning DiT with a linear schedule and Stable Diffusion with its default schedule.

---

References

[1] Efficient Diffusion Training via Min-SNR Weighting Strategy. ICCV 2023.
[2] Addressing Negative Transfer in Diffusion Models. NeurIPS 2023.
[3] Denoising Diffusion Implicit Models, ICLR 2021
[4] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, Neurips 2022
[5] Scalable diffusion models with transformers, ICCV 2023
[6] Stability. Stable diffusion v1.5 model card, https://huggingface.co/runwayml/stable-diffusion-v1-5, 2022.
[7] Score-Based Generative Modeling through Stochastic Differential Equations, ICLR 2021
[8] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022
[9] EDM repo URL: https://github.com/NVlabs/edm

---

We hope that our additional clarifications and discussion address your questions and concerns.

We sincerely appreciate the time and effort you have taken to review our manuscript and for your constructive feedback. We will address the concerns you raised and revise the paper accordingly, as your comments provide valuable insights for improving our work.

---

W1: Use more metric to show the results of class-conditional generation.

While FID is based on features from models trained on ImageNet, the calculation between the downstream dataset and generated data does not directly reflect any bias from ImageNet training data. In fact, intuitively, using ImageNet might even be perceived as potentially detrimental to the FID value.

Despite FID's widespread use as a metric for evaluating modern generative models, we acknowledge the need for a broader set of evaluations. We agree that additional metrics, such as sFID [1], Precision, and Recall [2], provide a more comprehensive comparison. During the rebuttal phase, we have evaluated Diff-Tuning with these metrics, and the results are as follows:

Table D: More class-conditional generation metrics evaluated on Stanford Car

These results demonstrate that Diff-Tuning enhances the quality of generation with comparable precision and recall. We also provide showcases under the same random seeds. Please refer to Figure 1 in the attached PDF file.

Q1: Would it be useful to use pre-trained domains?

On the one-hand, using the original pre-training data is indeed applicable to Diff-Tuning. By comparing Table 1 and Figure 6(d), using the entire pre-training dataset outperforms vanilla fine-tuning, showing the validity of using pre-trained domains. On the other hand, note that using pre-training dataset may not yield optimal results compared with using generated samples as illustrated in Figure 6(d). We believe this is because that, as discussed in Section 4.4, the fine-tuning stage typically involves fewer training steps and utilizes only a small subset of the pre-training data, whereas the samples generated from a learned model can be more representative than sampling from a large empirical dataset. Hence using pre-trained data, even with accessibility, is sub-optimal.

Q2: Do augmented data generated by different sampling methods achieve similar results

This is an important concern regarding the robustness of Diff-Tuning to different augmented datasets. In our original experiments, the augmented data are generated using the default evaluation protocol provided by the pre-trained model (e.g., the pre-trained DiT-XL-2 was evaluated using DDIM with 250 steps and a CFG of 1.5, so we maintained the same strategy). To further investigate the impact of different sampling methods, we analyze the results with 50 and 500 DDIM steps. These results, in conjunction with Section 4.4 and Figure 6(d), should address your concerns about the impact of different sampling methods on performance.

Table E: Analysis on the different sampling steps of augmented data

Q3: Would fine-tuning a pre-trained model achieves better results than training from scratch on small dataset

The pre-training and fine-tuning paradigm generally outperforms training from scratch, especially for small datasets. Fine-tuning offers numerous advantages, including higher-quality results, lower computational costs, reduced training data requirements, and more efficient training processes.

---

References
[1] Generating Images with Sparse Representations, ICML 2021
[2] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019

---

We hope that our additional clarifications and discussion address your questions and concerns. Please let us know if you have any further concerns.

We sincerely appreciate the careful review and insightful suggestions provided by the reviewer. Our responses to the concerns raised are detailed below:

---

W1&2: The motivation of knowledge retention and a potential varient of Diff-Tuning with knowledge distillation

Thank you for your insightful comments on the motivation behind knowledge retention and the concerns regarding knowledge distillation (KD). Firstly, it is crucial to emphasize that a novel and critical contribution of our work is the unveiling of the chain of forgetting tendencies, both experimentally and theoretically, the theoretical analysis presented in Theorem 1 (Lines 151-162) is never discussed before. The chain of forgetting tendencies provides a guideline for designing transfer methods.

Utilizing an augmented data buffer in a generative model has been a rational practice since [1], and KD is a viable implementation only if it adheres to the principle of chain of forgetting. Both techniques should be effective within the framework of Diff-Tuning. Below, we summarize reasons why KD is not our initial choice:

• GPU Memory: KD requires maintaining a copy of the pre-trained model alongside the fine-tuning model. For large models, this significantly increases memory costs. For example, we can run Diff-Tuning with a batch size of 32 on a single A100-40GB GPU, whereas the KD variant decreases to a batch size of 24.
• Computational Cost: KD doubles the forward computation cost by necessitating the matching of output distributions between two models. Notably, pre-computing the KD labels is not feasible due to the inherent noise in diffusion training. For instance, for a batch size of 24, we achieve 2.1 training steps per second, compared to 1.34 for the KD variant.
• Training Instability: Transferring a pre-trained model to a domain significantly different from its training data can introduce out-of-distribution corruption during distillation, potentially causing instability in the fine-tuning process. We invested considerable effort to tune a suitable trade-off for the KD loss (0.05), and sometimes the training is easily disrupted due to an unsuitable KD loss setting.
• Implementation difficulty: An elegant KD implementation requires users to be familiar with the code framework and introduces a large design space. In contrast, an augmented replay buffer introduces only a small set of extra source data and changes the training data sampled related to $t$, which is considerably easier to implement across various fine-tuning scenarios.

Considering scenarios where KD might be preferred over an augmented data buffer, we have also implemented a new variant of knowledge retention incorporating KD. Due to time constraints, we have tuned a preliminary implementation and present the KD results below for a quick overview during the rebuttal phase, and we will revise the methodology section to include this discussion accordingly.

In light of potential settings to prefer KD over an augmented data buffer, we also implement a new varient of knowledge retention incorporating KD. Due to time constraints, we have tuned a preliminary implementation and present the KD results below for a quick overview during the rebuttal phase, and we will revise the methodology section to include this discussion accordingly.

Table C: The FID results comparison of the KD varient. (KD loss is trade-off by 0.05)

W3: Similar phenomena noted in previous literature [2,3].

See Common Concern #1 in Author Rebuttal.

Q1: Does Diff-Tuning train the class embeddings?

Yes, we fine-tune the class embeddings in Diff-Tuning using the same methods as in the pre-training stage. To further note, we observed that either reinitializing the class embeddings or directly updating existing ones shows similar performance.

Q2: Why using the whole ImageNet dataset performs worse than 5k Images and the intuition behind.

The slight difference in performance between using the entire ImageNet dataset and 5k augmented images (5.54 vs. 5.52, with vanilla fine-tuning at 6.04) is not statistically significant. As discussed in Section 4.4, one potential explanation is the size of the dataset relative to the fine-tuning duration. Given that fine-tuning only requires 24k steps, a large source dataset is underutilized, where only a fraction of images are iterated (about 1/3 of an epoch). In this context, data sampled from a learned model acts as a form of knowledge distillation, representing the original distribution more effectively than direct sampling from a vast dataset.

[4] explores how synthetic training loops affect generative model performance when the real dataset is fixed. However, this does not directly apply to fine-tuning diffusion models to new domains with new data.

We sincerely thank you for your insightful comments. We will expand the discussion in Section 4.4 in the revised paper accordingly.

---

References
[1] Transferring GANs: generating images from limited data ECCV 2018.
[2] Perception Prioritized Training of Diffusion Models, CVPR 2022.
[3] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers, 2023.
[4] Self-consuming generative models go mad, ICLR 2024.

---

We hope that our responses adequately address your queries and clarify our methodologies. We welcome any further questions or feedback.

We sincerely thank the reviewer for the careful review and insightful suggestions. Our responses to the concerns raised are outlined below:

---

W1: More discussion on how to set the hyperparameter $\xi(t)$ and $\psi(t)$.

Thank you for highlighting the importance of hyperparameter selection for $\xi(t)$ and $\psi(t)$. To clarify, as detailed in Lines 194-196, we set $\xi(t) + \psi(t) = 1$ to ensure uniform weighting across timesteps. This design simplifies implementation and emphasizes the trade-off between retention and adaptation, without the complexity of a detailed scheduling mechanism. Our approach diverges from conventional MLE-based weighting schedules but can be adapted to include them by modifying $\xi(t)$ and $\psi(t)$ to align with a predefined weight function $w(t)= \frac{\beta_t}{(1-\beta_t)(1-\alpha_t)}$ as discussed in DDPM [1].

• Robustness of $\xi(t)$ and $\psi(t)$: As discussed in** Section 4.4 Lines 314-321 and shown in Figure 6(c)**, our results indicate that Diff-Tuning is robust to variations in the design of $\xi(t)$ and $\psi(t)$. This robustness highlights the primary importance of managing the chain of forgetting tendency over specific scheduling details. Our choice, linear to $t$, while simple, proves effective in achieving the desired behavior.
• Difference with (MLE-based) weighting schedules: We recognize that we missed an emphasis in the method details and cause confusion. We here clarify that, as stated in Lines 194-196, we set $\xi(t) + \psi(t) = 1$ to maintain a consistent weight across all timesteps, thereby the weighting strategy of $\xi(t)$ and $\psi(t)$ focuses on the trade-off between retention and adaptation, instead of the reweighting across all timestep $t$.
• Applicability for any (MLE-based) weighting schedules: We agree that a maximum likelihood analysis or other sophisticated weighting schedules could potentially aid in the optimal selection of hyperparameters, and present how our Diff-Tuning strategy can be potentially combined with (MLE-based) weighting schedules: Suppose a weighting schedule is defined by the weight function $w(t)$ on the timestep $t$ (such as the MLE-based weighting schedule $w(t)= \frac{\beta_t}{(1-\beta_t)(1-\alpha_t)}$, referring to the original derivation in DDPM [1]), then one can set

$$\xi’(t)=w(t)\xi(t),\quad \psi’(t)=w(t)\psi(t),$$

and replace $\xi(t),\psi(t)$ in Eq.(3)(4) with $\xi’(t),\psi’(t)$ to satisfy $\xi’(t)+\psi’(t)=w(t)$, i.e., in usage of the corresponding weighting schedule. For simplicity, we have retained the most widely used weighting of $w(t)=1$ and do not alter the weight or the distribution of $t$ for fair comparison.

W2: Whether using the datasets applied for pre-training during retention would be better (or worse).

The primary reason we do not use the pre-training data is the typical unavailability in most fine-tuning scenarios. However, using the original pre-training data is indeed feasible with Diff-Tuning. As discussed in Section 4.4 Lines 325-328 and shown in Figure 6(d), using the entire pre-training dataset may not yield optimal results. We believe this is because the fine-tuning stage usually involves fewer training steps and utilizes only a small subset of the pre-training data (about 1/3 epoch here), whereas the samples generated from a learned model can be more representative than sampling from a large empirical dataset.

W3: how the method can be extended to scenarios where transfer learning is applied multiple times or where the data distribution changes online.

The scenario you mentioned aligns with the setting known as continual learning, which is a meaningful extension of Diff-Tuning. It is feasible to extend the replay buffer to collect generated data (or a subset of training data) as the data distribution evolves over time. We conducted experiments on the continual learning dataset Evolving Image Search [2], which consists of images with 10 categories collected in recent years, and splitted into 3 folds: 2009-2012, 2013-2016, and 2017-2020. We perform transfer learning sequentially across these splits, and calculate the FID value in the the accumulated test set. For Diff-Tuning, we maintain $60\%$ of the pre-sampled augmented data and collect other $40\%$ from the fine-tuned model before fine-tuning to the next split. The experimental results are shown below:

Table B: Continual learning with multiple fine-tuning.

From the results, is observed that Diff-Tuning demonstrated significant improvements, underscoring the importance of addressing the forgetting phenomenon in continual learning scenarios.

---

References

[1] Denoising Diffusion Probabilistic Models, Neurips 2020.
[2] Active Gradual Domain Adaptation: Dataset and Approach, IEEE Transactions on Multimedia 2022.

---

We hope that our additional clarifications and discussion address your questions and concerns. Please let us know if you have any further concerns!