Efficient Transfer Learning in Diffusion Models via Adversarial Noise

Thank you for your thoughtful review and feedback.

1. Overlap of Results

• Overlap: The phenomenon of overlapping results, as suggested by the standard deviation values, is a widespread challenge in this area of study, not just specific to our research. This issue is also evident in methodologies such as CDC, DCL, and DDPM-PA. For example, the Intra-LPIPS result of DDPM-PA on FFHQ $\to$ Babies is 0.599 $\pm$ 0.024 and LSUN Church $\to$ Haunted houses is 0.628 $\pm$ 0.029, while the Intra-LPIPS result of CDC on FFHQ $\to$ Babies is 0.583 $\pm$ 0.014 and LSUN Church $\to$ Haunted houses is 0.620 $\pm$ 0.029.
• P Value: We also present a new table about p-value between our method and DDPM-PA to demonstrate the tangible improvement of our approach. The Inter-LPIPS score, our primary metric, often faces an overlap problem due to the highly competitive advancements in this field. Nevertheless, the p-value table still illustrate significant improvements of our method despite these constraints. | Datasets | Babies | Sunglasses | Raphael's paintings | Haunted houses | Landscape |-----|:-----:|:-----:|:-----:|:-----:|:-----:| | P-value (AP with TAN) | $2.6 \times 10^{-6}$ | $1.9 \times 10^{-25}$ | $7.0 \times 10^{-54}$ | $8.7 \times 10^{-86}$ | $9.5 \times 10^{-48}$ |

2. Qualitative Results on FFHQ

• In Figure 3 of the main paper, we have already presented samples transferred from the FFHQ dataset, providing a qualitative comparison with DDPM-PA and other methods. The FFHQ $\to$ Raphael's paintings transformation showcases the strengths of our method, particularly in managing complex features. Our approach achieves more accurate color reproduction and fewer artifacts compared to DDPM-PA. We would greatly appreciate it if the reviewer could share any relevant references, enabling us to thoroughly examine and pinpoint any potential discrepancies or overlooked aspects in our approach.

3. Comparison with Modern Text-to-Image Methods

• Different Tasks: The tasks of text-to-image transfer learning and few-shot image generation are fundamentally different in their objectives and methodologies. Text-to-image transfer learning primarily focuses on subject-specific generation. It aims to capture the essential features of a target domain while maintaining diversity, even with a limited set of images. Renowned methods in this field, such as CDC, DCL, and DDPM-PA, have not typically been applied to text-to-image conditional generation tasks. For instance, Dreambooth fine-tunes an existing text-to-image model using a small set of subject-specific images, enabling the model to form a distinct association with that subject. In contrast, few-shot image generation concentrates on preserving similarities and differences among instances in the source domain, without a specific focus on maintaining the primary subject of the target images.

• Dreambooth Results: Our new experiment, illustrated in Appendix Figure 7, reveals the difficulties Dreambooth faces in few-shot generation tasks, as the generated images closely resemble the target images. Additionally, it's challenging to conduct a fair comparison between our method and text-to-image approaches. These text-to-image methods typically require pre-training on large text and image paired datasets, enabling them to generate images of the target domain without fine-tuning. For example, the model used in Dreambooth can create images of people wearing sunglasses without any fine-tuning. The goal of Dreambooth's fine-tuning is to produce images that resemble the training data but set against different backgrounds.

I hope this message finds you well. I am writing to express my gratitude for your thoughtful review and feedback on our work. We have carefully considered your comments and have prepared a detailed rebuttal to address your queries and concerns.

In our rebuttal, we have clarified the quantitative improvement demonstrated by our results, differentiated our approach from current text-to-image methods, and validated the fairness of our comparative analyses with comprehensive statistical and visual evidence.

We carried out an additional anonymous user study to assess the qualitative performance of our method in comparison to DDPM-PA. In this study, participants were shown three sets of images from each dataset, featuring DDPM-PA, our method (DDPM+TAN), and images from the target domain. For each set, we displayed five images from each method or the target image, as illustrated in our main paper. To maintain anonymity and neutrality, we labeled the methods as A/B instead of using the actual method names (PA and TAN). We recruited volunteers through an anonymous online platform for this study. During the study, participants were tasked with choosing the set of images (labeled as A or B, corresponding to PA or TAN) that they believed demonstrated higher quality and a closer resemblance to the target image set.

Of the 60 participants, a significant 73.35% favored our method (DDPM+TAN), indicating that it produced images of superior quality and more effectively captured the intricate types of target domains. While this experiment did not comprehensively account for factors such as the participant's gender, age, regional background, and others, the results nonetheless suggest that our images possess better visual quality to a notable extent.

We kindly request you to review our rebuttal your feedback is invaluable to us, and we would greatly appreciate your insights on our responses.

Thank you for your time and consideration.

Thank you for your thoughtful review and feedback on our work. We will address each of your queries, clarifying any misunderstandings and delving into pivotal academic questions that merit exploration. We are also pleased to share some of our insights with you.

1. Qualitative Performance and Comparison with Existing Methods

• Illustration of Performance: Figure 3 in our study showcases our method's notable advancements over existing approaches, particularly DDPM-PA. Our technique effectively captures the intricate type of target domains. This is clearly visible in the top and bottom rows of the figure, where our method (4th and 5th images) aligns more closely with the target domain's type than DDPM-PA (3rd image).

• Specific Examples: We provide a detailed comparison of certain features, such as the watercolor brush strokes in the up images of churches and mountains, and the red hat or yellow curly hair in the bottom row images. These examples further highlight our method's enhanced ability to adapt to the target domain's type, offering a significant improvement over the results achieved by DDPM-PA.

2. PGD Step Number and Adversarial Noise Selection

• Min Max at Timestep t: As detailed in Equation 8 of the main paper, we employ a multi-step variant of Projected Gradient Descent (PGD) for the inner maximization. This approach helps in identifying the "worst-case" adversarial noise, which corresponds to the current state of the neural network. Subsequently, we use this adversarial noise as the "maximum" noise input to minimize our model's loss, as outlined in Equation 9 of our paper.
• Ensures the Performance: The choice of maximizing Gaussian noise at step t is strategically aligned with our aim to challenge the model with the “worst-case” scenarios. This approach ensures that the model is robust enough to handle the most difficult noise patterns, thereby improving overall performance. As demonstrated in Section 5.1 of the main paper, the Adervsrial noise with PDG can rectify the issue of incorrect gradients caused by limited data. Figure 2 (a) in main paper shows the output layer gradient direction for four different settings in the first iteration, all with identical noise and timestep $t$. The red line, derived from ten thousand varied samples, serves as a dependable reference direction (approximately 45 degrees southwest). For comparison, 10-shot samples are repeated a thousand times in one batch, aligning them with the ten thousand distinct samples. The dark blue and sienna lines represent the gradient computed using traditional DDPM as the baseline and similarity-guided training in a 10-shot sample, respectively. The orange line illustrates our method, DDPM-TAN, applied to a 10-shot sample. The gradient with adversarial noise (represented by the orange line) is closer to the reliable reference direction than the gradient without adversarial noise (illustrated by the sienna line). This proximity indicates the method's effectiveness in ensures the prformance.

3. Time and GPU Memory Conservation

• GPU Memory: To further elucidate these aspects, we will include graphical representations in the revised version of the paper, showcasing the efficiency gains in training procedures involving these components. The Table in Appendix A.5 illustrates the GPU memory usage for each module in batch size 1, comparing scenarios with and without the use of an adaptor. It reveals that our module results in only a slight increase in GPU memory consumption.
• Time: In Section 5.3 of our main paper, we discuss the time efficiency of our method. Our approach not only effectively but also efficiently transfers the model from the source to the target domain. Compared to other methods requiring about 5,000 iterations, our method needs only approximately 300 iterations with limited parameter fine-tuning. The incorporation of Training Adversarial Noise (TAN) further enhances efficiency. Effectively, our approach equates to 3,300 training iterations, significantly less than the standard 5,000 iterations and the 6,000 - 10,000 equivalent iterations needed for DDPM-PA. In terms of time cost, the baseline with an adaptor undergoing 5,000 iterations (the same as DDPM-PA) consumes about 4.2 GPU hours. In contrast, our model (DPMs-TAN) requires just 3 GPU hours for its 300 iterations, demonstrating notable time efficiency.

Thank you for your thoughtful review and feedback on our work. We will address each of your queries, clarifying any misunderstandings and delving into pivotal academic questions that merit exploration. We are also pleased to share some of our insights with you.

1. Additional Insights and Analysis

• Further Analysis and Visualization: In response to your request for deeper analysis, we provide a new experiment in Appendix A.4 about generated during each iteratuions as our exploration of similarity and adversarial noise. This include more detailed visualizations, especially comparing the reverse image of our method with that of the standard model with the different training iteraion step. As shown in Figure 5, the first row demonstrate that the orangial train the DPMs with limited iterations is hard to get a successfully transfer. The second raw shows that training with our similarity-guide method can boost the convergence to the taget domain. The third rows shows that training further with adversrial noise can even more faster converge. As shown the 150 iteration of right pictures, compared with the training only with similarity-guide (2nd row) TAN can get the face with sunglasses image.

<!-- This expanded analysis aims to provide more comprehensive insights that our method can boost the transfer learning. -->

• KL Divergence Analysis: Our methodology employs KL divergence between the source and target domains as a fundamental analytical tool. This approach aids in understanding the differences between well-established source and target models and also provides insights into effectively utilizing similarities in transfer learning for DPMs. We initially leverage existing knowledge by fine-tuning a diffusion model derived from a pre-trained model. Therefore, examining the differences between the pre-trained model in the source domain and the fine-tuned model in the target domain is essential Furthurmore, equation 5 in our paper, $`D`_{\textrm{KL}} \left( p_{\theta_{\textrm{S}}, \phi}(x_{t-1}^{\textrm{S}} | x_{t}), p_{\theta_{\textrm{T}}, \phi}(x_{t-1}^{\textrm{T}} | x_{t}) \right) = \mathbb{E}_{t, x_0, \epsilon} \left [ \left\| \nabla_{x_t} \log p_{\phi}(y=\textrm{S}|x_t) - \nabla_{x_t} \log p_{\phi}(y=\textrm{T}|x_t) \right\|^2 \right ]$, demonstrates how the divergence depends on $x_t$ at specific timesteps $t$. This highlights that transfer learning in DPMs can efficiently leverage this divergence for more effective training.

2. Application Beyond Transfer and Few-shot Learning

• Domain Adaptation, Control and Other Domains: Our method, centered on adversarial noise and similarity-guided training, can indeed be extended beyond transfer and few-shot learning. It holds potential in domain adaptation and control tasks, especially where the target domain has limited data but is somewhat related to a well-trained source domain. Our current focus has been predominantly on 2D images. However, the underlying principles of our method are adaptable to other domains. Thank you for your suggestion. We intend to explore the application of our technique across various fields and domains, including domain adaptation and 3D modeling, among others. This will enable us to fully utilize the versatility of our approach in our future work.

3. Adversarial Noise in General DPMs Training

Our method is applicable in general DPMs training, but the effectiveness of adversarial noise might be reduced during the initial unstable gradient phase and when dealing with large datasets.

• Training From Scratch: PGD may not be effective in the initial stages of the training process. The model's gradient is typically unstable when training begins from scratch, and adversarial noise requires a relatively accurate model gradient to function effectively. If the initial model's gradient is unstable, it fails to provide beneficial adversarial noise for training.

• Large Data Sets: The application of adversarial noise has substantially enhanced the training of DPMs under conditions of limited data and iterations. As shown in Figure 2 (Page 6) of our paper, the adversarial noise method helps address incorrect gradients, which is advantageous for training on smaller datasets. For larger datasets, however, the benefits of our approach might be less noticeable. This is due to the extensive iterations required for model convergence, as represented by the cyan line (associated with 10,000 points data) in Figure 2.

Therefore, although adversarial noise shows promising performance in few-shot transfer learning, its effectiveness may be less pronounced in general DPMs training.

I hope this message finds you well. I am writing to express my gratitude for your thoughtful review and feedback on our work. We have carefully considered your comments and have prepared a detailed rebuttal to address your concerns.

In our rebuttal, we clarify our methodology's nuances with added experiments and visualizations. Our discussion extends to the method's applicability beyond transfer and few-shot learning. Additionally, we explore the impact of adversarial noise in training general DPMs, considering various training scenarios and dataset sizes. We kindly request you to review our rebuttal and your feedback is invaluable to us, and we would greatly appreciate your insights on our responses.

Thank you for your time and consideration.

Thank you for your thoughtful review and feedback on our work. We will address each of your queries, clarifying any misunderstandings and delving into pivotal academic questions that merit exploration. We are also pleased to share some of our insights with you.

1. Pre-trained DPMs

• Clarification on Pre-trained DPMs: We provide additional details in Section 5.2 (Experimental Setup) about the pre-trained DDPM (Denoising Diffusion Probabilistic Models)[1] and LDM (Latent Diffusion Models)[2]. The pre-trained DDPM are from [1] and pre-trained LDM are from [2].
• Generalization of Our Method: Our method is general on different DPMs' pre-trained model since our method focuses on the how to transfer based on DPMs' theories instead of the structure of neural networks. Our method's generalizability is demonstrated through its successful application to two well-known DPM pipelines, DDPM and LDM. This highlights the method's versatility and wide applicability in various transfer learning scenarios within the DPM framework.

2. Visualizations of Adversarial Noise

• 2D Visualization: The background red point of Figure 2 (a) (page 6) presents a 2D visualization of adversarial noise, showing a transition from a standard Gaussian distribution (circular shape) to an elliptical one. The elliptical shape aligns with the gradient direction of the model's parameters. This illustrates how our approach fine-tunes the model by focusing on the most challenging noise scenarios.
• Challenges in Visualizing High-Dimensional Adversarial Noise: Visualizing high-dimensional adversarial noise, especially for real images involving vectors of 3 * 256 * 256 dimensions, is significantly challenging. These visualizations can be less intuitive and may struggle to convey critical information due to their complexity and high dimensionality.

3. PGD Step Number in Adversarial Noise Selection

• PGD Step Number: In all experiments presented in our paper, we set the PGD step number to $J = 10$ to demonstrate the generalizability of our method, as detailed in subsection 5.2 of the main paper.

[1] Ho, J., Jain, A. and Abbeel, P., 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, pp.6840-6851.

[2] Rombach, R., Blattmann, A., Lorenz, D., Esser, P. and Ommer, B., 2021. High-resolution image synthesis with latent diffusion models. 2022 IEEE. In CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 10674-10685).

I hope this message finds you well. I am writing to express my gratitude for your thoughtful review and feedback on our work. We have carefully considered your comments and have prepared a detailed rebuttal to address your concerns.

In our rebuttal, we clarify the origins and features of the pre-trained DPMs utilized in our study. We also provide details on our visualization techniques for adversarial noise. Additionally, we specify the PGD step number used in our experiments, highlighting the robustness and versatility of our approach. We kindly request you to review our rebuttal and your feedback is invaluable to us, and we would greatly appreciate your insights on our responses.

Thank you for your time and consideration.