SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation

I appreciate the authors' effort in providing additional results and explanations. Since most of the concerns have been addressed in the rebuttal I'm raising my score.

We sincerely thank you for the valuable suggestions to improve our work. The following content provides a point-by-point response to address your concerns. The corresponding modifications have been refined in the revised submission, highlighted in blue.

Q1: My main concern centers on the assumption underlying this approach—that parameters with the smallest absolute values are inherently ineffective. This seems more empirical than rigorously substantiated.

A1: Thanks for your question. 1) Many works[1,2,3,4] have studied the influence of these parameters with the smallest absolute values and confirmed the ineffectiveness of these parameters across many model architectures, including the most widely-used Convolution and Attention modules. Therefore, we further conduct experiments on stable diffusion series (SD 1.5, 2.0, 3.0, XL) and validate the ineffectiveness of these parameters in Sec. 3.1. Although this is not strictly proved, but extensive experiments show that the mainstream models have the same property. Maybe there exist some exceptions, therefore we will include this into the limitation part (Sec. M in Appendix) for a more comprehensive analysis. 2) Also, some pruning studies [5,6] also find that removing unnecessary weights can reduce the model's complexity and decrease the risk of overfitting. Setting these parameters to zero can suppress noise, allowing the model to focus more on important features, thereby improving its performance.

Q2: The generalizability of the 'adapting small-value parameters' strategy across different architectures is crucial to ensure broader applicability.

A2: Thanks for your question. 1) Stable diffusion 1.0 and 3.0 have totally different model architectures, where SD 1.0 employs Unet structure and SD 3.0 employs DiT structure. Our model performs well on both of the two model architectures. 2) Moreover, we also validate the effectiveness of our model on SD XL, which is a two-stage diffusion model. Our SaRA also achieves the best performance on it, validating the robustness of our model.

Q3: In the caption of Figure 3, weight distributions are claimed to be Gaussian without further clarification, which seems empirical rather than solid.

A3: Thanks for your question. For a continuous model, due to random data sampling and the stochastic nature of the noise addition and denoising processes in diffusion models, it is reasonable to assume that the probability of any given parameter's gradient update increasing or decreasing is uniform. Assume that each parameter's gradient update $\Delta$ $\theta$ is an independent random variable, and the probability of increasing or decreasing is uniform (i.e., symmetric distribution). We can then assume that the expected value $$\mu$$ of these gradient updates is 0, and the variance is $$\sigma^2$$.

For N gradient updates, the cumulative effect is:

According to the Central Limit Theorem, when $N$ is large, the cumulative gradient update $$\theta$$ will approach a normal distribution:

Therefore, for a model with high randomness, the parameters are very likely to be a Gaussian distribution. However, since this is not strict proof, we modify the corresponding Caption from "which all belong to a Gaussian distribution" to "which are all similar to a Gaussian distribution". Thanks for your reminding to improve our work.

Q4: The choice of threshold for selecting low-absolute-value parameters could be better justified. If this choice is sensitive, it could limit SaRA's robustness and generalizability across different diffusion models and tasks. An ablation study on this threshold choice would strengthen the claims.

A4: Thanks for your suggestion. We have conducted experiments on different thresholds in Tab.1, where the parameter numbers are controlled by the threshold. To study the influence of the threshold more comprehensively, we further conduct an experiment on different thresholds (ranging from 2e-4 to 1e-3) on Expedition dataset and SD 1.5, where the results are shown below:

It can be seen that when the threshold is too small (e.g., 2e-4), the FID becomes much higher, indicating learning less target domain knowledge. And when the threshold is large (i.e., threshold>2e-3), the model performs quite stably. Since we have a low-rank loss, the model with a high threshold also keeps the CLIP score well. In summary, our SaRA performs well in different thresholds, demonstrating the robustness of our model. (supplemented in Sec. H-Tab. 8)

We sincerely thank you for the valuable suggestions to improve our work. The following content provides a point-by-point response to address your concerns. The corresponding modifications have been refined in the revised submission, highlighted in blue.

Q1: Although the paper differentiates itself from selective fine-tuning, I think this method still appears to be a form of selective fine-tuning. The memory efficiency improvement seems to stem from implementing a separate node for gradient storage and selectively fine-tuning only those nodes, rather than from an inherent algorithmic difference.

A1: Thanks for your question. Our method can be interpreted as a selective fine-tuning method and this is not denied in the paper. Nevertheless, the differences between ours and conventional selective fine-tuning methods include: 1) we select a different subset of parameters ("unimportant" parameters) to finetune, and 2) our unstructual backpropagation strategy, which is proposed to enhance the GPU memory efficiency by designing separate nodes. This strategy is not only limited to our method, but can be extended to other selective PEFT methods, as illustrated in Lines 408~410 of the main paper. 3) To make it clearer, we add more relative descriptions in the end of Sec. 4.4:” Notably, our unstructual backpropagation strategy is not limited to our method only. It can be employed in other SFT methods like LT-SFT, which can advance the development of future SFT fields.”

Q2: Selecting parameters based on a specific threshold seems not a new concept. For example, the related work PaFi is described that it also trains based on absolute values.

A2: Thanks for your question. There are several differences between PaFi and Our SaRA. 1) Motivation: The motivation between our method and PaFi is quite different, PaFi only aims to fix the parameters before training, therefore, it selects the trainable parameters based on the parameter values. But it does not analyze why these parameters can be used. In contrast, our SaRA first conducts extensive experiments to validate the temporal inefficiency and potential effectiveness of the selected parameters. And then this observation motivates us to make use of the parameters with the smallest absolute values. 2) Technical contribution: Beyond simply using these parameters, we further proposed a low-rank constraint and progressive training scheme, along with the unstructual backpropagation strategy, which effectively improves the model performance and training efficiency.

Q3: The reasoning behind inefficient parameters becoming efficient during training due to the randomness of the training process is unclear. Since the initial weights are set randomly, many values are likely to change through training. There is no strong basis to assume a correlation with the initial values. Even parameters with initially small values are expected to converge toward the average distribution, making Figure 3 somewhat self-evident.

A3: Thanks for your question. 1) In Fig. 3, the initial weights come from a pre-trained diffusion model on FFHQ, rather than a random distribution. We continue training it and record the changes of all the parameters in Fig.3. It shows that most of the ineffective parameters in the pre-trained model become effective again, validating their potential effectiveness. 2) This positive transformation from ineffective to effective is ensured by the backpropagation of training loss gradients. Specifically, the fine-tuned parameters are adapted and trained on a specific training dataset for a particular task.

Q5: CLIP L/14 used by authors is trained to provide overall image captions and could miss the details. The visual language models-based evaluations used in T2I-CompBench++[Huang et al. 2023] could be more accurate.

A5: Thanks for your suggestion. We have evaluated the model performance by the visual language model in T2I-CompBench++ for the models trained on stable diffusion 1.5 (the metric is denoted as B-VQA), and the detailed results are shown in Tab. 6 of the appendix. We summarize the results in the following table, where for each method, we compute the average FID, B-VQA and VLHI across all the datasets and model scales (5M, 20M, and 50M parameters). It shows that our model achieves the best performance across all three metrics.

Q6: The authors skip the most popular Stable Diffusion XL 1.0 version, whereas they include 2.0.

A6: We have conducted experiments on Stable Diffusion XL 1.0, comparing DoRA, Lora, LT-SFT, Adaptformer, and our model. The detailed results are provided in Tab.5 and Sec. C in the Appendix. Similar to the table above, we compute the average FID, CLIP score, and VLHI across all the datasets and model scales, and show the results in the following table. It can be seen that our model achieves the best performance across these 3 metrics, validating the effectiveness and robustness of our model.

Q7: Typographical mistakes such as: *) Table 1:wrong column 2&3 names ) Figure 1: addictive-> additive

A7: Thanks for your suggestion. We have carefully revised the re-submission.

Reference

[1] Liu S, Wang C Y, Yin H, et al. DoRA: Weight-Decomposed Low-Rank Adaptation[C]. ICML, 2024

[2] Gal R, Patashnik O, Maron H, et al. Stylegan-nada: Clip-guided domain adaptation of image generators[J]. ACM Transactions on Graphics (TOG), 2022.

[3] Ojha U, Li Y, Lu J, et al. Few-shot image generation via cross-domain correspondence[C]//CVPR, 2021.

[4] Mo S, Cho M, Shin J. Freeze the discriminator: a simple baseline for fine-tuning gans[J]. arXiv, 2020.

[5] Wang Y, Gonzalez-Garcia A, Berga D, et al. Minegan: effective knowledge transfer from gans to target domains with few images[C]//CVPR, 2020.

We sincerely thank you for the valuable suggestions to improve our work. The following content provides a point-by-point response to address your concerns. The corresponding modifications have been refined in the revised submission, highlighted in blue.

Q1: The authors introduce a novel Visual-Linguistic Harmony Index (VLHI) metric; however, it's described only in the Appendix.

A1: Thanks for your suggestion. Considering the constraints of the paper's length, we describe the novel VLHI metric in Sec.B in Appendix. Following your advice, we have moved its definition from the supplementary material to the beginning of the experiment section (Sec. 5) for better presentation.

Q2: No comparison with the recent SOTA PEFT techniques (e.g., DoRA [Liu et al. 2024] that is available for different models on mentioned in the paper CIVITAI).

A2: Thank you for your suggestion. We conducted further comparisons of our model with DoRA [1] using SD 1.5 and SD 2.0. Detailed results can be found in Sec. C of the Appendix. The key metrics are summarized in the following table for DoRA and SaRA with different parameter numbers (e.g., SaRA-50M refers to SaRA with 50M parameters). The results show that DoRA performs similarly to LoRA but fails to achieve as good an FID as our model, indicating that it captures less target-domain knowledge compared to our approach. Although DoRA achieves slightly better CLIP scores under certain conditions, the superior VLHI achieved by our model highlights its ability to effectively learn target-domain knowledge while maintaining the original model's prior.

Q3: No ablation on scaling the trained SaRA weights for the inference (as the lora_scale parameter controls the influence of LoRA weights) or mentioning it in the limitations.

A3: Thanks for your suggestion. The scaling weight is indeed an important property for the PEFT methods in generative models. We conduct additional experiments on different SaRA weights and the visualization results can be seen in Sec. F of Appendix. As the scaling weight grows, SaRA further emphasizes the target-domain knowledge, while sacrificing the generalization ability. When the weight is too big, the model tends to generate some artifacts. This phenomenon is quite similar to LoRA-Scale.

Q4: The authors say that for the FID computation they sampled 5K images from the source and generated data; however, BarbieCore dataset has only 315 images which is definitely not enough for the proper FID evaluation. The details about the sizes of the used datasets should be in the paper.

A4: Thanks for your suggestion. Although the FID may not be so accurate when the image number is limited, but it can still be used as a relative comparison, which is widely used in few-shot (less than 1000 images) domain adaptation tasks [2,3,4,5]. Compared to the previous method, our approach achieves the best FID score at most of the conditions, demonstrating the effectiveness of SaRA. Moreover, We have provided additional detailed explanations of the datasets in Sec. B of the Appendix, where the size of each dataset can also be found in the following table. Although there are only hundreds of images, the dataset size is still larger than the few-shot datasets [2,3,4,5].

Dear Reviewer RosU,

We greatly appreciate the time and effort you have devoted to reviewing our paper!

Since there is only one day left to upload the revised PDF, we kindly hope you might find a moment to review our response, if your schedule allows. Should there be any further points requiring clarification or improvement, we remain fully committed to addressing them without delay. Thank you once again for your valuable feedback on our work!

Best regards,

The Authors

Dear Reviewer RosU,

We sincerely apologize for our repeated requests and any inconvenience they may have caused. We truly appreciate your time and effort in reviewing our paper, and we understand that you have many other commitments.

As the discussion period is coming to a close, we kindly ask if you might be able to provide your feedback at your earliest convenience. If there are any further points that require clarification or revision, we are fully committed to addressing them promptly.

Thank you once again for your patience and invaluable feedback.

Best regards,

The Authors

We sincerely thank you for the valuable suggestions to improve our work. The following content provides a point-by-point response to address your concerns. The corresponding modifications have been refined in the revised submission, highlighted in blue.

Q1: Results are not showing overall superiority over other methods.

A1: Thanks for your question. For image generation tasks, FID and CLIP scores exhibit a certain degree of mutual exclusivity in finetuning a text-to-image model to downstream tasks (i.e., an overfitted model will result in the best FID but the worst CLIP score), and this is why we introduce a new metric VLHI to balance both the two metrics (as illustrated in Lines 725-727). Evaluated by VLHI, our model achieves the best performance in learning the task-specified knowledge and keeping the model prior.

Q2: The authors should consider splitting Figures 2 b) and 5 into more subplots. In my opinion, they are too cluttered and it takes too much time to read them.

A2: Thanks for you suggestion. We have splitted Figure 2 (b) and Figure 5 into subplots, which is more clear to read.

Q3: In Table 1, the use of "optimal" and "sub-optimal" is not correct, e.g. optimal FID is 0. "Best" and "second best" or something similar should be used.

A3: Thanks for your reminding. We have corrected them in Tab. 1, along with some corresponding corrections in the main text.

Q4: Could the authors provide more qualitative results for the same prompts (Appendix C) with different seeds to see the diversity of the samples?

A4: Thanks for your suggestion. We have included more visualization results in Sec. C (Figs. 11-13), which show the diverse generation capabilities of our model.

Q5: It would be good to include some of the visual results in the main text.

A5: We have included Fig. 7 (Part of Fig. 8 in Appendix) in the main paper. Since the main manuscript contains quite many quantitative experiments, we can only show part of the visual results limited by number of pages. Thanks for your understanding, and we have also illustrated this Lines 499-501 in Sec. 5.2.

Q6: Can authors elaborate on how CLIP score is related to overfitting?

A6: Thanks for your question. If the model is overfitted, the model can only generate the images that resemble the images in training dataset, whose output images do not align well with the given texts. In this case, the CLIP score will be extremely low. So, a highly overfitted model will result in a very low CLIP score. We also add a detailed explanation in Sec. B.

We sincerely thank you for the valuable suggestions to improve our work. The following content provides a point-by-point response to address your concerns. The corresponding modifications have been refined in the revised submission, highlighted in blue.

Q1: This paper poorly extends the concept in static model to the finetuning area.

A1: Thanks for your suggestion. 1) We have analyzed the dynamic change of these unimportant parameters in Fig. 3 (Sec. 3.2), where we find that these ineffective parameters are not born to be ineffective, but come from the unstable training process. After finetuning the model, these parameters can be effective again. 2) Moreover, in Sec. 4.3, we dynamically adjust the parameters to be trained, which is designed to further improve the effectiveness of our learned parameters. 3) Meanwhile, we find that these sparse parameters even have a larger feature space than LoRA as analyzed in Sec. G, where SaRA can learn more task-specified knowledge, (which is also validated by the FID score in Tab. 1), indicating a larger parameter search space in our model.

Q2: How to merge multi tasks’ parameters, and their merging performance.

A2: Thanks for your reminding. The multi-task parameter merging is indeed an important property of LoRA. Comparably, we propose a new parameter combination method to combine different parameters learned from different datasets in Sec. G. Specifically, for two SaRA parameters $\Delta P_1$ and $\Delta P_2$ learned from different datasets, we can merge them by:

In this way, the merged SaRA parameter $\Delta P$ will contain the knowledges from both the datasets, whose output image contains both the target features. The visualization results can be seen in Fig. 16 and we also include this parameter merging technique in Sec. G.

Q3: How to understand the better performance after setting some parameters to 0 in Section 3.1? This is actually interesting and may be useful for understanding the behavior of diffusion model.

A3: Thanks for your question. 1) In Sec 3.2, we conduct exploration experiments and empirically find that the parameters close to 0 are caused by the unstable training process. We argue that these parameters may introduce some harmful randomness into the generated results. After setting these parameters to 0, the model would not be influenced by them, therefore it may improve the FID score a little bit. 2) Also, some pruning studies [1,2] also find that removing unnecessary weights can reduce the model's complexity and decrease the risk of overfitting. Setting these parameters to zero can suppress noise, allowing the model to focus more on important features, thereby improving its performance.

Reference

[1] Han S, Pool J, Tran J, et al. Learning both weights and connections for efficient neural network[J]. NeurIPS, 2015.

[2] Choudhary T, Mishra V, Goswami A, et al. A comprehensive survey on model compression and acceleration[J]. Artificial Intelligence Review, 2020.

Dear Reviewer EfNM,

We greatly appreciate the time and effort you have devoted to reviewing our paper!

Since there is only one day left to upload the revised PDF, we kindly hope you might find a moment to review our response, if your schedule allows. Should there be any further points requiring clarification or improvement, we remain fully committed to addressing them without delay. Thank you once again for your valuable feedback on our work!

Best regards,

The Authors

Dear Reviewer EfNM,

We sincerely apologize for our repeated requests and any inconvenience they may have caused. We truly appreciate your time and effort in reviewing our paper, and we understand that you have many other commitments.

As the discussion period is coming to a close, we kindly ask if you might be able to provide your feedback at your earliest convenience. If there are any further points that require clarification or revision, we are fully committed to addressing them promptly.

Thank you once again for your patience and invaluable feedback.

Best regards,

The Authors

Thank you for the reminder.

We have included and discussed SHiRA and SpIEL in the related work section. Briefly, the comparison between SHiRA and our SaRA is as follows: 1) SHiRA focuses on improving adaptation ability through a high-rank sparse matrix, while our SaRA aims to preserve model priors while effectively learning target-domain information. Therefore, these two works differ in their objectives and contribute distinct technical advancements. 2) Regarding the memory improvement strategy, we modify the optimizer such that only a single line of code is required to call AdamW-SaRA, which efficiently updates specific parameters and gradients (Sec. B in Appendix). In contrast, SHiRA uses a hook mechanism to achieve a similar purpose. However, due to the lack of open-source code, we cannot perform further quantitative or qualitative comparisons. We would appreciate more details about your code implementation, as a combination of the strengths of both approaches may lead to a more efficient and enhanced version of the training process.

Thanks for your attention to our work!

            self.optimizer_dict[id(p)].step()
            self.optimizer_dict[id(p)].zero_grad(set_to_none=True)
            self.scheduler_dict[id(p)].step()

        # Register the hook onto every parameter
        for p in self.model.parameters():
            if p.requires_grad:
                p.register_post_accumulate_grad_hook(optimizer_hook)
                
        def spatial_optimizer_hook(p):
            self.optimizer_dict[id(p)].step()
            self.optimizer_dict[id(p)].zero_grad(set_to_none=True)
            self.scheduler_dict[id(p)].step()
        
        if self.other_model is not None:
            for p in self.other_model.parameters():
                if p.requires_grad:
                    p.register_post_accumulate_grad_hook(spatial_optimizer_hook)
            
    @property      
    def get_better_parameters_name(self):
        lisa_target_modules = [
                "to_q",
                "to_k",
                "to_v",
                "to_out.0",
                "proj_in",
                "proj_out",
                "ff.net.0.proj",
                "ff.net.2",
                "conv1",
                "conv2",
                "conv_shortcut",
                "downsamplers.0.conv",
                "upsamplers.0.conv",
                "time_emb_proj",
            ]
        return lisa_target_modules

You can use it as follows

# Initialize your model

    lisa_trainer = LISADiffusion(unet, spatial_head, rate=0.25)
    lisa_trainer.insert_hook(optimizer_class=optimizer_class,
                        get_scheduler=get_scheduler,
                        accelerator=accelerator,
                        optim_kwargs=dict(lr=args.learning_rate,
                                          betas=(args.adam_beta1, args.adam_beta2),
                                          weight_decay=args.adam_weight_decay,
                                          eps=args.adam_epsilon),
                        sched_kwargs=dict(name=args.lr_scheduler,
                                          num_warmup_steps=args.lr_warmup_steps,
                                          num_training_steps=args.max_train_steps))
    lisa_trainer.register(optimizer_class=optimizer_class,
                        get_scheduler=get_scheduler,
                        accelerator=accelerator,
                        optim_kwargs=dict(lr=args.learning_rate,
                                          betas=(args.adam_beta1, args.adam_beta2),
                                          weight_decay=args.adam_weight_decay,
                                          eps=args.adam_epsilon),
                        sched_kwargs=dict(name=args.lr_scheduler,
                                          num_warmup_steps=args.lr_warmup_steps,
                                          num_training_steps=args.max_train_steps))
# ...
global_step=0
for i in range(epoch):
  for image in enumerate(dataloader):
    # forward
    if global_step % 20 == 0 and global_step != 0: # you can use other number to replace 6
      lisa_trainer.lisa_recall()
      accelerator.clear()
    global_step += 1

    def lisa_recall(self):
        param_number = len(list(self.model.parameters()))
        lisa_p = 8 / param_number if self.rate is None else self.rate
        self.lisa(model=self.model,p=lisa_p)
        self.last_epoch += 20
        self.model.zero_grad()
        if self.accelerator is not None:
            self.accelerator.clear()
        gc.collect()
        torch.cuda.empty_cache()
        
        if self.grad_aware:
            new_probability = []
            for i, param in enumerate(list(self.model.parameters())[1:-1]):
                if self.grad_record.get(param, None) is None or len(self.grad_record[param]) < 200:
                    v = 1. if len(new_probability) == 0 else max(new_probability)
                else:
                    v = torch.stack(self.grad_record[param],0).std().item() + 1e-6
                new_probability.append(v)
            new_probability = [j/sum(new_probability) for j in new_probability]
            for i, new_p in enumerate(new_probability):
                self.probability[i] = self.probability[i] * 0.01 + new_p * 0.99
            self.probability = [j/sum(self.probability) for j in self.probability]

        
    def initialize(self):
        self.optimizer_dict = dict()
        self.scheduler_dict = dict()
        self.grad_record = dict()

    def register(self, optimizer_class=None, get_scheduler=None, accelerator=None, 
                 optim_kwargs={}, sched_kwargs={}):
        
        self.lisa_recall()
        for p in self.model.parameters():
            self.grad_record[id(p)] = []  
            if p.requires_grad:
                self.optimizer_dict[id(p)] = optimizer_class([{"params":p}], **optim_kwargs)      
                if accelerator is not None:
                    self.optimizer_dict[id(p)] = accelerator.prepare_optimizer(self.optimizer_dict[id(p)])

        for p in self.model.parameters():
            if p.requires_grad:
                self.scheduler_dict[id(p)] = get_scheduler(optimizer=self.optimizer_dict[id(p)], **sched_kwargs)
                if accelerator is not None:
                    self.scheduler_dict[id(p)] = accelerator.prepare_scheduler(self.scheduler_dict[id(p)])
        
        if self.other_model is not None:
            for p in self.other_model.parameters():
                if p.requires_grad:
                    self.optimizer_dict[id(p)] = optimizer_class([{"params":p}], **optim_kwargs)
                    if accelerator is not None:
                        self.optimizer_dict[id(p)] = accelerator.prepare_optimizer(self.optimizer_dict[id(p)])

            for p in self.other_model.parameters():
                if p.requires_grad:
                    self.scheduler_dict[id(p)] = get_scheduler(optimizer=self.optimizer_dict[id(p)], **sched_kwargs)
                    if accelerator is not None:
                        self.scheduler_dict[id(p)] = accelerator.prepare_scheduler(self.scheduler_dict[id(p)])
    
    def insert_hook(self, optimizer_class=None, get_scheduler=None, accelerator=None, 
                 optim_kwargs={}, sched_kwargs={}):
        
        for param in self.model.parameters():
            param.requires_grad = True
            
        def optimizer_hook(p):
            if p.grad is None:
                self.optimizer_dict[id(p)].zero_grad(set_to_none=True)
                self.optimizer_dict[id(p)].state_dict().clear()
                self.scheduler_dict[id(p)].state_dict().clear()
                del self.optimizer_dict[id(p)]
                del self.scheduler_dict[id(p)]
                return
            else:
                if self.grad_aware:
                    with torch.no_grad():
                        self.grad_record[id(p)].append(((p ** 2).sum() / p.numel()))
                        while len(self.grad_record[id(p)]) > 200:
                            self.grad_record[id(p)].pop(0)
                        
                if id(p) not in self.optimizer_dict:
                    self.optimizer_dict[id(p)] = optimizer_class([{"params":p}], **optim_kwargs)
                    if accelerator is not None:
                        self.optimizer_dict[id(p)] = accelerator.prepare_optimizer(self.optimizer_dict[id(p)])    
                if id(p) not in self.scheduler_dict:
                    self.scheduler_dict[id(p)] = get_scheduler(optimizer=self.optimizer_dict[id(p)], **sched_kwargs)
                    self.scheduler_dict[id(p)].last_epoch = self.last_epoch
                    if accelerator is not None:
                        self.scheduler_dict[id(p)] = accelerator.prepare_scheduler(self.scheduler_dict[id(p)])
                
            if accelerator is not None and accelerator.sync_gradients:
                accelerator.scaler.unscale_(self.optimizer_dict[id(p)])
                torch.nn.utils.clip_grad_norm_(p, 1.0)
                

Thanks for your attention.

1. Your suggestion to improve the first and second-order moments in the optimizer is highly insightful, and we will further explore this in the future.

2. Additionally, our method can indeed be applied to other approaches. Inspired by an analysis of the pre-trained diffusion model parameters, we developed our method to enhance the fine-tuning capabilities of diffusion models. In the future, we will explore its potential applications in other models.

3.  Lastly, It seems that our method is not similar to LISA. LISA primarily focuses on a Layerwise Importance Sampling approach for LLMs, whereas our method introduces a progressive low-rank sparse fine-tuning approach tailored for diffusion models. The two methods are quite different in terms of both motivation and technical details. In the revised version, we have cited LISA to provide a more comprehensive discussion of the existing fine-tuning methods.