S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

Thank you for your valuable comments and suggestions. We will carefully revise our paper based on your comments. Our responses to your questions are detailed below. We would greatly appreciate your input on whether our revisions address your concerns.

Q1: The symbols in the proof process of this paper can be slightly organized.

A1: Thank you for pointing out this issue. In the camera-ready version of our work, we will simplify the symbols and add additional notations before our proof to improve readability.

Q2: There are few figures for the method in this paper, and the overall picture is not clear enough.

A2: Thank you for your suggestion. We will include more figures to illustrate our selection and permutation strategy specific to transformer architectures in detail. The current Figure 2 serves as an abstract demonstration of each coupled structure in a standard Transformer model. Additionally, we will ensure that our figures are clearer in the camera-ready version.

Q3: Does this method extract the same layer for each different model?

A3: Our method employs the same extraction strategy for all models discussed in our paper. Specifically, we always extract all feed-forward network (FFN) layers for Transformer models, with the trainable parameters uniformly distributed across the Up, Down, and Gate Projection Layers. This approach is based on our findings in Figure 4, which show that fine-tuning FFN layers results in better performance compared to fine-tuning Attention layers under the same budget. To demonstrate the effectiveness of our uniform allocation strategy among FFN layers, we have included an additional ablation study in Table R1.

In Table R1, we have added an ablation study regarding layerwise allocation of trainable parameters. This study includes the following design patterns:

1. Increasing ($n_{i+1} > n_i$): the number of trainable parameters in every layer gradually increases (or remains the same);
2. Uniform ($n_{i+1} = n_i$: the number of trainable parameters in every layer is the same;
3. Decreasing ($n_{i+1} < n_i$): the number of trainable parameters in every layer gradually decreases;
4. Random One: only one randomly selected layer has trainable parameters.

The results show that maintaining a uniform distribution of trainable parameters among different layers leads to the best performance. We will include these results in the camera-ready version of our paper if accepted.

Table R1: Performance of different layer allocation strategies on commonsense reasoning tasks.

While our allocation strategy is both model- and task-independent, there is great potential to further improve performance by selecting trainable parameters based on model weights or downstream tasks. We leave this problem for future research.

Q4: Will the distribution of data affect the data form of the U matrix and the form of the U matrix basis?

A4: Thank you for your question. Our work employs a random selection strategy for channel selection in Section 5.3, which is task-independent. To demonstrate how different U matrices affect performance on various tasks, we used three different random seeds for channel selection. The results for both commonsense reasoning and math reasoning tasks are presented in Table R2. Our findings indicate that the optimal form of the U matrix varies across different tasks, suggesting that the "best U matrices" should be task-specific. We have left this direction for future research.

Table R2: Ablation study of channel selection strategies on the commonsense reasoning and math reasoning tasks.

Q5: Can the author analyze each task more fully?

A5: We would like to clarify that in Sections 5.1 and 5.2, we train the base model on a single dataset and evaluate it across several sub-tasks. Therefore, the average performance should be the primary focus, as our method achieves significant improvements in both settings. These results sufficiently demonstrate the superiority and robustness of S$^2$FT. It is expected that our method may not always perform the best in every sub-tasks as different sub-tasks will affect each other. Nonetheless, S$^2$FT consistently achieves the best or second-best performance in sub-tasks, which cannot be characterized as "obviously degraded."

To further address your concerns, we have provided more in-depth analyses of the results in Table 1 and Table 2 of the original paper. For the commonsense reasoning tasks in Table 1, S$^2$FT achieves the best performance in most sub-tasks, while maintaining second-best performance in the remaining tasks. These results demonstrate the superiority and robustness of S$^2$FT in memorizing common knowledge.

Table 2 further verifies this conclusion, showing that S$^2$FT outperforms LoRA, especially in memorization tasks such as Writing or Humanities within instruction-following scenarios. It also excels in tasks requiring pre-trained knowledge, such as those in STEM. This indicates that S$^2$FT's performance improvements are primarily due to its enhanced ability to memorize new information and retain pre-trained knowledge. For other tasks, such as Reasoning and Math, which focus on multi-hop reasoning with limited knowledge, the performance of LoRA and S$^2$FT is similar. It is common for LoRA to slightly outperform S$^2$FT in some of these tasks. Consequently, SFT-based methods are more effective for knowledge-rich tasks. In comparisons among SFT-based methods, LISA performs slightly better than S$^2$FT in some tasks. This is because LISA updates much more parameters than S$^2$FT, resulting in similar behavior and better performance in certain tasks.

Thank you so much for your dedicated review of our paper. We recognize the significant time and effort involved in your review, and we greatly appreciate it. With only 2 days remaining before the conclusion of the discussion phase, we wish to extend a respectful request for your feedback about our responses. Thank you!

Thank you for your constructive comments and suggestions. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.

Q1: Novelty in the pruning technique itself.

A1: Thank you for your comment. We acknowledge that structure sparsity is commonly used in model pruning. However, we are the first to adopt structure sparsity for the parameter-efficient fine-tuning (PEFT) of large language models (LLMs). Therefore, our method is a gradient selection technique instead of a pruning technique. Motivated by the challenges of practical efficiency and scalability in previous SFT-based methods, we use coupled structures for flexible and fine-grained gradient selection, introducing a completely new gradient selection strategy. This idea is both novel and effective in this research line.

Before the era of LLMs, methods like Diff pruning [1] and Fish Mask [2] mainly focused on unstructured selective fine-tuning, as model sizes were not very large. These methods used a binary mask to enable sparse gradient updates during training, which led to large memory footprints and time costs. In the era of LLMs, researchers additionally prioritize the memory efficiency of PEFT methods during training and scalable serving ability during inference, leading to the popularity of LoRA. SFT-based methods like LISA [3] only enable layer-wise selection and have limitations in serving scalability. In comparison, our method addresses these efficiency bottlenecks and revitalizes SFT-based approaches, surpassing LoRA in both performance and efficiency. Given the current research trends in PEFT methods, our approach is novel in this area.

Q2: Will the authors open-source the code?

A2: We strongly agree that open-sourcing the code is critical for reproducibility and supporting future research in this area. We therefore will definitely make our code publically available and results easy to reproduce upon acceptance.

References:

[1] Guo D, Rush A M, Kim Y. Parameter-efficient transfer learning with diff pruning[J]. arXiv preprint arXiv:2012.07463, 2020.

[2] Sung Y L, Nair V, Raffel C A. Training neural networks with fixed sparse masks[J]. Advances in Neural Information Processing Systems, 2021, 34: 24193-24205.

[3] Pan R, Liu X, Diao S, et al. LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning[J]. arXiv preprint arXiv:2403.17919, 2024.

Thank you so much for your dedicated review of our paper. We recognize the significant time and effort involved in your review, and we greatly appreciate it. With only 2 days remaining before the conclusion of the discussion phase, we wish to extend a respectful request for your feedback about our responses. Thank you!

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we will revise our paper according to your suggestions. We would appreciate it if you could let us know whether your concerns are addressed by our response.

Q1: Empirical Results about the overfitting and forgetting issues?

A1: To verify that S$^2$FT can prevent overfitting and forgetting issues, we evaluate its performance in an out-of-distribution (OOD) scenario using MT-Bench, as shown in Table 2 of the original paper. Additionally, we conduct an experiment on arithmetic reasoning tasks, demonstrating that S$^2$FT significantly outperforms LoRA in both near OOD and far OOD settings.

In Table 2, we present the results of training the model on the Alpaca GPT-4 dataset and evaluating it on the MT-Bench benchmark, representing an OOD setting. Our method significantly outperforms both LoRA and Full FT in this task, demonstrating that it leads to much less forgetting and better generalization to new datasets.

To further address your concern, we trained LLaMA-7B/13B on the Math10K dataset and evaluated its performance on three in-distribution (ID) tasks (GSM8K, AQuA, MAWPS), four near OOD tasks (SVAMP, MultiArith, AddSub, SingleEq), and one far OOD task (MMLU). The results are shown in Table R1.

Table R1: Performance comparison between S$^2$FT and LoRA for LLaMA-7B/13B on the arithmetic reasoning tasks and MMLU benchmark.

According to the results, S$^2$FT led to improvements of 4.3%, 3.0%, and 1.4% for LLaMA-7B in far OOD, near OOD, and ID settings, respectively, compared to LoRA. For LLaMA-13B, the corresponding improvements are 5.9%, 2.6%, and 0.6%. As the distribution difference between the training data and test data increases, the performance gap between S$^2$FT and LoRA enlarges, demonstrating that our method is effective in preventing overfitting and forgetting. We will include these results in the camera-ready version of our paper if accepted.

Q2: Experiments on larger models.

A2: Thank you for your suggestion. Following LISA, we have added experiment results for LLaMA2-70B on MT-Bench and GSM8K in Table R2. The results show that S$^2$FT outperforms other PEFT methods for larger models, providing strong evidence to support S$^2$FT’s scalability under large-scale training scenarios. We will include these results in the camera-ready version of our paper if accepted.

Table R2: Performance comparison between different methods for LLaMA2-70B on MT-Bench and GSM8K.

Q3: The name SFT may incur some confusions as it already has the meaning of supervised fine-tuning.

A3: Thank you for your suggestion. We will replace "SFT" with "sparse FT" in the camera-ready version to avoid any confusion.

Q4: The experiment parts are not sufficient.

A4: In the original paper, we included experiments with five different base models and more than ten tasks, covering both commonsense reasoning and instruction-following tasks. We have also added some new experiments in Table R1 and R2. We hope these findings provide more comprehensive results of our method, and we will include them in the camera-ready version of our paper if accepted.

In Table R1, we further present experimental results for arithmetic reasoning tasks, demonstrating the effectiveness of our method across various downstream tasks. The results in near OOD and far OOD settings further highlight its ability to address overfitting and forgetting issues. Additionally, in Table R2, our experiments on LLaMA2-70B showcase S$^2$FT's scalability in large-scale training scenarios.

Q5: What about the allocation in different layers?

A5: Thank you for pointing out this issue. In Section 5.3, we maintain a uniform allocation of parameters across different layers. To further address your question, we have added an ablation study in Table R3 concerning layer-wise allocation. This study includes the following design patterns: (i) Increasing ($n_{i+1} > n_i$): the number of trainable parameters in every layer gradually increases (or remains the same); (ii) Uniform ($n_{i+1} = n_i$: the number of trainable parameters in every layer is the same; and (iii) Decreasing ($n_{i+1} < n_i$): the number of trainable parameters in every layer gradually decreases. The results in Table R3 indicate that maintaining a uniform distribution of trainable parameters across different layers leads to the best performance. A more detailed analysis in this direction will be left for future research.

Table R3: Performance of different allocation strategies on commonsense reasoning tasks for LLaMA-7B.

Q6: Can this paper be combined with quantization just like QLoRA

A6: Thank you for highlighting the potential of combining S$^2$FT with quantization. Our method can indeed be integrated with quantization, similar to QLoRA, by using mixed precision storage. Once the trainable parameters are determined, we retain these parameters in their original precision while quantizing the other parameters to low bits. This approach enables quantized PEFT, similar to QLoRA. By maintaining the trainable parameters as small, dense submatrices after permutation, our storage remains relatively hardware-efficient, even with mixed precision. We plan to conduct more experiments in this direction in the future.

Thank you so much for your dedicated review of our paper. We recognize the significant time and effort involved in your review, and we greatly appreciate it. With only 2 days remaining before the conclusion of the discussion phase, we wish to extend a respectful request for your feedback about our responses. Thank you!

Thank you so much for the insightful and valuable comments! They are very helpful for further improving the clarity and quality of our paper. We'll revise our manuscript in the camera-ready version to address all of your concerns.

Q1: How can we verify the model adaptation ability in real-world settings?

A1: To verify the model's adaptation ability in real-world settings, we evaluate its performance in an out-of-distribution (OOD) scenario, as shown in Table 2 in the original paper. Our method, S$^2$FT, significantly outperforms both LoRA and Full FT by a large margin in this task, demonstrating its superior model adaptation ability. Additionally, we conduct an experiment on arithmetic reasoning tasks to demonstrate that S$^2$FT significantly outperforms LoRA in OOD settings (See Empirical results for generalization task part from rebuttal to all reviewers).

Q2: Ablation study between SFT and S$^2$FT.

A2: Thank you for your suggestion. Our results in Table R2 show that SFT primarily leads to performance improvement, while our structure enhances efficiency and scalability, as discussed in Section 6 in the original paper.

Table R2: Performance comparison between S$^2$FT and SFT on the commonsense reasoning tasks.

Q3: Reproduction of the paper.

A3: We strongly agree that open-sourcing the code is critical for reproducibility and supporting future research in this area. We therefore will definitely make our code publically available and results easy to reproduce upon acceptance.

Q4: How are the hyperparameters chosen?

A4: Thank you for highlighting the importance of detailing our hyperparameter configuration. We have included a more comprehensive list of the hyperparameters used in this paper in Tables R4 and R5. Most hyperparameters follow previous work [2, 3], and we only tune the learning rates.

Table R4: Hyperparameter configurations of S$^2$FT for LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B on the commonsense reasoning tasks.

Table R5: Hyperparameter configurations of S$^2$FT for LLaMA2-7B and Mistral-7B on the instruction-following task.

Q5: How to identify the important weights?

A5: In Section 5.2, we discuss our weight selection strategy. For a standard transformer architecture, we first uniformly allocate the trainable parameters across different transformer layers. Next, we freeze all attention modules and evenly assign parameters to the Up, Gate, and Down projection modules. Since these three modules represent a coupled structure within the FFN module, we randomly select the same channels to update for each module. In Table 3, we also introduce an alternative activation-based selection strategy, which resulted in inferior performance. We leave the exploration of more advanced metrics to identify important weights as a topic for future research.

Q6: How to apply S$^2$FT to different models?

A6: Thank you for your interest in the application of S$^2$FT across different model architectures. It is indeed versatile and can be applied to Transformers, CNNs, RNNs, and GNNs.

As detailed in Section 3.2 of our work, S$^2$FT can be utilized in the Multi-Head Attention and Feed-Forward Network modules of standard Transformer architecture, which is sufficient for most LLMs and diffusion models currently. Additionally, such coupled structures in Figure 3(a) also exist in CNNs, RNNs, and GNNs as shown in [1]. Therefore, S$^2$FT is effective across various model architectures.

Q7: Most of the experiments were performed on LLaMA and LLaMA2. The reviewer wonders if there is any difficulty in applying the method to LLaMA3.

A7: In Table 1, we present the results of LLaMA3-8B on commonsense reasoning tasks following DoRA and ReFT. S$^2$FT achieves an average improvement of 1.4%, demonstrating its applicability to LLaMA3. For other experiments, we primarily focus on LLaMA and LLaMA2 to ensure a fair comparison, as these base models are commonly used in our baseline methods.

Q8: Could the authors make some comments on the confidence intervals for the tables in the paper?

A8: Thank you for your suggestion. We have added the full results with confidence intervals in the Appendix, and included comments in the tables in the main paper, which will be visible in the camera-ready version.

Q9: Could different datasets/tasks result in a different set of "important weights"?

A9: See The affect of selecting different trainable parameters on different datasets/tasks part from rebuttal to all reviewers.

References:

[1] Fang G, Ma X, Song M, et al. Depgraph: Towards any structural pruning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 16091-16101.

To facilitate reproduction, we have provided our code at https://anonymous.4open.science/r/S2FT_Rebuttal-7B17 for reviewer’s verification and will make it publicly available upon paper acceptance.

This repository contains the training and inference code necessary to fine-tune a LLaMA-7B model on commonsense reasoning tasks. We hope this addresses your reproduction concerns.