DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models

We thank the reviewer for the thoughtful feedback and for highlighting our detailed analysis of DARE, the development of two improved methods, strong theoretical and experimental validation, clear presentation, and consideration of different scenarios. Please see our responses below, which we hope will effectively address your concerns.

----Q1: Rationale for using different regularization strength instead of a unity strength -------

Answer: Thank you for your question. As mentioned in lines 862-867 in original submission (lines 909-916 in revised version), the different ranges of regularization strength were chosen to align with specific pruning rate requirements, allowing us to optimize performance for both moderate and extreme pruning levels. To elaborate, in cases of moderate pruning (e.g., p=0.99), a moderate regularization weight provides optimal results. For example, in Figure 6(c), a regularization weight of 0.05 achieves the highest performance at this pruning rate. However, for more extreme pruning demands, such as p=0.999, a stronger regularization weight is necessary to maintain performance. As shown in Figure 6 (d), a weight of 0.1 not only achieves the best performance for p=0.999 but also outperforms the non-regularized baseline by approximately 80%.

----Q2 : Experimental Setup----

Answer: Thank you for your question. We suspect that you are referring to the different regularization strengths shown in Figure 6, as Figure 4 is the flowchart; please correct us otherwise. Assuming that, then: Yes, in our experiments examining the effects of regularization strength, we kept all other hyperparameters the same, adjusting only the regularization weight.

---Q3: Combining LoRA with AdamR-----

Answer: Thank you for your question. Following your suggestion, we conducted experiments on LoRA with AdamR-$L_2$ and present the results of DARE applied to both standard LoRA tuning and AdamR-$L_2$ regularized LoRA tuning for SST2, COLA, MRPC, and STSB at a pruning rate of $p=0.99$.

It is shown that the AdamR-$L_2$ improved the performance. Thanks again for the suggestion. We have incorporated these new results into the Appendix G of our revision.

-----------W5: Dependence on Regularization. ----------

Answer: We believe that AdamR is a viable method for a wide range of fine-tuning tasks. Fundamentally, AdamR operates similarly to traditional weight decay; instead of penalizing deviations from zero, it penalizes deviations from the original weights. Given that weight decay is a standard practice in modern network training and fine-tuning, we extend this trust to AdamR. Furthermore, as evidenced by the results presented in Table 5, AdamR not only enhances the performance of pruned models but also shows promise in unpruned models in certain experiments. Additionally, the experimental results from previous answers on computer vision tasks reinforce the applicability of AdamR to other domains as well.

-----------Q3: Impact of AdamR on Training Dynamics--------

Answer: Good question. We have not observed any stability or convergence issues in the experiments reported in the paper. Motivated by your question, we empirically inspect the training loss curve of AdamR and compare it with the standard training with weight decay. To be specific, we finetune a pretrained BERT-base model on MRPC, COLA and STSB datasets by using weight decay and AdamR respectively, then we plot the training loss curves for three fine-tuning tasks under different regularization strengths.

We report the results in Figures 11 and 12 in the revised manuscript and summarize the observations below:

Similar training dynamic to weight decay: When using a reasonable regularization strength, as shown in Fig. 11, AdamR maintains stability and convergence on par with standard weight decay. This supports that AdamR having similar convergence and training dynamics.

Accommodate Large regularization: We further use a large regularization weight and demonstrate the results in Fig 12. The results show that AdamR maintains stable training under a large regularization strength, whereas standard weight decay struggles and fails to converge.

We hope these additional results address your concern effectively.

We sincerely thank the reviewer for their positive and detailed feedback on our work. We are pleased that you found our contributions, including DARq and AdamR, to be innovative and impactful, and that our analysis, empirical validation, and practical recommendations were appreciated. Please see our responses below to your questions. We hope these address your concerns.

-------------W1: Method Complexity---------

Answer: Thank you for your question.

As noted in the appendix (lines 1064–1066 in original submission, lines 1111-1113 in revision), tuning q is designed to be an efficient and lightweight process. It requires only a single batch of data and focuses on adjusting a single parameter, ensuring simplicity and ease of implementation. This approach minimizes both data handling and computational demands, making the overall process efficient.

Furthermore, the associated pruning step is performed only once, which means its computational cost is fixed and does not impact the inference process. This minor complexity enables us to effectively minimize performance degradation for pruned models.

--------- W3&Q1: Scalability and Applicability --------

Answer: Thank you for your great suggestion. We extended our experiments to evaluate the scalability and applicability of our method in the computer vision domain and with a larger 13B model.

Computer Vision: Following your suggestion, we applied our method to computer vision tasks using an ImageNet-21k pretrained Vision Transformer (ViT) model, fine-tuned on the CIFAR-100 dataset. As shown in the table below, our approach achieves a performance boost of over 50% at a pruning rate of 0.99, demonstrating its effectiveness beyond the NLP domain.

Larger 13-B Model: We further evaluated our method on a larger Llama2-13B model fine-tuned on the MetaMath dataset [1] and tested its performance on GSM8K. Despite computational limitations that prevented us from fine-tuning this 13B model within the limited rebuttal phase, DARq consistently delivered superior pruning performance across both pruning rates (0.9 and 0.99). As evidenced in the results below, it achieved a performance boost of over 44% at a pruning rate of 0.99. The consistent improvements observed across various tasks and model sizes strongly suggest that our method is scalable to even larger models.

Thanks again for the suggestion. We have incorporated these new results into the Appendix G of our revision.

[1] MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models

------------Q2: Automated Selection of Rescaling Factor in DARq and rely on a validation set .---------

Answer: Thank you for raising this insightful question. We fully acknowledge the importance of automating the selection of the rescaling factor in DARq to reduce manual intervention.

Solutions without validation data: As outlined in lines 253–255 of the original manuscript (lines 255-258 in revised version), the optimal $q$ can be determined by monitoring the mean absolute output changes on a single batch of arbitrary text data. This method minimizes reliance on labeled validation datasets, enhancing its practical utility.

Feasibility of Auto Search Algorithm: Our results in Figure 1 show that both the output change and test performance have a convex/concave-like relationship with the inverse rescaling factor $q$. This characteristic enables systematic search methods, such as the golden section search algorithm, which has logarithmic complexity $O(\log n)$, to efficiently identify the optimal $q$. We have adopted this approach in all our implementations to automate the selection process and will incorporate a detailed description of this automated selection process in the camera-ready version.

We hope this helps address your concern and provides a clearer point regarding automation, and we appreciate your thoughtful feedback.

We thank the reviewer for your time and highlighting our analysis of DARE’s limitations, the theoretical proof, and the robustness and applicability of DARq and AdamR across various models and datasets. Please see our responses below, which we hope address your concerns.

--------W1: Compare with importance-based methods. ---------

Answer: Great question: this was actually one of our own motivating questions. We have indeed adapted importance-based methods (such as magnitude pruning and the more recently introduced WANDA) for delta parameter pruning (see App. A). We discuss Importance-based DPP in Sec. 3.3 (lines 217-221 in original/revised submission) with detailed experimental results in the appendix (Tables 7, and 12). Overall, our methods significantly outperform importance-based DPP methods. To illustrate this, we summarize the results for the BERT model at a pruning rate of p=0.99 in the table below:

As shown in the table, our methods outperform importance-based methods, such as MP and WANDA, by a significant margin.

Similarly, for decoder models (Table 12 in Appendix): at p=0.95 on GSM8K, DARq outperforms WANDA and MP by over 30% across all models. At p=0.99, importance-based methods fail dramatically while our approach maintains strong performance.

-----------Q1: Elaboration on Table 3---------

Answer: Thank you for your question. Our experimental findings show that while local rescaling consistently outperforms global rescaling for encoder models (as noted in Table 2), the situation is more nuanced for decoder models. In decoder models, the performance gap between methods is relatively small (on average ~0.7% difference for q_v), with local rescaling maintaining an advantage in certain cases.

Based on these findings, we recommend global rescaling for decoder models as stated in lines 358-359 in original submission (lines 359-360 in revision), since it offers implementation simplicity and can outperform local rescaling, while maintaining minimal performance gap when it underperforms.

While we conducted extensive experiments to document these comparative performances, we acknowledge that the theoretical explanation for this encoder-decoder difference remains an open question for future research, and this is partly why we prioritized providing comprehensive empirical evidence to guide practical implementation choices.

-------------Q2: Higher pruning rate for AdamR--------

Answer: Thank you for your suggestion! Yes, AdamR-L2 enables effective pruning even at extremely high rates, as evidenced in Figures 6 and 7, where it supports delta parameter pruning rates of up to 0.999. Specifically, for the RoBERTa-COLA cell you referenced, Figure 6(c) (green line) illustrates a performance of 31.07 at this pruning rate, highlighting its robustness even under highly aggressive pruning scenarios.

Thank you for your time and thoughtful comments. We appreciate your interest in the good performance of DARq and the additional insights our paper brings. Please see our responses to your questions below. We hope these address your concerns and look forward to your updated feedback! We are happy to address any additional questions that you may have.

--------W1: Exploratory and Recommendation--------

Answer: Thank you for your comment, and we would like to take this opportunity to clarify that we have indeed provided recommendations.

Our recommendation : We have made recommendations and presented a flowchart in Figure 4 to assist practitioners. Specifically, this flowchart delivers a clear message that one can customize the pruning strategies to meet different requirements and needs (e.g., scenarios with/without fine-tuning or validation data). Instead of a single solution, we believe our recommendations better align with the varied needs of practical applications. The utility and flexibility of this framework have been recognized in other reviewers’ comments (e.g., reviewers S9Bb and 3h3b), affirming its value in addressing the diverse needs of practical applications.

Comprehensive analysis: Also, we respectfully believe that ‘exploratory’ is not a bad thing, particularly when it provides a unifying/broader viewpoint on a topic that, as the reviewer says, is of interest and not yet well-addressed in the literature. We aimed to approach the problem with a wide perspective that kicks off from the performance of vanilla DARE and identifies its limitations. We address these limitations by considering the circumstances of application (e.g., whether fine-tuning is possible) while investigating other candidate approaches to Delta Parameter pruning (e.g., importance-based methods). The study provides concrete insights, which have also been acknowledged by other reviewers including S9Bb, xpsx, and 3h3b.

-------- W2: Tightness of our bound ---------

Answer: Thank you for your comment, we respectfully disagree with this assessment. The upper bound in Theorem 3.1 is indeed tight with respect to p, which can be seen as follows:

From Eqn (2), the output change can be expressed as: $h = \sum_{j\in[n]} c_j (1 - 1/(1-p) \delta_j)$ where $\delta_j$ are iid $\text{Bernoulli}(1-p)$ RVs, we have dropped the subscript $i$ from Eqn (2) for simplicity and we have denoted $c_j=\Delta W_j x_j$. Our goal is to quantify the growth of the absolute output change $|h|$. Note that the random variables $Y_j= c_j (1 - 1/(1-p) \delta_j)$ have $E[Y_j]=0$ and $Var[Y_j]=c_i^2p/(1-p)$. Thus, by the central limit theorem, in the limit of large $n$, the (normalized) output change $(1/\sqrt{n})|h|$ is distributed as $\sqrt{p/(1-p)}\sqrt{\sum_i c_i^2}|G|$ where $G$ is a standard normal. Note the scaling $\sqrt{p/(1-p) \sum_i c_i^2}$ precisely matches the bound of Thm 3.1 for $p>1/2$, demonstrating that it captures the true behavior of the output change with respect to $p$. The exponential high-probability bound of Thm. 3.1 follows by a Hoeffding-type bound, but as explained in App. E.1, to achieve this optimal p-scaling, we use a bound by Berend and Kontorovich that improves upon Hoeffding’s bound on the moment generating function for generalized Bernoulli RVs.

We acknowledge the reviewer's point about precise language and have revised line 229 in revised submission to include "at most" for technical precision.

Finally, we emphasize that beyond the theoretical foundation of Thm 3.1, which successfully motivates our improved DARq rescaling, our extensive experiments on encoder- and decoder-based language models across multiple datasets demonstrate that DARq consistently outperforms state-of-the-art methods.

----------W3: Typo--------

Answer Thank you for pointing out this typo, we have corrected it in our revised version.

Thank you again for your time!