Expanding Sparse Tuning for Low Memory Usage

We would like to thank you for the detailed comments! We will diligently follow your guidance to further improve our work and manuscripts.

W1: The utilized kernel function.

Thank you for your suggestion! We will introduce the utilized kernel function in Section 3.2 in the revised paper.

W2: Implementation Details of Figure 4(a).

Thank you for your advice! We will introduce the location of implementation details in the caption of Figure 4(a) in the revised paper.

W3: Why memory savings of SNELL are more significant for larger models?

1. The memory saving mechanism: As Fig. 3(c) shows, the memory savings of SNELL are attributed to the reduction of learnable parameters stored in the optimizer. For a matrix $\mathbf{W}\in \mathbb{R}^{n\times n}$, the memory usage of SNELL-$r$ only takes a proportion of $\frac{2}{n}r$ to the full fine-tuning.
2. Memory significance for large models: As the model size increases, the value of $n$ and the number of weight matrices increase, making the parameter reduction becoming increasingly significant.

In the revision, We will introduce the memory-saving mechanism of SNELL in the Appendix.

W4: Comparison with MoSA and VQT.

Thank you for mentioning the two methods. We follow your suggestion and further conduct comparisons with MoSA and VQT on the VTAB-1K dataset.

The results demonstrate the superiority of our SNELL over these methods, and we will add these comparisons in the revision.

Thank you very much for your valuable comments that help us provide better clarification and explanation of our work! We hope the following responses can address your concerns.

W1: Competition-based Sparsification Mechanism Clarification

Thank you for your feedback. We now provide a more detailed explanation of the competition in sparsification regarding intuition and computation.

1. The Primary idea of the "competition" is the dynamic threshold rather than a soft-threshold function. Specifically, we utilize the quantile of values in $\Delta W$ as a dynamic threshold to control sparsity. This ensures that only a fixed proportion of weights remain non-zero. Therefore, the weights have to "compete" with each other to be selected instead of just having a larger value than a threshold.

2. Competition in the computation. In the forward computation of $\Delta W$, weights are zeroed out based on their ranks in the competition. However, the competition actually happens in backward computation where weights are updated based on gradient descent. Weights that contribute more to the task will gain more significant value and thus win the competition.

In the revision, we will incorporate the above clarifications into Section 3.2.

W2: Comparison between SNELL and LoRA

Thank you for your valuable suggestions that help us better express our novelty.

Compared to LoRA, our improvement is a significant increase in performance. We provide comparisons between SNELL and LoRA in terms of performance and memory usage.

1. Our Performance Improvement over LoRA primarily arises from the combination of kernelized LoRA and the sparsification mechanism.

   • LoRA with kernel functions achieves performance improvement (KLoRA in Table A8).
   • LoRA with sparse tuning shows performance degradation (Table 4(a)).
   • SNELL with both kernel functions and sparse tuning achieves better performance than LoRA (Tables 1, 2, and 3) and KLoRA (Table 4(b)). This is because the higher ranks of the weight matrix in KLoRA better support the sparsification process (L166-174).

2. Comparable Memory Usage. For memory reduction, SNELL shares the same mechanism as LoRA, which only stores the small low-rank matrices as learnable parameters (in Figure 3(c) and L275-278). The little memory increase of SNELL compared to LoRA is because of the nonlinear kernel functions. We provide a memory usage comparison of SNELL and LoRA. The impact of the kernel function on memory usage becomes increasingly negligible as the model size grows.

   Pre-trained Model	Mem. LoRA	Mem. SNELL	Mem. Delta / Mem. LoRA
   ViT-B/16	1546	1673	0.082
   ViT-L/16	4325	4519	0.045
   ViT-H/14	9325	9692	0.039

In the revision, we will incorporate the above explanation and comparison in the experiment section.

W3: Typos

Thank you for your advice! We will change Line 6 from "by increases" to "by increasing" in the revision.

Q1: High Memory Usage of Previous Sparse Tuning Methods

Thank you for your careful reading of our paper, we will further clarify L47-48.

1. Indeed, Pytorch supports storing only learnable parameters in the optimizer, but these learnable parameters must be stored in the form of structured matrices.
2. Unfortunately, sparse tuning involves selecting a variable number of parameters at different locations in the matrix, which is unstructured and makes it impractical to store only the tunable parameters in the optimizer.
3. Thus, current methods have to treat the entire weight matrix as a learnable matrix and incorporate a mask into the gradient during backpropagation. This strategy solely implements sparse updates of the matrix and is not superior to dense updates in terms of memory usage.

In the revision, we will emphasize the relation between the unstructured nature of sparse tuning and its high memory usage.

Thank you very much for your feedback, it has been very insightful for our work. Before addressing your questions, we would like to clarify some misunderstandings regarding Weaknesses 1, 3, and Limitation. All tables are presented in the PDF response due to the limitation of character number.

W1: Novelty of SNELL

We want to clarify that our contributions, inspired by the insights on neurodynamic systems and neuroscience, are essentially different from previous methods.

1. New perspective on LoRA. Inspired by the neurodynamic system (L75, L173-175), we interpret matrix vectors as coordinates in a dynamical space and endow the space with complex distances to improve expressivity. The complex distance can be measured by kernel functions. Therefore, our contribution is providing a new perspective for modeling weight matrix during fine-tuning, far more than just using kernel tricks. In addition, we introduce nonlinearity to LoRA, representing a fundamental departure from other linear extensions [R1].
2. Novel sparsification is inspired by the neuron competition phenomenon (L64-66). Instead of a soft threshold function, we propose the dynamic threshold, a method that selects weights based on their ranking and induces weight competition during end-to-end fine-tuning. This distinguishes our approach from others which select weights based on their values and employ a soft threshold as a parameter [R2] or increase using prior rules [R3].

In the revision, we will detail the theory behind SNELL and clarify our differences from other methods.

W3: Tunable Parameters Volume

We acknowledge the importance of this metric and will try our best to estimate it. Before reporting estimations, we want to stress that the difficulty in computing this metric does not stem from the sparse-tuning but from the combination of LoRA and sparse-tuning (L644-656).

1. Difficulty of computing this metric: In (kernelized) LoRA, this metric counts elements in the two low-rank matrices that are used to recover weight matrices. However, the sparsification in SNELL requires tracing from the recovered matrix to the low-rank matrices and locating the sparsified elements, which is difficult.
2. Volume Estimation: Please see Table R1 in the response pdf for details.
3. Memory usage is more demanding when fine-tuning large models. A PEFT method can require fewer tunable parameters but still more memory usage, such as pure sparse-tuning with larger sparsity compared to SNELL.

Limitation: Applicability of SNELL and Problems in Pure Sparse-tuning.

1. SNELL is applicable in practice: SNELL only exhibits a very marginal increase compared to LoRA in training time (Table A9), which does not impede its practical application. This is proved by our implementation of SNELL on large models (ViT-L&ViT-H) in Table A8. Appropriate GPU operators can further benefit the SNELL's training speed.
2. Pure sparse-tuning CANNOT achieve LoRA-level memory usage by recomputing during back-propagation: The low-rank matrices in the optimizer allow SNELL to recompute the large merged matrix during the back-propagation without saving them in the optimizer. In contrast, pure sparse-tuning methods without low-rank matrices have to consistently store large weight matrices in the optimizer.

W2: Comparing with GPS

Thank you for mentioning the important work to us. Regrettably, we did not mention GPS in the submitted draft, partially due to the unavailability of its source code at the submission deadline.

With the published codes, we now provide fair comparisons with GPS in performance (Table R2 in the response pdf) and memory usage (Table R3 in the response pdf). SNELL has performance on par with GPS while requiring lower memory usage.

In the revision, we will discuss GPS as the latest SOTA sparse-tuning method and conduct more comparisons with GPS.

W4: Method Taxonomy

This question is interesting and prompts us to discuss the taxonomy of PEFT methods. We acknowledge the lack of a standard taxonomy in the community. We follow the taxonomy of SPT[R4], classifying based on whether using additional modules during inference or not. This differs from Delta-tuning[3] according to the training mechanism. Additionally, we compare SNELL with SSF[2] (Table R4 in the response pdf). SNELL demonstrates better performance.

In the revision, we will provide a detailed discussion of this taxonomy and add the comparison results.

Q1: Sparse-Tuning without KLoRA

Your question is insightful and has inspired us to explore the relationship between low-rank properties and performance. Please see Table R5 in the response pdf for comparisons between SNELL and pure sparse-tuning. SNELL achieves better performance than pure sparse-tuning.

Q2: Parameter and Memory Savings of Our Sparsification

We want to clarify that our saved memory usage in weight indices does not reduce the tunable parameter volume. These indices are saved as constant float tensors (not parameters) to multiply with gradients during fine-tuning. Compared to the pure sparse-tuning method, which stores an index for each parameter, our sparsification method saves memory proportional to the model's parameter volume. Therefore, the more parameters a model has, the more memory our method saves.

In the revision, we will discuss the memory-saving effect of our sparsification mechanism.

Q3: Fine-tuning with fixed sparse matrix

Your question is very enlightening for us to discover more advantages of our sparsification mechanism. Please see Tables R6 and R7 in the response pdf for details. The performance of our sparsification mechanism surpasses that of fixed sparse matrices.

[R1] Krona, arXiV'22.

[R2] STR, ICML'20.

[R3] ST-3, WACV'23.

[R4] SPT, ICCV'23.

[R5] ETLSRR, AAAI'23

Thank you for the insightful comments that help us extend the applicability and strengthen the novelty! We will diligently follow your guidance to further improve our work.

W1: Additional Results on Large Language Models.

Following your suggestion, we apply SNELL on LLaMA2-7B to adapt to the commonsense reasoning benchmark.

Experiments: We compare SNELL with LoRA and find that SNELL achieves a better performance. This shows the applicability of SNELL to NLP tasks. Many other vision PEFT approaches lack this capability, as they necessitate a full level of memory usage for fine-tuning as Figure 3(a) shows.

In the revision, we will further explore the applicability of SNELL on different LLMs and benchmarks and update the experiment results.

W2: SNELL based on Pre-defined Fixed Masks.

Thank you for your suggestion. To address your concern, first, we respectfully clarify that dynamic masking can preserve the low memory usage and the ability to serve multiple adapters of LoRA.

• Memory usage: SNELL achieves comparable performance with LoRA (Figure 3(b)).
• Ability to serve multiple adapters: Our masking is determined by the learnable low-rank matrices, so it is dynamic during fine-tuning and deterministic after fine-tuning. Therefore, compared with LoRA's reparameterization process, SNELL only adds a fixed mask. Both LoRA and SNELL can facilitate multiple adapters.

Second, by incorporating pre-defined masks, it is indeed possible to retain LoRA's quick training speed.

1. Experiments: We first provide the training time of kernelized LoRA with pre-defined fixed masks (KLoRA-8-Fixed) and SNELL.

   Method	K-LoRA-8-Fixed	SNELL-8
   Training time (s/img)	0.629	0.657

   Then we compare the performance on FGVC datasets between kernelized LoRA (KLoRA-8-Fixed) with pre-defined fixed masks, and SNELL. The fixed masks are generated by SPT [R1].

   Method	CUB-200	NABirds	Oxford Flowers	Stanford Dogs	Stanford Cars	Mean
   KLoRA-8-Fixed	88.0	82.1	99.0	89.4	88.4	89.4
   SNELL-8	89.0	83.9	99.3	90.6	88.6	90.3

2. Analysis: We find that fine-tuning with fixed masks can improve training speed (0.629 vs. 0.657).
   However, compared to our dynamic masking, fixed masking can hardly identify and adjust the most task-relevant weights in an end-to-end fashion, which leads to performance degradation (89.4 vs. 90.3).

Nevertheless, we believe that quick task switching is worth studying, and we will keep exploring it in the future.

We will incorporate the above experiments and analysis in the revision.

[R1] Sensitivity-aware visual parameter-efficient fine-tuning, ICCV'23.