Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning

The author should not claim as "for free", as with sparsity = 0.5 would often need storage that can be more than that with no sparsity. Also, additional tensor sparse operation is needed making the latency to grow a little up, as it is not really N:M sparsity. So, please tone down on your claim, title, and writing.

Thank you so much for your comment. We apologize for our previous unclear expression, and we would like to further explain as follows.

1. Inference phase. Based on Equation 4 and 5 in manuscripts, we can combine parallel weights into one and get $B^* = B\odot m_{b_1}+...+ B\odot m_{b_d}$ , and $A^* = A\odot m_{a_1}+...+A\odot m_{a_d}$, where both $B^*$ and $A^*$ have the same size as corresponding original PEFT methods. Thus, CapaBoost inference phase is ‘for free’, as there is no extra storage or computational cost.
2. Fine-tuning phase. Our CapaBoost implementation is for free because of the fact that
   • As illustrated in response to question 2 and question 5, our parameter number is less than corresponding PEFT methods without CapaBoost. Mask storage can also be avoided through same random seed regeneration, and thus the storage cost is smaller than original PEFT methods.
   • As shown in Figure 4 and Figure 5, a 2-layer CapaBoost with sparsity=0.5 is enough to achieve competitive performance, which is also the setting we used in most experiments presented in the paper(Table 2,3,4 and 5). With this setting, the theoretical FLOPs of CapaBoost are as same as original PEFT methods. With our newly implemented N:M sparsity random masks, we can enjoy Nvidia sparse tensor core acceleration and realize comparable FLOPs compared to original PEFT methods in reality. Therefore, our CapaBoost implementation in experiments is also nearly for free.
   • We also acknowledge that CapaBoost is a general framework, its FLOPs depends on selection of layer number and mask sparsity, and might be more or less than corresponding PEFT methods without CapaBoost. However, our implementation in the manuscript’s experiments is nearly for free.

3. The authors claims of speeding up the sparse operation via Nvidia sparse tensor core, however, that is only applicable for N:M sparsity, which the current sparse mask generation process does not satisfy.

Thank you so much for pointing out this problem. We now kindly include the GLUE test results with RoBERTa-base as the pre-trained model and inner dimension = 8, and consider two masking strategies: fully random masks and N:M sparsity random pruning masks. These results are detailed in the table below.

The results indicate that CapaBoost performs similarly with both fully random masks and N:M sparsity random masks. Thus, we can directly replace fully random masks with N:M sparsity random masks and benefit from the hardware acceleration provided by the Nvidia sparse tensor core. In our experiment, the fine-tuned weight matrix $W$ with N:M sparsity displays the same rank as $W$ with fully random masks. Therefore, it’s not surprising that the result is similar.

4. Can you explain with an example how with sparsity of 0.5 you can have more low rank matrices? And at what point it surpasses the params for a scenario when the sparsity is 1.0 (meaning that of Eq. 2)

Thank you for the question.

In the manuscript's Equation 2 (not CapaBoost), since $W_1,W_2...W_d$ are all distinct and don’t derive from a common matrix, the total parameter number is $d$ times than parameter number of $W$.

However, in CapaBoost case, as explained in the 2-layer CapaBoost example of question 2, with two different random masks $m_1,m_2$ applying to the same $W$, we can obtain two different weight matrices $W\odot m_1, W\odot m_2$. Similarly, in the $d$ layer of CapaBoost, with $d$ different random masks, we can obtain $d$ diverse sparse weight matrices, where all matrices elements are a subset of dense matrix $W$. The parameter number in CapaBoost will never exceed $W$ in the original $z=WX+b$ since all sparse weight matrices originate from $W$. Thus, our maximum parameter number is the same as $W$.

5. Isn't CapaBoost with d>2 would require more fine-tuning params with s=0.5?

Thank you for the question. Suppose the original matrix $W$ has a parameter number $n$, for the $d$-layer CapaBoost with sparsity of $s$, if all pruning masks are randomly generated, the parameter number is $(1-s^d)*n$ according to the binomial distribution. If $s=0.5, d=2$, we only need 75% of original parameter number. With more layers, CapaBoost will require more fine-tuning parameters, but it will never exceed $n$.

When the author says training phase with the random generated frozen mask, does he/she mean fine-tuning phase?

Yes, training phase refers to fine-tuning on the pre-trained model.

Dear reviewer LzY5:

Thank you very much for the review and feedback, we kindly address your questions as follows.

1. This sentence is not clear, "Furthermore, experiments conducted in Lialin et al. (2023) reveal that transformers with low-rank updates significantly underperform compared to full-rank baselines during pre-training.".. does it mean the Low rank adaptation during fine-tuning perform poorly? If so, how the LoRA or similar models perform close to baseline full-model fine-tuning? I suppose event the foundation of this paper, assumes that low rank adaptation is useful, and then tries to improve their rank. So, please clarify here.

First we apologize for the previous inaccurate expression and have since updated the paper.

We do acknowledge and agree that low-rank adaptation is a useful and efficient fine-tuning method capable of reaching or even exceeding full-model fine-tuning performance in certain cases. However, it's important to note that it can underperform when the fine-tuning weight rank is insufficient. This is supported by Figure 4 in [a], Figure 2 in [b], and our Figure 4(b) and Figure 5. By increasing the rank with CapaBoost, we can enhance the performance of low-rank adaptation.

2. Generation of mask randomly does increase storage cost, as we need to store the index location of the mask while performing forward and backward pass. So, it being storage free is not really a correct statement.

First we apologize for the misunderstanding we caused. We have revised our statement in the manuscript with the blue color.

During fine-tuning stage, the pruning mask $m_1,m_2$ in CapaBoost are randomly generated and fixed, ideally, we only need to store random seed used to generate masks, rather than saving pruning mask index. Each time during forward and backward pass, we regenerate pruning mask using same random seed and discard masks after use, which may only induce temporary storage.

Additionally, we would also like to emphasize that the dense fine-tuned weight matrix $W$ in PEFT methods can be partially pruned.

Starting from Equation 3 in manuscript, we use an example to demonstrate this point. Suppose we have a 2-layer CapaBoost framework with 50% sparsity, which can be written as $z=(W\odot m_1+W\odot m_2)X+b$,

we have $X=[x_1,x_2,x_3,x_4]^{T}$, $W=\begin{bmatrix}w_{11}& w_{12}& w_{13}& w_{14} \\\ w_{21}& w_{22}& w_{23}& w_{24} \\\ w_{31}& w_{32}& w_{33}& w_{34} \\\ w_{41}& w_{42}& w_{43}& w_{44}\\\ \end{bmatrix}$, $m_1=\begin{bmatrix}1& 0& 1& 0 \\\ 1& 1& 0& 0 \\\ 0& 1& 1& 0 \\\ 0& 0& 1& 1 \\\ \end{bmatrix}$, $m_2=\begin{bmatrix}1& 1& 0& 0 \\\ 1& 0& 0& 1 \\\ 1& 0& 0& 1 \\\ 0& 0& 1& 1 \\\ \end{bmatrix}$

since elements $w_{14}, w_{23}, w_{41}, w_{42}$ are pruned in both mask $m_1$ and $m_2$, they are never used in computation. As a result, they can also be pruned in fine-tuned matrix $W$. Therefore, CapaBoost always leads to less trainable parameters. The accurate parameter reduction number can refer to answer for question 5.

Given the reduction in number of trainable parameters and negligible increase in temporary mask storage, our method in general doesn’t require more storage compared to original PEFT methods without CapaBoost.

[a] He, Junxian, et al. "Towards a Unified View of Parameter-Efficient Transfer Learning." International Conference on Learning Representations. 2021.

[b] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2022.

Dear Reviewer LzY5,

Thank you again for your review and comments. Since the discussion period is closing soon, we are wondering whether our responses have addressed your concerns. If not, we would be happy to continue the discussion and provide further clarifications or explanations.

Dear reviewer ZjAd:

Thank you so much for your positive rating and constructive advice. The experiment of ProPETL for Vtab-1k task is running; we will update it to the paper as soon as possible.

Dear reviewer ZjAd:

Thank you for your patience. We kindly include additional benchmarks in the Vtab-1k experiment and present you with the new results in the table below.

1. Visual Prompt Tuning (VPT) [a]. VPT prepends a set of learnable tokens to the input of a Transformer block, which can be viewed as adding some learnable pixels in the input space.
2. NOAH [b]: NOAH is a neural architecture search method (NAS) that learns the optimal design of several parameter-efficient subnetworks, including VPT, LoRA and Adapter. This type of neural search method requires more parameter numbers and computational costs.
3. ProPETL[c]. Since there is no existing implementation for the ProPETL method on Vtab-1k tasks, we reproduced it based on open source code provided by the author and fine-tuned it for optimal results. The results indicate that ProPETL-LoRA offers a performance gain of about 0.2% compared to the original LoRA. This is consistent with the slight performance gain ProPETL-LoRA has in the GLUE dataset.

The results indicate that CapaBoost-LoRA still outperforms all other competitors, demonstrating the effectiveness of the CapaBoost framework.

Note: * indicates the results are reported in prior works [a] and [b] respectively.

Thank you again for your feedback. We respectfully request that you reconsider the review score if we have addressed all of your concerns.

[a] Jia, Menglin, et al. "Visual Prompt Tuning." European Conference on Computer Vision. 2022.

[b] Zhang, Y., Zhou, K., & Liu, Z. (2022). Neural Prompt Search. ArXiv, abs/2206.04673.

[c] Guangtao Zeng, Peiyuan Zhang, and Wei Lu. One network, many masks: Towards more parameter-efficient transfer learning. In Proceedings of ACL, 2023.

Dear Reviewer ZJAd,

Thank you again for your review and comments. Since the discussion period is closing soon, we are wondering whether our responses have addressed your concerns. If not, we would be happy to continue the discussion and provide further clarifications or explanations.

I find it hard to understand / believe Table 1 for multiple reasons. First, the claim is that you generally get a d - 1 increase over LoRA (remark 3.3) but you consistently report dx instead (d - 1)x increase. Second, the basis for the intuition for the dx expansion, is theorem 3.1 which assumes that X and Y are generated independently from Gaussians. In general, this would be the upper bound on the rank of the sum. The matrices you instantiate are not independent (since they are generated from the same B,A combination -- though masked) ... but somehow your results (from Table 1) suggest that you are always achieving this upper bound which is quite surprising. And what about the case where the rank of A, B = 1 ?

Thank you for your question. We apologize for any confusion caused by our previous explanation. In the paper we intended to express CapaBoost’s rank is d times of LoRA’s rank (d-1 times larger), we have revised the manuscript with blue color to clarify this point.

To better address your concern, we conducted a simple Monte Carlo simulation of a 2-layer CapaBoost with rank of $B$ and $A$ $=1,2,8,16,32$. In this simulation, we randomly generate $B$, $A$, their corresponding pruning masks $m_{A1},m_{A2}, m_{B1}, m_{B2}$ and compare rank of $BA$ and $(B\odot m_{B1})(A\odot m_{A1})+(B\odot m_{B2})(A\odot m_{A2})$. The results from 10,000 simulations consistently show that $(B\odot m_{B1})(A\odot m_{A1})+(B\odot m_{B2})(A\odot m_{A2})$ always achieves 2 times rank expansion compared to $BA$, which clearly indicates this rank expansion is not a coincidence. One possible explanation behind is that with different random pruning masks, $(B\odot m_{B1})(A\odot m_{A1})$ and $(B\odot m_{B2})(A\odot m_{A2})$ are of great difference and can be viewed as pseudo-independent. But since the $B,A$‘s distribution after training is unknown, it's challenging to provide a definitive proof.

"Fine-tuning large pre-trained foundation models, such as the 175B GPT-3, has become the prevailing approach for downstream tasks." -- I don't think this is true -- the defacto approach for using models at these scales is prompting and not fine-tuning.

"..... replacing a single layer’s heavyweight matrix w ∈ Rd1×d2 with two consecutive layers possessing much smaller inner dimension r, such as B ∈ Rd1 ×r , A ∈ Rr×d2 . This approach results in a significant reduction in parameter count (evident in Adapters and LoRA), achieving performance comparable to full-size fine-tuning while retaining only 1% of trainable parameters" -- the original matrices are not "replaced" by these smaller ones but this statement makes it seem like the full parameter versions are swapped for the low-rank version

Thank you so much for pointing it out, we have updated our expression in PDF.

How do you measure/compute the flops in your paper ? You only mention that its "relative to lora"

Thank you for the question. We calculate FLOPs needed in each layer based on layer type, size and Sparse core acceleration, and then calculate total number. For convenience, we only show the FLOPs factor relative to that of LoRA

I am surprised that full finetuning from Figure 4b performs substantially worse that the low rank methods. What hyper-parameters were used ? Was there regularization when training this model ?

Thank you for the question you bring out. Figure 4b is an experiment on CoLA dataset. The full fine-tuning hyper-parameters settings follow [b], which is regularized via using weight decay.

The results remain consistent with those in Table 2 and Table 3 in our paper, where full fine-tuning consistently shows significantly worse performance than CapaBoost in CoLA dataset. Similar findings are also reported in [b] and [c]. This may because full fine-tuning contains large number of parameters and tends to overfit CoLA dataset, even with help of regularization.

In the next step, we plan to cover this experiment on the entire GLUE dataset to make the result more convincing.

[b]Guangtao Zeng, Peiyuan Zhang, and Wei Lu. One network, many masks: Towards more parameterefficient transfer learning. In Proceedings of ACL, 2023.

[c]Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2022a.

Dear reviewer 7oiV:

Thank you very much for the review and feedback. We kindly address your questions as follows:

Flop improvement from method only exist in the presence of specialized hardware for sparse matrices.

Thank you for bringing out this question. We fine-tuned the RoBERTa-base model on the GLUE dataset with an inner dimension of 8, and considered two masking strategies: fully random masks and N:M sparsity random pruning masks. We present experimental results in the table below. CapaBoost with N:M sparsity shows comparable performance to fully random pruning masks. These N:M sparsity masks enable us to have Nvidia sparse tensor core (e.g., A100) acceleration and achieve FLOPs improvement in the real world.

Were any of the non-CapaBoost methods regularized? One possible explanation for the improvements of CapaBoost -- is that the random masking creates implicit regularization. It would be good to compare the performance to other methods, once they also enjoy some level of regularization.

Thank you so much for your question. To address this concern, we first include two other variants of CapaBoost: 1) same-mask CapaBoost, 2) Dropout CapaBoost and 3) AdaMix [a] which has explicit regularization.

• In same-mask CapaBoost, we maintain CapaBoost parallel structure and replace the different parallel pruning masks with one identical mask. With this design, same-mask CapaBoost still benefits from implicit regularization but no longer possesses the rank expansion property.
• Similarly, we replace different parallel masks by Dropout in Dropout CapaBoost.
• Explicit regularized method AdaMix [a].

We reported the GLUE test results with RoBERTa-base as the pre-trained model and an inner dimension of 8 in the table below. The results show that CapaBoost can both outperform explicit regularized and implicit regularized competitors. It’s also worth noticing that all non-CapaBoost methods in our paper use regularization method with control of weight decay.

Based on the above observation, we agree regularization may also contribute to CapaBoost’s superior performance, but most of its improvement stems from rank expansion.

Remarks: 1) Some ablation study about same-mask and Dropout are also shown in Figure.4 in the paper 2) The experiment here has a different inner dimension from Table.2 in the manuscript.

[a] Wang, Yaqing, et al. "AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Dear Reviewer 7oiV,

Thank you again for your review and comments. Since the discussion period is closing soon, we are wondering whether our responses have addressed your concerns. If not, we would be happy to continue the discussion and provide further clarifications or explanations.