SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors

We sincerely thank the reviewer for the rigorous review and valuable feedback. We have addressed all the weaknesses (W) and questions (Q) with extensive experiments and clarifications.

W1: Please refer to our global response G1 for a comprehensive answer.

W2:
1, 4. We appreciate you highlighting this oversight. The results marked with * in Table 3 were indeed obtained from BOFT [1].

2. We apologize for not clearly stating the target modules for baselines in the main paper. This information is provided in Appendix C.5. To clarify:

   • For Table 1: LoRA and DoRA adapt QKVUD modules as per [2]; BOFT adapts QV modules (adapting more leads to OOM); VeRA adapts GU modules (maximizing trainable parameters within its compatibility constraints).
   • For Table 3: All methods adapt all linear layers, following [1].

3. The statistics mentioned in lines 14-16 and 46-48 are extracted from Figure 1 (left) for Gemma-2B on GSM-8K. We will update this in the final version for clarity.

W3:
1, 2. We acknowledge that while SVFT reduces trainable parameters, it increases overall GPU memory usage compared to LoRA. As the reviewer correctly noted, reducing trainable parameters decreases memory required for gradients, activations, optimizer state, and various buffers. Following the reviewer's suggestion, we used HuggingFace's internal memory profiler to measure actual peak GPU memory usage. We've included these findings, along with the adapted modules for all baselines (see G2 in global response). In summary, SVFT uses ~1.2x more memory than LoRA but is comparable to or less than DoRA.

W4/Q2: We appreciate the reviewer pointing out the inconsistency in adapted modules. For SVFT, we initially adapted more modules (QKVUDOG) compared to baselines (QKVUD) due to the lower total trainable parameters. However, for LLaMA-3-8B, adapting QKVUDOG exceeds A100-80GB GPU memory (true for baselines as well). Based on our ablation study in Section 5.3, which showed UDOG modules contributing most to performance gains, we chose to adapt UDOG while staying within memory limits. We agree that this inconsistency could make it challenging to attribute gains specifically to SVFT. Following the reviewer's advice, we re-ran experiments with SVFT adapting only QKVUD, matching the baselines. We've included these new results in a table below, demonstrating that SVFT maintains similar performance gains even in this consistent setting.

Gemma-2B

LLaMA-3-8B

W5/Q4:

1. We selected different datasets for ViT-B and ViT-L in Table 4 for two primary reasons: i) To provide a balanced evaluation, we ran ViT-B on relatively simpler datasets (CIFAR100, Flowers102), while testing ViT-L on more complex ones (Food101 and Resisc45). ii) To enhance the table's readability and prevent it from becoming overly large. The complete version of this table, including all dataset-model combinations, is available in Table 9 of the appendix.

2. We acknowledge the reviewer's observation that SVFT variants may not consistently outperform LoRA/DoRA across all benchmarks, as evidenced by the Resisc-45 results with ViT-B. However, it's important to note that these results are still comparable to full fine-tuning outcomes. We will update the main paper to reflect this nuanced finding. Additionally, it's worth considering that in vision benchmarks, the classifier head is fully tunable, which may increase the potential for overfitting.

3, 4. To clarify our methodology: i) We adopted the target modules for adaptation from [1] and the data split setup from [2]. While [1] uses VTAB-1K for vision experiments, they don't release data splits or sources. Upon careful examination of [3], we discovered that unlike other datasets where evaluation is done on the test split, [3] evaluates Resisc45 on all remaining samples. For consistency, we evaluate on the test split for all vision datasets in our study. ii) To address the second point, we have re-run all vision experiments, adapting only the QV modules – please refer to G3 in global response for table, which we will include in the revised paper. In short, SVFT offers competitive performance. Adapting QV modules for these vision tasks seems sufficient for all baselines.

Discrepancy in Food101: Table 5 in [3] shows ViT-B significantly outperforming ViT-L on Food101, contrary to trends in other datasets. We were unable to reproduce these results for ViT-B, even with full fine-tuning.

Q1: See Appendix C.5 L451, or W2.2
Q3: For Math reasoning, we fine-tune on the MetaMathQA-40K (L181) and evaluate on GSM-8K, MATH following [1]. For Commonsense reasoning, We follow the setting outlined in prior work [2] i.e. all the training splits from the respective datasets are amalgamated into the training set (L184). For baselines, we use the recommended hyperparams from the respective papers, and for SVFT, we do not perform hyperparam tuning and select params outlined in Appendix C.5.

References
[1] Liu et al. Parameter-efficient orthogonal finetuning via butterfly factorization. ICLR 2024.
[2] Liu et al. Dora: Weight-decomposed low-rank adaptation. ICML 2024.
[3] Kopiczko et. al: VeRA: Vector-based Random Matrix Adaptation. ICLR 2024

We thank the reviewer for their valuable feedback. We address the weaknesses (W) and questions (Q) raised by the reviewer below.

W1: We thank the reviewer for pointing us to these papers (SAM-PARSER [1], SVDiff [2]). Indeed we are not the first one to explore the singular vectors/values space. [1] and [2] are essentially equivalent to our SVFT$^P$ variant, but we note that our main insight and contribution is that significant gains can be achieved by learning off-diagonal interactions in $M$.

Doing our due diligence, in the following Table, we compare SVFT$^P$ against SVF (which is also equivalent to SAM-PARSER and SVDiff) on GSM-8K and Math benchmarks. Results in the Table below show SVF performs similarly to SVFT$^P$ and worse than SVFT$^R$ as we had expected (the same holds true for SVFT$^B$ and SVFT$^T$ from the table in W2). We will acknowledge these papers and add a discussion in our final version of the paper.

To clarify our contribution, we will revise the highlighted statement to:
SVFT enables a trade-off between the number of trainable parameters and model expressivity by allowing a flexible number of off-diagonal interactions between singular vectors in $\Delta$W, distinguishing it from previous SVD-based methods.

W2/Q3: We note that simple heuristics for sparsity patterns in $M$ significantly improve performance over previous PEFT techniques. While most variants (except SVFT$^P$) offer comparable performance, the banded pattern (SVFT$^B$) slightly outperforms others. We hypothesize that a more principled approach to finding optimal sparsity patterns for specific tasks and base models may yield further performance gains. We leave it as future work to explore task-specific and model-specific sparsity patterns. We will revise our paper to reflect these new results and findings.

Q1: For SVFT$^R$ and SVFT$^T$ variants, we select the number of trainable parameters in each target module's weight matrix to match SVFT$^B$'s parameter count for a given 'k'. This is calculated as $k=n(2d+1) - d(d+1)$, where $d$ is the number of off-diagonals in the banded matrix and $n$ is the matrix's smaller dimension. This is to have a fair comparison between methods. Eg: For $Q$ matrix on Gemma-2B ($n$=2048), when off-diagonals $d=2$, then $k=10,234$ for both SVFT$^T_{d=2}$ and SVFT$^R_{d=2}$ which is the same as that of SVFT$^B_{d=2}$. In practice, ‘$k$’ can be set to any value for these two variants.

Q2: We propose SVFT$^T$ to explore a broad spectrum of possible selection criteria of $M$’s sparsity pattern. Our experiments show that simple heuristics-based sparsity patterns can improve performance. In future work, we will explore more principled approaches for finding the optimal pattern.

Q4: We did not observe any instability in our experiments. In the worst-case scenario, SVFT$^T$ will be equivalent to SVFT$^R$, which performs well in our experiments.

References
[1] Peng, Zelin, et al. SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction, AAAI 2024.
[2] Han, Ligong, et al. SVDiff: Compact Parameter Space for Diffusion Fine-Tuning, ICCV 2023.

1. For $\text{SVFT}^{\mathbf{R}}$ and $\text{SVFT}^{\mathbf{T}}$, $d$ is a meaningless notation. $\text{SVFT}^{\mathbf{R}}_d$ and $\text{SVFT}^{\mathbf{T}}_d$ will be misleading to the reader.

2. As a parameter-efficient fine-tuning method, Why not compare VeRA, LoRA, DoRA, BOFT, and SVFT with the same (or similar) number of parameters in Tables 1-4? What are the criteria for evaluating the performance of PEFT methods?

3. In both the original and newly provided comparison results, $**SVFT**^{T}$ is not the best-performing performance. Why is $**SVFT**^{T}$ proposed to explore the broad spectrum of possible selection criteria of $\mathbf{M}$`s sparsity pattern?

4. If the main insight and contribution is that significant gains can be achieved by learning off-diagonal interactions in $\mathbf{M}$, a comparison of Random and Top-k with Plain in the same parameter quantities must be provided.

5. It is worth noting that the singular value space is not the only avenue to explore the effects of off-diagonal interactions in $\mathbf{M}$. Like OFT[1] and BOFT[2] which introduced in Related Work section, $\mathbf{h} = (\mathbf{M} \mathbf{W_0}) \mathbf{x}$ or $\mathbf{h} = (\mathbf{W_0} \mathbf{M}) \mathbf{x}$ can also learn the off-diagonal interactions in $\mathbf{M}$, and the four sparsity patterns of ``Random, Top-k, Plain and Banded‘’ are also applicable choices in this situation. As a reparameterization-based method, why can not the sparse trainable matrix $\mathbf{M}$ directly update the pre-trained weight matrix $\mathbf{W}_0$? Why are the Singular Vectors a better way to guide fine-tuning? Please provide detailed analysis and comparison results to enhance the insight and contribution of this work.

[1] Controlling text-to-image diffusion by orthogonal fine-tuning. NIPS23. [2] Parameter-efficient orthogonal finetuning via butterfly factorization. ICLR24.

C4. Comparing $SVFT^R$ and $SVFT^T$ with $SVFT^P$ using the same number of parameters isn't very meaningful. $SVFT^P$ applies a constrained full-rank update. For $SVFT^R$ and $SVFT^T$ , when the total number of trainable parameters is equal to that in $SVFT^P$ (i.e., the total elements in the main diagonal of Σ for a chosen W), the rank of the perturbation is reduced compared to adapting the entire main diagonal (Rank Lemma in Sec 3.2). We introduced SVFT variants to offer flexibility in the total number of trainable parameters. Significant gains are observed when the total number of learnable parameters is greater than in Plain, starting notably at d=2. In line 162, from equation (2), in our paper, $u_i$​ appears in multiple summation terms in SVFT, which is less likely when only learning main diagonal number of parameters in W. To substantiate our point, please see below table -- we only report the rank of $\Delta W$ for Q for clarity. However, similar findings hold across remaining target modules.

C5.

• (i) We agree that there are other avenues worth exploring regarding the effects of the off-diagonal interactions we introduced in M. However, OFT and BOFT impose specific structures on the pre/post multiplication matrix $R$ to preserve orthogonality and the spectral norm of $W$. Applying our sparsity patterns to $R$ would violate these conditions, potentially leading to training instability (see the spectral properties discussed in Section 5 of BOFT).

• (ii) Fine-tuning with fixed sparse masks has been studied before and compared to early PEFT methods like Bit-Fit [2] and Adapter Tuning [3]. For example, [1] demonstrates that directly updating $W_0$ with a sparse trainable random matrix $M$ is inferior to Bit-Fit. Recent PEFT methods, such as LoRA and VeRA, show significant improvements over Bit-Fit. We compare our approach against LoRA and VeRA in our experiments.

Our work is largely empirical with a few theoretical insights. The question of why Singular Vectors provide a better way to guide fine-tuning is intriguing and could be a potential direction for future theoretical study.

[1] Training Neural Networks with Fixed Sparse Masks. NeurIPS 2021.
[2] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. ACL 2022
[3] Parameter-Efficient Transfer Learning for NLP. ICML 2019

We thank the reviewer for their prompt response and new feedback. We address their comments (C) below. Please see Response 2/2 for C4, C5.

C1. We agree that using $d$ for Top-k and Random methods is redundant and potentially confusing. We initially aimed for consistency across all methods, including $SVFT^B$, to facilitate comparisons. In the final version, we will: 1. Explicitly state the number of trainable parameters in notation for Top-k and Random. 2. Clearly differentiate notation between banded and non-banded methods. 3. Briefly explain this change in notation. 4. Update all relevant tables, figures, and discussions accordingly.

C2. We indeed compared VeRA, LoRA, DoRA, BOFT, and SVFT with similar parameter counts as shown in Figures 1 and 5. Note we cannot precisely match the number of parameters because, for some baselines, adjustments can only be made in increments greater than 1, e.g increasing rank by 1 introduces 2d parameters.

Method-specific constraints limited parameter adjustments:

• VeRA: This method shares frozen adapters across compatible layers, making it difficult to significantly increase the total trainable parameters by simply increasing intermediate dimensions.
• BOFT: Adapting any additional module beyond QV leads to memory issues, as the pre-multiplication matrix R is a product of multiple orthogonal matrices. This limitation is also noted in Section 7 of the BOFT paper.

For reviewer’s convenience, We provide tables with clear demarcation for easier comparison. SVFT variants show notable gains with comparable parameter counts.

For criteria for evaluating PEFT methods, we considered several factors:

1. Performance: How well the method improves the model's performance on the target task (Tables 1-4).
2. Parameter Efficiency: The ratio of performance improvement to the number of added parameters (Figure 1 and 5).
3. Memory Efficiency: The method's ability to optimize memory usage during training and inference (Please see our global response).
4. Scalability: How well the method performs across different model sizes and task types (Table 1).

C3 In our study, we investigate various simple sparsity patterns, including $SVFT^T$, and analyze their impact on performance. This analysis serves as an ablative study to understand the effectiveness of different sparsity patterns. Considering reviewer's feedback, our final paper will highlight that other sparsity patterns outperform the $SVFT^T$.

We sincerely thank the reviewer for their feedback and for actively engaging in the discussion. We address their concerns below:

"The exploration of this work is neither comprehensive nor thorough."

We respectfully disagree with this assessment. Our work includes experiments on 22 datasets across both language and vision tasks, involving 4 different language models and 5 baselines, with various configurations to match total trainable parameters. Moreover, during the rebuttal phase, we provided additional analyses, including memory usage and an ablation study on different sparsity patterns. If there are specific areas where the reviewer feels our exploration is lacking, we would appreciate further clarification.

“The author does not clearly explain how to design an appropriate sparse pattern based on the characteristics of downstream tasks."

Our primary insight focuses on the inclusion of additional off-diagonal elements, with empirical evidence showing that even simple sparse patterns can outperform previous PEFT methods across different tasks. While designing task-specific sparse patterns is indeed valuable, it is beyond the scope of this work.

"The author suggests further exploration of SVFT$^T$…"

We believe there may be a misunderstanding here. As clarified in our previous responses, our analysis was intended as an ablation study to evaluate the effectiveness of different sparsity patterns.

Throughout the rebuttal, we have addressed several of the previously raised concerns with new experiments and referenced relevant published works to provide additional context. We kindly request the reviewer to reconsider their evaluation of our work in light of these responses.

We thank the reviewer for their valuable feedback. We address the weaknesses (W) raised by the reviewer below.

W1: We emphasize that our main contribution is making off-diagonal elements in $M$ learnable; this allows us to smoothly trade-off between the number of trainable parameters and expressivity (which is not possible by only fine-tuning the singular values -- the main diagonal of $M$). We justify this theoretically (Expressivity Lemma in Section 3.2) and empirically (in all our results SVFT$^B$, SVFT$^R$, SVFT$^T$ outperforms SVFT$^P$).

W2: We acknowledge that while SVFT reduces trainable parameters, it increases overall GPU memory usage compared to LoRA. However, reducing trainable parameters decreases the memory required for gradients, activations, optimizer state, and various buffers. To address the reviewer's concern, we used HuggingFace's internal memory profiler to log actual peak GPU memory usage. We've included these findings, along with the adapted modules for all baselines, in the table below. In summary, SVFT outperforms LoRA while using ~1.2x more memory. It is worth noting that SVFT's GPU memory usage is either similar to or lower than that of DoRA.

Gemma-2B

Gemma-7B

Llama-3-8B

W3: We thank the reviewer for pointing us to these papers (SVF [1], SVDiff [2]). Indeed [1] and [2] are essentially equivalent to SVFT$^P$, but we note that our main insight and contribution is that significant gains can be achieved by learning off-diagonal interactions in $M$.

Doing our due diligence, in the following Table, we compare SVFT$^P$ against SVF on GSM-8K and Math benchmarks. Results in the Table below show SVF performs similar to SVFT$^P$ and worse than SVFT$^{R}$ as we had expected. We will acknowledge these papers and add a discussion in our final version of the paper.

References
[1] Sun et al.: Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning. NeurIPS 2022
[2] Han et al. SVDiff: Compact Parameter Space for Diffusion Fine-Tuning, ICCV 2023.

We thank the reviewer for their valuable feedback. We address the weaknesses (W) raised by the reviewer below.

W1: Recall that VeRA injects frozen low-rank random matrices with learnable scaling vectors resulting in weight updates independent of the pre-trained weights. SVFT$^P$ derives its weight updates directly from the pre-trained matrix $W$.
VeRA differs from SVFT$^P$ in two ways:
i) VeRA achieves high parameter efficiency by sharing frozen adapters across compatible layers, a choice that no other previous methods take (e.g., LoRA, DoRA). This particular design choice gives it lower expressiveness than what may have been possible for other methods with the same intermediate dimension.
ii) The perturbation directions that VeRA operates on (i.e., its frozen adapters) are chosen independent of the underlying matrix that it is trying to perturb.

We note that despite the sharing of parameters across layers in VeRA, it ends up having more trainable parameters per layer compared to SVFT$^P$. For example, in Table 2, we achieve better results with SVFT$^P$ using dims=4096 (0.51M params) compared to VeRA with dims=2048 (1.49M params). SVFT$^P$'s use of singular vectors gives it layer-wise specialization.

W2: As '$d$' (off-diagonals) (or '$r$' for other PEFT methods) increases, performance typically improves. However, overfitting can occur with any method, especially with smaller datasets or models. Figure 5 (left) in the paper shows $DoRA_{r=16}$ underperforming DoRA$_{r=4}$ on language tasks. In Tables 4 and 9, the additional fine-tuning of the classifier head may contribute to overfitting.

W3: We note that simple heuristics for sparsity patterns in $M$ significantly improve performance over previous PEFT techniques. While most variants (except SVFT$^P$) offer comparable performance, the banded pattern (SVFT$^B$) slightly outperforms others. We hypothesize that a more principled approach to finding optimal sparsity patterns for specific tasks and base models may yield further performance gains. We leave it as future work to explore task-specific and model-specific sparsity patterns. We will revise our paper to reflect these new results and findings.

Dear Reviewers and Authors,

Thank you very much for all the informative discussions. We appreciate you for the big efforts.

To reviewers: if you need any further clarifications from authors, please make sure they can be addressed by text-only response without additional experimental results.

To authors: If you have any clarification and summary you would like to share with reviewers, or request their comments on some of your rebuttal and/or discussion points, please add those and/or kindly remind the reviewer(s).

Thank you.