KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models

Thank you for your comprehensive and insightful review of our paper. We greatly appreciate your recognition of the novelty of our work.

My main concern -- most experiments, except experiments in Table 4 on instruction following, only use a small model (less than 1B). Need more experiments and fair comparison with the baselines on larger models.

For fair comparison, we adopt the same experimental setups of existing PEFT methods [1, 2, 3, 4]that utilizes the small language model (less than 1B) for NLU and NLG tasks. For example, VeRA [1] and FourierFT [2] utilize RoBERTa-base (125M) and RoBERTa-large (355M) for NLU tasks (refer to Table 2), and GPT-2 medium (354.92M) and GPT-2 large (774.03M) for NLG tasks (refer to Table 3). PiSSA [3] employs RoBERTa-large (355M) and DeBERTaV3-base (184M) for NLU tasks. Similarly, the latest SARA [4] uses GPT-2 medium (354.92M) for NLG tasks (see Table 3).

And the improvements on instruction following evaluation results are not significant, and there are even drops in performance on LLaMA3 8B.

To demonstrate the significant improvements of our method, we perform an independent two-sample t-test to evaluate the statistical significance of the differences between our method and the baselines (excluding FFT due to its high computational demands). This evaluation is conducted over five runs with different random seeds to ensure robustness. All tests are carried out at a significance level of $\alpha = 0.05$. The performance comparison between our KaSA and key baselines is illustrated in the following Table.

As evidenced, KaSA significantly outperforms all baselines, achieving p-values < 0.05 in 51 out of 60 experimental settings across the Gemma 7B, Mistral 7B, and LLaMA 3 8B on 5 datasets. Therefore, the significance tests substantiate the significant improvement of our method over the baselines, verifying the superiority and robustness of our approach.

Gemma 7B

Mistral 7B

LLaMA3 8B

LLaMA2 13B

How well does KaSA scale to even larger models?

To showcase the scalability of our KaSA model with even larger architectures, we conducted an instruction-following experiment using the Llama 2 13B model. Unfortunately, we are unable to test the LLaMA 3 70B model due to computational resource constraints, which require approximately 140GB of GPU memory.

As indicated in the last row of the Tables in Part 1, KaSA consistently and significantly outperforms all baseline models in 13 out of 15 experimental settings, with p-values less than 0.05 when scaled to the larger 13B model. This demonstrates the scalability and robustness of our proposed method.

Reference

[1] Kopiczko, Dawid Jan, Tijmen Blankevoort, and Yuki Markus Asano. "Vera: Vector-based random matrix adaptation." ICLR, 2024.

[2] Gao, Ziqi, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. "Parameter-Efficient Fine-Tuning with Discrete Fourier Transform." ICML, 2024.

[3] Meng, Fanxu, Zhaohui Wang, and Muhan Zhang. "Pissa: Principal singular values and singular vectors adaptation of large language models." NeurIPS, 2024.

[4] Gu, Jihao, Shuai Chen, Zelin Wang, Yibo Zhang, and Ping Gong. "Sara: Singular-value based adaptive low-rank adaption." arXiv preprint arXiv:2408.03290 (2024).

Some important baselines are missing, like [1] CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning

It is a good suggestion to include the comparison with this important baseline in our study. To ensure a fair comparison, we have reached out to the authors of CorDA to verify the implementation details on the GLUE benchmark and have reproduced their results successfully. (We would like to express our sincere gratitude to the authors of CorDA for their generous assistance.)

In accordance with the experimental settings outlined in the CorDA paper, we present the experimental results for our KaSA with a rank of $r=128$. The results are displayed in the Table below. We observe that KaSA consistently outperforms all baselines across 6 datasets, with a slight exception for the QNLI dataset, where it performs marginally worse than FFT (92.71 vs. 92.8).

The PEFT methods are sensitive to hyperparameters like the learning rate. How does the paper ensure a fair comparison with the baseline models? For instance, in Section 4.2, a hyperparameter search is conducted for KaSA for each dataset. Is a similar hyperparameter search performed for all the baseline methods? For Section 4.3 and 4.4, do KaSA and baselines share the same hyperparameters?

To ensure a fair comparison among various PEFT methods, we adhere to a common practice to directly utilize the experimental results reported in their original papers. For instance, VeRA [2], FourierFT [3], and LoRA [5] all incorporate experimental results from prior works for NLU tasks on the GLUE benchmark and NLG tasks on the E2E benchmark.

Specifically,

(1) In our paper, Section 4.2, Table 1, we adopt the baseline results—excluding PiSSA and MiLoRA—from Table 2 of FourierFT [3]. In Table 2, we use baseline results—excluding PiSSA and MiLoRA—from Table 2 of BiLoRA [6]. We follows [2] and [3] to perform a hyperparameter search to determine optimal parameters.

(2) In Section 4.3, Table 3, baseline model results are sourced from SARA [4] and BiLoRA [6]. Our KaSA employs the same hyperparameters as LoRA that is specified at https://github.com/microsoft/LoRA/tree/main/examples/NLG.

(3) In Section 4.4, for fair comparison, we make extensive efforts to meticulously tune the hyperparameters to achieve optimal performance for each baseline. To avoid randomness in grid searching, we refer to the LoRA NLG configuration available on the official GitHub repository at https://github.com/microsoft/LoRA/tree/main/examples/NLG.

(4) Regarding PiSSA and MiLoRA, the PiSSA implementation has been integrated into the official PEFT packages. Thus, we use PiSSA from PEFT, following the guidelines at https://huggingface.co/docs/peft/v0.13.0/en/developer_guides/lora#pissa. For the MiLoRA implementation, we modify the PiSSA implementation with a few lines of code, as they are complementary initialization approaches. Since PiSSA and MiLoRA are parameter initialization methods for LoRA, their configurations should be strictly consistent with the original LoRA configuration. Thus, for NLU tasks, we employ the LoRA configuration for PiSSA and MiLoRA, following the guidelines at https://github.com/microsoft/LoRA/tree/main/examples/NLU. However, for instruction-following tasks, we adhere to the same LoRA configuration. Unlike LoRA, the rank $r$ should be equal to $\alpha$, as mentioned in their paper.

Thank you for your comprehensive and insightful review of our paper. We appreciate your valuable feedback and suggestions.

The proposed method has a more complex training objective compared to baselines like LoRA, PiSSA, and MiLoRA. Consequently, the training time for KaSA may be longer than that of the baselines. However, the paper does not provide detailed information on this.

Thank you for bringing this to our attention. We have conducted a comprehensive efficiency and complexity comparison between LoRA and SVD variants across different tasks and model scales, as shown in the table below. We clarify that the dynamic singular value adaptation introduced in our KaSA is a learnable one-dimensional vector of size $r \ll m$ and requires parameter regularizations, incurring negligible training overheads compared to the standard LoRA. In addition, due to low-rank approximation of the original matrix, we reduce the rank of $\mathbf{W}$ from $m$ to $m-r$, accelerating the inference particularly for small-scale language models like RoBERTa-base 125M (i.e., with small $m$). We show the experimental results as follows.

As can be seen, compared to LoRA, the extra training overhead is less than 20% (resp. 3%) for the NLU (resp. NLG) task, while speeding up the inference by 1.45x (resp. 1.02x) times. When compared to PiSSA and MiLoRA, our approach incurs an average of less than 13% extra training overhead for NLU tasks, while maintaining comparable or improved inference latency. For NLG tasks, our method introduces similar training overhead or inference latency.

RoBERTa-base 125M on Single NVIDIA GeForce RTX 3090 (24GB) GPU for NLU task

LLaMA3 8B on Single NVIDIA A100-SXM4 (80GB) GPU for NLG task

The setup for the instruction-following experiments is questionable. Given the availability of numerous instruction tuning datasets, why not conduct experiments using these existing datasets? Since other researchers may not have access to the synthetic dataset, reproducing the results could be challenging.

Thank you for highlighting this important aspect.

Reason: Many existing instruction datasets are synthetic dataset generated by LLMs. For example, Alpaca 51k and WizardLM-Evol-Instruct 70k (building upon Alpaca) are generated by OpenAI's text-davinci-003 and gpt-3.5-turbo models, respectively. However, as highlighted in numerous studies, the original Alpaca 51k dataset has several issues, including poor data quality, hallucinations, merged instructions, empty outputs, and empty code examples. Detailed issues can be found at https://github.com/gururise/AlpacaDataCleaned. Additionally, the model used for data synthesis is somewhat outdated, which limits the quality of the instruction dataset. Due to these limitations, we did not employ these synthetic datasets in our experiments.

In this work, we contribute a high-quality synthetic instruction datasets by GPT4o to to enhance the functionality of PEFT and support future research endeavors. As acknowledged by Reviewer 9ooo, the 'no robots' dataset is a good choice for seeding synthetic data, as it was generated by human annotators. To maintain anonymity, we cannot provide you with the specific dataset link on Hugging Face at this time. We are committed to publicly releasing the dataset soon.

Moreover, as per your advice, we have followed the VeRA [2] and FourierFT [3] to fine-tune on Alpaca 51k and report the evaluation results on MT-Bench, with GPT4 serving as the judge, yielding scores within 10. The results are presented in the Table below.

As can be seen, our KaSA consistently and significantly outperforms the LoRA and SVD variant baselines, achieving p-values < 0.05 in 9 out of 12 experimental settings across Gemma 7B, Mistral 7B, LLaMA 3 8B, and LLaMA 2 13B. These findings align with experiments conducted on our synthetic datasets.

Reference

[1] Yang, Yibo, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. "CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning." NeurIPS, 2024.

[2] Kopiczko, Dawid Jan, Tijmen Blankevoort, and Yuki Markus Asano. "Vera: Vector-based random matrix adaptation." ICLR, 2024.

[3] Gao, Ziqi, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. "Parameter-Efficient Fine-Tuning with Discrete Fourier Transform." ICML, 2024.

[4] Gu, Jihao, Shuai Chen, Zelin Wang, Yibo Zhang, and Ping Gong. "Sara: Singular-value based adaptive low-rank adaption." arXiv preprint arXiv:2408.03290 (2024).

[5] Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. "Lora: Low-rank adaptation of large language models." ICLR, 2021.

[6] Qiang, Rushi, Ruiyi Zhang, and Pengtao Xie. "BiLoRA: A Bi-level Optimization Framework for Overfitting-Resilient Low-Rank Adaptation of Large Pre-trained Models." arXiv preprint arXiv:2403.13037 (2024).

The training objective introduces new hyperparameters, including \beta and \gamma, but there is no analysis regarding their impact.

We appreciate the reviewer's insightful feedback on this matter. We put great effort into meticulously tuning the two trade-off loss coefficients, $\beta \in [\text{1E-5}, 1]$ and $\gamma \in [\text{1E-5}, 1]$, to explore their impacts. The sensitivity analysis is presented in the Table below. The results demonstrate that KaSA is not sensitive to variations in $\beta$ and $\gamma$.

How does KaSA perform at different ranks? Can it consistently exceed the baselines across various ranks? This aspect also needs to be validated.

It would be very helpful to have a performance comparison at different ranks. We examine the impact of rank across LoRA and SVD variant baselines, with the range of ranks set to $r = \\{1,2,4,8,16,32,64,128 \\}$. The experimental results are presented in the Table below.

We observe that KaSA consistently surpasses LoRA and the SVD variant baselines, including PiSSA and MiLoRA, across various ranks except in 4 out of 96 cases on the MRPC, CoLA, and RTE datasets, highlighting the efficacy and robustness of our method.

Thank you again for your feedback. We are very pleased to know that most of your concerns have been addressed. To further address your concerns regarding experiments on fair comparison and on large language models and well-known LLM benchmarks, we have conducted new experiments with relentless efforts so that the response is slightly delayed.

Ensuring a Fair Comparison Between PEFTs: While the authors offer some explanations, I still find it difficult to conclude that KaSA is superior to PiSSA, MiLoRA, or CorDA due to the hyperparameter setups.

To ensure a fair comparison, we have reached out to the authors of MiLoRA to verify the implementation details on the eight commonsense reasoning benchmarks and have reproduced their results successfully. (We would like to express our sincere gratitude to the authors of MiLoRA for their generous assistance and guidance.)

Specifically, to maintain fairness in our comparison, we undertook the following measures:

1. All methods are implemented within the open-source framework LLM-Adapters [1], available at https://github.com/AGI-Edgerunners/LLM-Adapters, following MiLoRA [2].
2. We adhered strictly to the hyperparameter configurations for training and evaluation as specified in papers [1] and [2], without any tuning, as advised by the authors of MiLoRA to ensure fair implementation.
3. We have anonymously uploaded the training and evaluation logs, including hyperparameter configurations and performance results, for your verification. Please check the link at train_llama-2-7b-kasa.log and eval_llama-2-7b-kasa-benchmarks.log.

The results of LLaMA2 7B are presented in the Table below. With identical hyperparameter setups, we observe that KaSA consistently outperforms all strong baselines across the 8 commonsense reasoning datasets.

The symbol $*$ indicates that the results are taken from the original MiLoRA paper [2].

Redesign of Experimental Sections: As suggested by Reviewer Xp8u, the authors have not conducted comprehensive experiments on large language models and well-known LLM benchmarks. Although additional results were provided during the rebuttal, these are not sufficient to comprehensively demonstrate that KaSA is a better choice compared to existing baselines.

To date, our experiments have been conducted in NLU (RoBERTa-Base, RoBERTa-Large, and DeBERTaV3-base), NLG (GPT-2 Large and GPT-2 Medium), instruction tuning (Gemma 7B, Mistral 7B, Llama 3 8B, and Llama 2 13B), and commonsense reasoning (Llama 2 7B). We utilize the E2E NLG Challenge benchmark, four 128k synthetic datasets, the 51k Alpaca dataset, an MT-Bench benchmark, a commonsense170k dataset, and 8 commonsense reasoning benchmarks. We believe these comprehensive experiments sufficiently verify the superiority of our KaSA compared to existing baselines. For ease of reference, in what follows, we list the positions of the aforementioned experiments in our revised paper.

Reference

[1] Hu, Zhiqiang, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models." EMNLP, 2023.

[2] Wang, Hanqing, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. "Milora: Harnessing minor singular components for parameter-efficient llm finetuning." arXiv preprint arXiv:2406.09044 (2024).

Dear Reviewer,

Thank you again for your feedback.

The results of LLaMA3 8B, which has just completed its training and evaluation, are presented in the Table below. With identical hyperparameter setups, we observe that KaSA consistently outperforms all strong baselines by a large margin across the 8 commonsense reasoning datasets, with a slight exception for the WinoGrande dataset, where it performs marginally worse than MiLoRA (85.5 vs. 85.6).

The symbol $*$ indicates that the results are taken from the original MiLoRA paper [2].

The training and evaluation logs, including hyperparameter configurations and performance results, have also been uploaded for your verification. Please check the link to train_llama-3-8b-kasa.log and eval_llama-3-8b-kasa-benchmarks.log.

Thank you.

Authors

The SVD truncation stage may discard knowledge components deemed noisy or irrelevant. However, there’s a risk that some truncated components might contain task-relevant information. Did you experiment with different truncation thresholds, or analyze the potential trade-offs of truncation in various tasks? A discussion here would enhance understanding of KaSA’s robustness.

Nice comments. Based on the formulation in Equation (7), the reconstruction error between the original and truncated weights is given by $\Vert \mathbf{W} - \mathbf{W} _ {truncated} \Vert_F=\sum_{j=m-r}^{m}{\Delta\sigma_j^2}$.

In PEFT research, choosing a rank ($r \ll m$) ensures that the approximate matrix ($\Vert \mathbf{W} - \mathbf{W}_{truncated} \Vert_F$) closely resembles the original matrix ($\mathbf{W}$).

However, as the rank ($r$) increases, there is a potential risk that some truncated components may still contain task-relevant information. To investigate this, we examine the impact of varying the SVD rank (denoted as $k \in \\{1, 2, 4, 8, 16, 32, 64, 128\\}$) on model performance. This analysis uses RoBERTa-base across the MRPC, CoLA, and RTE tasks, as illustrated in Figure 8 of Appendix D.1 in our manuscript. As shown in Figure 8, model performance improves as $k$ increases from 1 to 8. This demonstrates that truncating the long-tail noise or less relevant information, associated with much smaller singular values, can positively enhance task-relevant knowledge adaptation. Conversely, increasing $k$ from 8 to 128 results in a performance decline, indicating that some truncated components may still contain valuable task-relevant information.

These findings emphasize the importance of identifying an optimal SVD truncation rank that balances incorporating world knowledge with large singular values and excluding disruptive noise with smaller singular values. Achieving this balance is crucial for optimizing model performance.

The SVD rank and regularization coefficients play an important role in KaSA’s performance. Could you provide practical guidelines or a sensitivity analysis to help practitioners choose these values for new tasks or datasets? This could make the method more accessible to those less familiar with PEFT or SVD.

According to our empirical results, we recommend that when there is a significant disparity between the distribution of the pre-training tasks and the downstream fine-tuning task, opting for a larger SVD rank can effectively filter out knowledge that is not pertinent to the downstream task. Conversely, a smaller SVD rank should be selected when the distributions are more aligned. A comprehensive sensitivity analysis of the SVD rank is provided in Figure 8 of Appendix D.1 in our manuscript.

Additionally, for more complex fine-tuning tasks, we advise using larger regularization coefficients to enhance the stability of model training and address potential issues such as gradient vanishing or explosion. The sensitivity analysis of these regularization coefficients for RoBERTa-base on CoLA, RoBERTa-large on SST-2, and DeBERTa-v3-base on MRPC is presented in the Table below. The findings indicate that KaSA exhibits robustness to variations in the regularization coefficients $\beta$ and $\gamma$.

Reference

[1] Gu, Albert, and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv preprint arXiv:2312.00752 (2023).

Thank you for your thorough and insightful review. We greatly appreciate your recognition of the contributions of our work.

The method adds some complexity, especially with the dynamic singular value adaptation, which might make it harder to implement compared to simpler PEFT methods.

Compared to LoRA, KaSA requires a small amount of additional efforts. Specifically, we have provided a PyTorch-style pseudocode block illustrating our KaSA implementation, which requires only three additional operations:

1. Knowledge-based SVD truncation, which involves just 4 additional lines of code.
2. Knowledge-aware singular-value adaptation, necessitating only 1 additional line of code.
3. Parameters regularization losses, which require 20 additional lines of code.

class KaSA(nn.Module):
    def __init__(self,
	    rank: int = 8, # kasa rank
	    alpha: int = 16, # kasa alpha
	    base_layer: nn.Module # pre-trained layer
    )
        # definitions
        self.r = rank
        self.alpha = alpha
        self.scaling = alpha / rank
        self.in_features, self.out_features = base_layer.in_features, base_layer.out_features

        # Step 1: knowledge-based SVD truncation
        self.svd_rank = self.in_features - self.r
        U, S, Vh = torch.linalg.svd(base_layer.weight.data, full_matrices=False)
        base_layer.weight.data = U[:, :self.svd_rank] @ torch.diag(S[:self.svd_rank]) @ Vh[:self.svd_rank, :]
        self.base_layer = base_layer

        # Step 2: knowledge-aware singular-value adaptation
        self.delta_v = nn.Linear(self.in_features, self.r, bias=False)
        self.delta_sigma = torch.diag(nn.Parameter(torch.randn(self.r), requires_grad=True))
        self.delta_u = nn.Linear(self.r, self.out_features, bias=False)

    def forward(self, x: torch.Tensor):
    	# Step 3: merge W + Delta_W (Eq.7)
        Delta_W = self.delta_u @ self.delta_sigma @ self.delta_v
        result = self.base_layer(x)
        result = result + torch.einsum('ijk,kl->ijl', x, Delta_W) * self.scaling
        return result
 
def regularization_loss(model, beta, gamma):
    l2_loss, l3_loss = 0.0, 0.0
    num_param = 0
    for name, param in model.named_parameters():
        if param.requires_grad:
            if 'delta_sigma' in name:
                num_param += 1
                diag_norm = torch.sum(param ** 2)
                l2_loss += diag_norm
            elif 'delta_v' in name or 'delta_u' in name:
                if 'delta_v' in name:
                    matmul_result = torch.matmul(param.T, param)
                else:  
                    matmul_result = torch.matmul(param, param.T)

                I = torch.eye(matmul_result.size(0), device=matmul_result.device)
                diff_I = matmul_result - I
                matrix_loss = torch.norm(diff_I, p='fro')
                l3_loss += matrix_loss
    auxi_loss = (beta * l2_loss/num_p + gamma * l3_loss) / num_param if num_param > 0 else 0.0
    return auxi_loss

It would also help to see this method applied to a wider range of LLM architectures to confirm its generalizability. While KaSA does well on the tested models, applying it to newer transformer architectures or other types of large models could demonstrate its versatility even more.

Good suggestion. In this study, we examine two types of LLM architectures: encoder-only language models, which include RoBERTa-base, RoBERTa-large, and DeBERTaV3-base, and decoder-only language models, such as GPT-2, Gemma, Mistral, and LlaMA3. We agree that applying our KaSA method to newer Transformer architectures, such as Mamba [1], can further enhance our work. Due to time limitations, we promise to explore this later.

My current understanding is that the method requires (1) LoRA finetuning followed by (2) SVD truncation followed by (3) “knowledge aware singular-value adaptation.” Is this correct? If so, doesn’t this mean that it will require more training/compute than vanilla LoRA? Can you do a direct comparison of training compute/FLOPs for LoRA vs. KaSA? The authors claim no extra inference latency due to the technique. It would be helpful to show these numbers. If I understand correctly, however, there will be extra training latency. Can you expand on this in the paper?

Thank you for your insightful feedback regarding the training compute/FLOPs comparison between KaSA and other baselines. We clarify that the dynamic singular value adaptation introduced in our KaSA is a learnable one-dimensional vector of size $r \ll m$ and requires parameter regularizations, incurring negligible training overheads compared to the standard LoRA. In addition, due to low-rank approximation of the original matrix, we reduce the rank of $\mathbf{W}$ from $m$ to $m-r$, accelerating the inference particularly for small-scale language models like RoBERTa-base 125M (i.e., with small $m$). We show the experimental results as follows. As can be seen, compared to LoRA, the extra training overhead is less than 20% (resp. 3%) for the NLU (resp. NLG) task, while speeding up the inference by 1.45x (resp. 1.02x) times.

RoBERTa-base 125M on Single NVIDIA GeForce RTX 3090 (24GB) GPU for NLU task

LLaMA3 8B on Single NVIDIA A100-SXM4 (80GB) GPU for NLG task

"KaSA employs a small set of parameters $\psi$ to reparameterize the task-specific update $\Delta \Theta$" is the task specific update full finetuning or LoRA? I assume the task specific update is vanilla LoRA.

To clarify, the task-specific update $\Delta \Theta$ does not require full fine-tuning. Similar to LoRA, our KaSA method employs low-rank matrices, parameterized by $\psi$, to effectively learn $\Delta \Theta$. This approach allows the model to efficiently learn task-specific updates without the need for extensive parameter updates.

How do you calculate $W_{fft}$ for $L_2$? Do you have to do full finetuning? I assume not…

We confirm that we do not perform full fine-tuning. In fact, instead of directly calculating $W_{fft}$, we compute L2 regularization loss without involving $W_{fft}$ as a lower bound of $||W_{fft}-W_{world}||_F$.

What is the rank for instruction following? I don’t see it in the manuscript

We set the ranks $r_q$ and $r_v$ to 8 for instruction following. Due to page constraints, we provide the detailed implementation configurations for instruction-following tasks in Table 10 of the Appendix.

What exactly is the reader supposed to take away from the visualizations in Figure 4? It seems somewhat unclear. It might be helpful to have a comparison with and without KaSA

Figure 4 shows that different scales of singular values are allocated across positions in various layers, suggesting a dynamic prioritization of knowledge across parameters. This confirms the effectiveness of the innovative knowledge-aware singular-value adaptation in KaSA. Note that the knowledge-aware singular-value adaptation is a core component of KaSA, which cannot be trivially applied to other models.

Why scale between 0 and 100 when using GPT4o as a judge? It seems a bit arbitrary/unnecessary. Do you have strong motivation for using this scaling?

We appreciate the reviewer's insightful feedback on this matter. Following the common setting in previous work [1], [2], and [3], we have conducted additional experiments using a 0 to 10 score range. The updated results are presented in the Tables in Part 3. The general trends are similar, demonstrating the superiority of our KaSA method.

Some small nits:

• Typo: line 043 “gets popular”
• Typo: line 426 “four synthetic datasets using GPT-4”; is it GPT-4 or GPT4o?

Thank you for bringing these typos to our attention. We have addressed and corrected them in the revised version of our manuscript.

Reference

[1] Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." NeurIPS, 2023.

[2] Kopiczko, Dawid Jan, Tijmen Blankevoort, and Yuki Markus Asano. "Vera: Vector-based random matrix adaptation." ICLR, 2024.

[3] Gao, Ziqi, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. "Parameter-Efficient Fine-Tuning with Discrete Fourier Transform." ICML, 2024.

It feels like important baselines are missing from Table 4 - how well do the models do out of the box? Since these datasets are created specifically for this study, it is hard to get a sense of task difficulty or to compare to other papers. The most similar methods to KaSA are PiSSA and MiLoRA. In Table 4 it could be helpful to do a comparison with PiSSA and/or MiLoRA.

Following your valuable suggestion, we have incorporated important baselines, such as the models' performance out of the box and SVD variant methods, into the updated Table 4 below.

As can be seen, KaSA consistently and significantly outperforms LoRA and the SVD variant baselines across Gemma 7B, Mistral 7B, LLaMA 3 8B, and LLaMA 2 13B models. Notably, the baseline models, when used out of the box, tend to generate random and poor-quality responses for specific downstream tasks. This highlights the base model's lack of specific domain knowledge.

Gemma 7B

Mistral 7B

LLaMA3 8B

LLaMA2 13B

Thank you for your comprehensive and insightful review of our paper. We appreciate your valuable feedback and suggestions.

The experimental results are overall quite strong. The main weakness of the manuscript is that it lacks a straightforward explanation of the method. I’m not sure I fully understand the KaSA method here based on the description, and the diagram is slightly confusing as well. It would be very helpful to have a pseudocode block/section.

It is a nice suggestion to incorporate a pseudocode block for a better explanation of our method. In what follows, we provide the PyTorch-style pseudocode of KaSA. Specifically, KaSA consists of three main steps, including (Step 1) Knowledge-Based SVD Truncation, (Step 2) Knowledge-Aware Singular-Value Adaptation, and (Step 3) Integration of Fine-tuned Parameters into the Base Model.

class KaSA(nn.Module):
    def __init__(self,
	    rank: int = 8, # kasa rank
	    alpha: int = 16, # kasa alpha
	    base_layer: nn.Module # pre-trained layer
    )
        # definitions
        self.r = rank
        self.alpha = alpha
        self.scaling = alpha / rank
        self.in_features, self.out_features = base_layer.in_features, base_layer.out_features

        # Step 1: knowledge-based SVD truncation
        self.svd_rank = self.in_features - self.r
        U, S, Vh = torch.linalg.svd(base_layer.weight.data, full_matrices=False)
        base_layer.weight.data = U[:, :self.svd_rank] @ torch.diag(S[:self.svd_rank]) @ Vh[:self.svd_rank, :]
        self.base_layer = base_layer

        # Step 2: knowledge-aware singular-value adaptation
        self.delta_v = nn.Linear(self.in_features, self.r, bias=False)
        self.delta_sigma = torch.diag(nn.Parameter(torch.randn(self.r), requires_grad=True))
        self.delta_u = nn.Linear(self.r, self.out_features, bias=False)

    def forward(self, x: torch.Tensor):
    	# Step 3: merge W + Delta_W (Eq.7)
        Delta_W = self.delta_u @ self.delta_sigma @ self.delta_v
        result = self.base_layer(x)
        result = result + torch.einsum('ijk,kl->ijl', x, Delta_W) * self.scaling
        return result

Thank you for your comprehensive and insightful review of our paper. We appreciate your valuable feedback and suggestions.

The improvement of the proposed method over the baselines is marginal. The variances and the results of the significance tests may help justify the improvement.

Thank you for your valuable suggestion. We agree that it is a common phenomenon that the average performance improvement of recent SOTA PEFT methods (e.g., PiSSA [1] and FourierFT [2] as illustrated in Table 2) over existing approaches is relatively marginal. However, as per your suggestion, we show that our KaSA method clearly outperforms baselines in terms of significance tests.

Specifically, we present the results as mean $\pm$ standard deviation and utilize an independent two-sample t-test to evaluate the statistical significance of the differences between our method and the baseline. This evaluation is conducted over five runs with different random seeds to ensure robustness. All tests are carried out at a significance level of $\alpha = 0.05$. The performance comparison between our KaSA and key baselines on NLU tasks is illustrated in the following Table.

As evidenced, our proposed KaSA significantly outperforms all baselines, achieving p-values < 0.05 in 13 out of 18 experimental settings with RoBERTa-base (125M) and 14 out of 18 settings with DeBERTaV3-base (184M). Therefore, the significance tests substantiate the significant improvement of our method over the baselines, verifying the superiority and robustness of our approach.