Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization

Q3: What are the main conclusions of the theoretical analyses in Sections 3 and 4? I found it difficult to follow your proofs. Please include a clear summary of the main conclusions from their theoretical analyses at the end of Sections 3 and 4. I would suggest to provide more intuitive explanations to help readers better understand the theoretical analyses and proofs.

A: According to the traditional statistical learning viewpoint, performance can be defined by the sum of optimization error and generalization error. Our theoretical analyses in Sections 3 and 4 correspond to generalization and optimization, respectively.

In Section 3 (generalization), we give Thereom 1 (Information-theoretic genralization bounds), showing that with the same r value, fine-tuning $W_q,W_v$ consistently achieves results comparable to or even surpassing those of fine-tuning $W_q,W_k,W_v$. This reduces the number of parameters for the same r, while improving generalization bounds and potentially providing memory benefits. (storage-friendly)

In Section 4 (optimization), we discuss the learning dynamics in fine-tuning attention mechanism, and we illustrate (Theorem 2) that the feature learning of attention mechanism is efficient when the learning rate for $W_v$ should be generally much larger than that of $W_q,W_k$ in fine-tuning. (time-friendly)

Building on our experimental and theoretical insights, one can develop new algorithms to improve the effectiveness (e.g., storage, and time) of fine-tuning (Example in Section 5).

Q4: Section 5 appears to be simply fine-tuning ( $W_q$ ) and ( $W_v$ ) with a search for $\lambda$. Do you have any insights on how to determine $\lambda$? could you provide guidelines or heuristics for determining an appropriate $\lambda$ value based on your theoretical and empirical results?

A: As discussed in Remark 4, the optimal ratio $\lambda$ depends on the architecture and the fine-tuning task via the constants in $\Theta$ in Theorem 2. This is a limitation of these asymptotic results since they do not offer any insights on how the constants are affected by the task and the neural architecture.

However, we can still employ some heuristic methods, such as: we can select an appropriate range by conducting a certain amount of experiments, as shown in Figure 1, it seems that a ratio of order $2^1-2^4$ is optimal. （That is why we do a simple search for $\lambda$ (2,4,8) in Section 5.）

Moreover, $\lambda$ should not be too large; otherwise, as shown in the MNLI subplot in Figure 1, the model's performance will collapse ($\eta_{QK}=4e^{-4},\eta_V=2e^{-3}$).

Q2: As you pointed out on line 205, ($W_q$ ) and ( $W_k$ ) can be treated as a single unit. In linear algebra, two matrices multiplied without an intermediate activation can be equivalent to a single matrix. This could explain why fine-tuning only ( $W_q$ ) and ( $W_v$ ) achieves comparable accuracy to tuning all matrices. If this is the case, have you tried fine-tuning only ( $W_k$ ) and ( $W_v$)? It seems that neither the LoRA paper nor your work has conducted this experiment. Could you please include this experiment in your study or explain why not to perform it. This could provide valuable insights and strengthen the paper's analysis.

A: That is a great question!

(1) As you mentioned, in linear algebra, two matrices multiplied without an intermediate activation can be equivalent to a single matrix. Therefore, the effects of fine-tuning $W_q,W_v$ and $W_k,W_v$ should be similar. Please review the supplementary experimental results (#) provided below. (We fine-tune only $W_k$ and $W_v$ of Roberta-base, others are the same to Table 1).

Task	$W_k,W_v$ #	$W_q,W_v$	$W_q,W_k,W_v$
SST2 (r=8)	0.920	0.919	0.922
SST2 (r=16)	0.920	0.921	0.923
QNLI (r=8)	0.887	0.889	0.895
QNLI (r=16)	0.888	0.890	0.890
QQP (r=8)	0.840	0.840	0.844
QQP (r=16)	0.840	0.839	0.844
MNLI (r=8)	0.821	0.820	0.827
MNLI (r=16)	0.822	0.824	0.828

(2) Explain why not to perform it (neither the LoRA paper nor our work)?

In Eq. (1) in Section 2, the conventional attention function includes a term $xW_qW_k^T$, $W_k$ can only exploit the transformed representation matrix $xW_q$ generated by the preceding transformation, thereby losing direct access to the original representation information.

This observation aligns with the insights in [1]: the information representation in LoRA also exhibits significant limitations, as in $\Delta h = xAB$, where $B$ similarly lacks access to the original representation information.

[1] Haotong Qin et al. Accurate LoRA-Finetuning Quantization of LLMs via Information Retention. ICML 2024.

Dear Reviewer TXeN,

We are grateful for your detailed review and the valuable insights you have shared. Your comments are highly appreciated, and we have addressed each point in the following responses.

Q1: My biggest concern with this paper is that the key observation seems to originate from the LoRA paper: "Adapting both ( $W_q$ ) and ( $W_v$ ) gives the best performance overall" (this sentence is quoted from the LoRA paper), and even the experiment setup is similar to Table 5 in the LoRA paper. Deriving key observations from other works isn’t an issue (or saying we share the same observation/insights), but I believe it should be directly and explicitly pointed out, rather than presented as your own observation, concerning the lora paper is very famous for several years and you do cite it. Or could you clarify what differences there are between your work and the LoRA paper? Please provide a detailed comparison between their findings and those in the LoRA paper, highlighting any novel contributions or deeper insights their work provides. This would help clarify the paper's originality and contribution to the field.

A: Thank you for giving us the opportunity to clarify our contributions. The key differences between our work and the LoRA paper are detailed as follows.

(1) Different experimental setup.

• In the LoRA paper (Section 7.1, Table 5), the authors "apply LoRA to different types of attention weights, given the same number of trainable parameters (Trainable Parameters = 18M)" — quoted from the LoRA paper.

Thus, their conclusion that "adapting both ($W_q$) and ($W_v$) gives the best performance overall" is drawn under the constraint of having the same total trainable parameters. (1. $W_q,W_v$ with rank 4; 2. $W_q,W_k,W_v,W_o$ with rank 2)

• In contrast, our study, as presented in Table 1, compares fine-tuning different types of attention weights under the same fine-tuning rank values (each row in Table 1 illustrates this comparison).

Based on this setup, one of our key insights is that, when using the same rank rrr, fine-tuning only the $W_q$ and $W_v$ matrices is not only computationally more efficient but also delivers results comparable to, or even surpassing, fine-tuning all three matrices ($W_q$, $W_k$, and $W_v$).

(2) Different conclusions.

• Empirically, in LoRA paper, given the same number of trainable parameters, "it is preferable to adapt more weight matrices than adapting a single type of weights with a larger rank". (this sentence is quoted from the LoRA paper)

• In our paper, given the same rank value in fine-tuning. (a) Empirically, fine-tuning both Wq and Wv often leads to performance that matches or even exceeds that achieved by fine-tuning all three matrices $W_q$, $W_k$, and $W_v$. (b) Theoretically, in Section 3.2 (Thereom 1), we provide generalization bounds for adapting $W_q, W_v$ and/or $W_k$. Specifically, fine-tuning only $W_q$ and $W_k$ reduces the number of parameters for the same rank rrr, while improving generalization bounds and offering potential memory advantages.

(3) This, however, represents just a fraction of our contributions detailed in Section 3. (For further details, please refer to our responses A through Q3.)

Dear Reviewer TXeN,

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

As the day that authors may upload a revised PDF phase nears completion, we wish to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago and hope they have adequately resolved any issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best regards.

Dear Reviewer TXeN,

As we approach the rebuttal discussion deadline, we would like to follow up with you regarding the additional ablation experiments, theoretical (generalization and optimization) analysis, and writing updates we provided. Have these addressed your concerns? If so, we kindly ask if you would consider revising your review score accordingly.

Thank you for your time and thoughtful evaluation of our work.

Best regards.

W2： For the "Different Impact", the authors claimed that fine-tuning Wq&Wv is favored over Wq,Wk,Wv together. However, this argument is not supported by their experimental results in Table 1. It can be seen from the table that fine-tuning Wq,Wk,Wv together still gets the best performance for most of the cases. I wonder if the authors are writing the arguments and conducting experiments separately without any discussion. The authors should summarize their findings according to the experiment results.

Q1： In line 160~193, the author writes "As seen in the table, we can see a clear trend where solely updating the Wv matrix outperforms just learning the Wq,Wk matrix. Interestingly, the combination of fine-tuning both Wq and Wv often leads to performance that matches or even exceeds that achieved by fine-tuning all three matrices Wq,Wk, and Wv". This statement is not supported by the results in Table 1, as solely updating the Wv is inferior to updating Wq,Wk together, and fine-tuning all three matrices Wq,Wk, and Wv achieves the best result in most cases. The authors should first rewrite the whole section of "Different Impact" to have a consistent argument with the experiment results, and investigate more tasks to see whether there are certain types of tasks in which "different impact" is true.

A: Regarding your comment on the "Different Impact" phenomenon and the results presented in Table 1, we believe there may have been a misunderstanding in interpreting the data. Please allow us to clarify the experimental results and our arguments.

Specifically, in our original statement:

(1) “We can see a clear trend where solely updating the Wv matrix outperforms just learning the Wq,Wk matrix” .

(See the third and fourth columns)

(2) “Interestingly, the combination of fine-tuning both Wq and Wv often leads to performance that matches or even exceeds that achieved by fine-tuning all three matrices Wq, Wk, and Wv”.

(See the fifth and sixth columns)

In your feedback:

(1) "as solely updating the Wv is inferior to updating Wq,Wk together. "

Compare the third (Wv) and fourth (Wq,Wk together) columns of the Table 1, the values in the third column are generally superior to those in the fourth column.

(2) "fine-tuning all three matrices Wq,Wk, and Wv achieves the best result in most cases."

This statement is correct, but our main point is that the performance is comparable to, and in some cases may even exceed (especially fine-tuning Llama3.1-8b on QQP, MNLI), rather than being the best.

Once again, thank you for your valuable feedback. We hope our clarification addresses your concerns and resolves any possible misunderstandings.

Dear Reviewer f8uV,

Thank you for taking the time to review our work and for offering thoughtful comments and constructive questions. Your feedback is instrumental in refining the manuscript. Please see our detailed replies below.

W1： The writing and presentation are rather poor. Both sentence level and logical level polishment are suggested for this work. E.g. In line 16~22, "In this paper, we investigate two remarkable phenomena ... with a higher learning rate for the Wv matrix expediting convergence." The ordering of the sentences and the way of structuring the arguments add unnecessary difficulty to reading. This kind of problem happens across the whole manuscript. A suggestion is, use multiple statement sentences instead of clauses. A rewritten version of these sentences is " In this paper, we investigate two remarkable phenomena associated with attention mechanism, during the fine-tuning of LLMs. One is 'Different Impact' and another is 'Efficient Convergence'. ......" Another point is that the naming of "Different Impact" and "Efficient Convergence" doesn't help with understanding. Consider other ways to name them.

A： We appreciate your feedback regarding the clarity and structure of the writing, particularly with respect to sentence flow and logical organization. In response, we will revise the relevant sections to simplify the sentence structures and ensure that the arguments are presented in a clearer and more direct manner. Below is the preliminary version of the revised text.

In this paper, we investigate two notable phenomena related to the attention mechanism during the fine-tuning of large language models (LLMs). The first phenomenon, termed "Effect of Weight Combination on Performance," highlights the impact of fine-tuning different weight matrices. It demonstrates that optimizing the Wv matrix leads to a more significant performance improvement compared to optimizing the Wk matrix. Fine-tuning only the Wq and Wv matrices is not only computationally more efficient but also yields results comparable to, or even better than, fine-tuning all three matrices (Wq, Wk, and Wv). The second phenomenon, "Impact of Learning Rate on Convergence," emphasizes the crucial role of distinct learning rates for these matrices in achieving optimal performance, with a notably higher learning rate for the Wv matrix compared to Wq and Wk, which accelerates convergence.

W1: Generalizability of the Proposed Approach: The primary concern is the generalizability of the proposed approach. Specifically, the authors could enhance the analysis by demonstrating that the proposed method consistently improves LLM performance across diverse scenarios. To this end, it would be beneficial to include performance results under more complex, open-ended generation tasks, such as MT-Bench or comparable challenging benchmarks. Additionally, considering variations in model behavior, evaluating the approach on a broader range of LLMs, such as Mistral, would further strengthen the claim of general applicability.

A: Good suggestions !

(1) In alignment with the experimental setup (hyperparameter setting for Llama3.1-8b) described in our Section 4.1, Figure 2, we have evaluated the RTE and MNLI task performances of our approach on Mistral-7B (you mentioned). Below are the results from this additional analysis:

($\eta_{QK}=1e^{-5},\eta_V=\lambda\eta_{QK}$)

Method	Hyperparameter	RTE	MNLI
LoRA (QKV)	$r=16$,$\lambda=1$	81.28	87.80
LoRA (QV)	$r=16$, $\lambda=1$	80.51	88.87
LoRA (QV)	$r=16$, $\lambda=2$	81.59	89.04
LoRA (QV)	$r=16$, $\lambda=4$	80.87	88.64
LoRA (QV)	$r=16$, $\lambda=8$	83.75	88.78

(2) Given that our work is on the theoretical analysis of attention mechanisms during fine-tuning and improving various fine-tuning methods, and MT-Bench [1] is designed as an evaluation benchmark dataset (the dataset contains only 80 samples), could you kindly elaborate on why you recommend this specific task? Are you suggesting that we fine-tune and evaluate our approach on this benchmark, or is there a particular aspect of MT-Bench that you believe would align with our theoretical contributions? Your clarification would help us better understand and address your suggestion.

[1] https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge#mt-bench

W2: Visualization for Better Insight: To deepen the analysis and understanding of the findings, visualizing the learned attention distributions across different settings could be valuable. By examining how attention distributions vary under different configurations, the authors could offer a more nuanced understanding of the observed effects, shedding light on the underlying phenomenon.

A: We appreciate the suggestion to visualize attention distributions to deepen the analysis. However, due to the complexity of the model architecture (e.g., multi-layer and multi-head attention) and the variability of attention scores across different input data, it may be challenging to establish a standardized evaluation metric for determining the quality of the visualized attention distributions.

We wonder if we might have misunderstood your suggestion. Could you kindly clarify or provide further details? Your guidance would greatly help us in addressing your point effectively and improving our work.

Dear Reviewer SWzh,

We sincerely appreciate your thorough review and the insightful comments you provided. Your feedback is invaluable in improving our paper. Please find our detailed response below.

Q1: How will the proposed method perform on other PEFT techniques, such as DoRA [1]?

[1] Liu, Shih-Yang, et al. "Dora: Weight-decomposed low-rank adaptation." arXiv preprint arXiv:2402.09353 (2024).

A: Following the experimental setup described in our Section 5, Table 2, we conducted additional evaluations on DoRA. Below are the results of our supplementary experiments:

Method	Trainable #Param (M)	RTE	STS-B	MRPC	CoLA	MNLI	SST-2	QQP	QNLI
Full Fine-tune (QKV)	21.85	73.64	90.49	84.55	60.34	86.68	93.23	90.48	92.37
LoRA (QKV) $r=8$	1.62	70.76	90.25	85.04	58.03	86.70	93.92	89.15	92.17
LoRA (QKV) $r=16$	2.07	70.39	90.25	86.03	58.04	86.78	93.92	89.26	92.18
DoRA (QKV) $r=8$	1.06	70.75	90.39	85.78	56.79	86.73	93.58	89.34	92.22
DoRA (QKV) $r=16$	1.51	70.40	90.31	86.03	57.81	86.77	93.92	89.30	92.48
Full Fine-tune (QV) $\lambda=2$	14.76	73.53	91.01	86.02	60.57	62.03	93.11	90.56	91.96
Full Fine-tune (QV) $\lambda=4$	14.76	72.29	90.56	87.01	61.88	35.44	91.05	89.81	88.85
Full Fine-tune (QV) $\lambda=8$	14.76	72.29	90.02	88.97	61.86	35.44	84.75	85.93	50.54
LoRA (QV) $r=8, \lambda=2$	1.48	71.84	90.37	86.02	58.54	86.85	94.03	89.47	92.33
LoRA (QV) $r=8, \lambda=4$	1.48	75.09	90.83	87.01	59.56	86.95	94.04	90.09	92.86
LoRA (QV) $r=8, \lambda=8$	1.48	76.13	90.75	88.97	61.88	86.93	93.46	90.01	92.34
LoRA (QV) $r=16, \lambda=2$	1.77	70.39	90.46	86.03	58.55	86.83	94.38	89.77	92.33
LoRA (QV) $r=16, \lambda=4$	1.77	76.17	91.05	87.99	60.06	87.19	94.03	90.30	92.73
LoRA (QV) $r=16, \lambda=8$	1.77	72.92	90.96	89.95	59.31	87.31	93.92	90.43	92.95
DoRA (QV) $r=8, \lambda=2$	0.90	71.12	90.29	87.01	58.54	87.08	93.96	89.60	92.60
DoRA (QV) $r=8, \lambda=4$	0.90	75.45	90.82	86.76	60.32	86.98	93.81	90.33	92.97
DoRA (QV) $r=8, \lambda=8$	0.90	70.76	90.38	87.75	57.01	87.12	94.15	90.45	92.48
DoRA (QV) $r=16, \lambda=2$	1.20	69.68	90.53	87.75	59.31	87.09	93.92	89.68	92.70
DoRA (QV) $r=16, \lambda=4$	1.20	76.16	90.77	88.48	60.84	86.96	94.15	90.34	93.01
DoRA (QV) $r=16, \lambda=8$	1.20	77.26	90.83	88.96	60.32	87.10	94.17	90.46	92.80

Dear Reviewer SWzh,

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

As the day that authors may upload a revised PDF phase nears completion, we wish to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago and hope they have adequately resolved any issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best regards.

Dear Reviewer SWzh,

We are eager to know if you have any additional concerns or suggestions regarding our paper. If there are none, we sincerely hope you might consider raising the review score, and we would be more than willing to engage in detailed technical communication to address any potential concerns.

Best regards.

Dear Reviewer FTN2,

Thank you for your careful evaluation and thoughtful suggestions. We deeply value your time and effort in providing such constructive feedback. Below, we address your comments point by point.

Q1 (W1): Baseline Performance: Could you provide the performance metrics of the base model before fine-tuning for each of the tasks? This would help in better understanding the relative improvements brought by your fine-tuning strategy.

A: The performance (100%) of the base model for each task is presented in the table below.

Roberta-base：

RTE	STS-B	MRPC	CoLA	MNLI	SST-2	QQP	QNLI
45.12	-3.18	66.66	1.09	32.95	49.31	44.72	50.81

Llama3.1-8b：(Notice that the MNLI task for Llama3.1-8b is still a complex task.)

QQP	MNLI
55.08	33.34

Q2 (W2): More Challenging Evaluations: Have you considered evaluating the LLaMA3.1-8B model on more complex benchmarks, such as tasks involving coding or mathematical problem? Including such challenging tasks would make your results more comprehensive and convincing.

A： We completely agree that evaluating the model on more challenging benchmarks is essential for a comprehensive understanding of its capabilities. To address this, we follow [1] to fine-tune the LLaMA3.1-8B model on the MetaMathQA [1] dataset (the training set consists of the first 10K samples selected from the 150K MetaMathQA dataset.) and evaluate the performance on the GSM8K [2] (a benchmark for mathematical problem-solving).

Method	GSM8K (100%)
Before fine-tune	25.55
LoRA (QKV) $r=16,\lambda=1$	57.70
LoRA (QV) $r=16,\lambda=2$	59.15
LoRA (QV) $r=16,\lambda=4$	58.23

[1] Longhui Yu, et al. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. ICLR 2024.

[2] Karl Cobbe, et al. "Training Verifiers to Solve Math Word Problems." arXiv preprint arXiv:2110.14168 (2021).

Q3 (W3): Ablation Studies: It would be helpful if you could add more ablation studies to isolate the effects of different components of the proposed method, such as the specific impact of Wq vs. Wv fine-tuning or the effect of learning rate scaling. This would provide a clearer picture of what drives the observed improvements.

A: Good suggestions! Thank you for giving us the opportunity to clarify the ablation studies presented in the main text.

(1) For the effects of different components ( such as Wq vs. Wv ).

In Section 3.1 (Table 1), we provide a detailed comparison of the impact of fine-tuning different weight matrices.

We can see a clear trend where solely updating the $W_v$ matrix outperforms just learning the $W_q,W_k$ matrix. Interestingly, the combination of fine-tuning both $W_q$ and $W_v$ often leads to performance that matches or even exceeds that achieved by fine-tuning all three matrices $W_q,W_k,$ and $W_v$. This pattern is consistently observed across various tasks and rank values, further emphasizing the importance of these two matrices over $W_k$ during fine-tuning.

(2) For the effect of learning rate scaling.

In Section 4.1 (Figure 1 and Figure 2), we compared the results of various learning rate combinations ($\eta_V,\eta_{QK}$).

We observe that (1) test accuracy consistently reaches its maximum for certain sets of learning rates where $\eta_{QK}< \eta_V$, outperforming the standard practice of setting $\eta_{QK}$ and $\eta_V$ equal. (2) More interestingly, the gap between the optimal choice of learning rates overall and the optimal choice when $\eta_{QK}=\eta_V$ varies across different tasks.

Q4 (W4): Hyperparamete λ: Could you provide more guidance on how to choose the hyperparameter λ?

A: As discussed in Remark 4, the optimal ratio $\lambda$ depends on the architecture and the fine-tuning task via the constants in $\Theta$ in Theorem 2. This is a limitation of these asymptotic results since they do not offer any insights on how the constants are affected by the task and the neural architecture.

However, we can still employ some heuristic methods, such as: we can select an appropriate range by conducting a certain amount of experiments, as shown in Figure 1, it seems that a ratio of order $2^1-2^4$ is optimal. （That is why we do a simple search for $\lambda$ (2,4,8) in Section 5.）

Moreover, $\lambda$ should not be too large; otherwise, as shown in the MNLI subplot in Figure 1, the model's performance will collapse ($\eta_{QK}=4e^{-4},\eta_V=2e^{-3}$).

Dear Reviewer FTN2,

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

As the day that authors may upload a revised PDF phase nears completion, we wish to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago and hope they have adequately resolved any issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you.

Best regards.

(3) The ablation studies presented in the main text.

(i) For the effects of different components ( such as $W_q$ vs. $W_v$ ).

In Section 3.1 (Table 1), we provide a detailed comparison of the impact of fine-tuning different weight matrices.

We can see a clear trend where solely updating the $W_v$ matrix outperforms just learning the $W_q,W_k$ matrix. Interestingly, the combination of fine-tuning both $W_q$ and $W_v$ often leads to performance that matches or even exceeds that achieved by fine-tuning all three matrices $W_q,W_k,$ and $W_v$. This pattern is consistently observed across various tasks and rank values, further emphasizing the importance of these two matrices over $W_k$ during fine-tuning.

(ii) For the effect of learning rate scaling.

In Section 4.1 (Figure 1 and Figure 2), we compared the results of various learning rate combinations ($\eta_V,\eta_{QK}$).

We observe that (a) test accuracy consistently reaches its maximum for certain sets of learning rates where $\eta_{QK}< \eta_V$, outperforming the standard practice of setting $\eta_{QK}$ and $\eta_V$ equal. (b) More interestingly, the gap between the optimal choice of learning rates overall and the optimal choice when $\eta_{QK}=\eta_V$ varies across different tasks.

(4) For more ablations to demonstrate.

We have added ablation experiments on different fine-tuning methods (Dora [1]) and models (Mistral-7B [2]). Please refer to our supplementary experiments for details.

Following the experimental setup (hyperparameter setting for Llama3.1-8b) described in our Section 5, Table 2, we conducted additional evaluations on DoRA: (Revision line 493- line 510)

Method	Trainable #Param (M)	RTE	STS-B	MRPC	CoLA	MNLI	SST-2	QQP	QNLI
DoRA (QKV) $r=8$	1.06	70.75	90.39	85.78	56.79	86.73	93.58	89.34	92.22
DoRA (QKV) $r=16$	1.51	70.40	90.31	86.03	57.81	86.77	93.92	89.30	92.48
DoRA (QV) $r=8, \lambda=2$	0.90	71.12	90.29	87.01	58.54	87.08	93.96	89.60	92.60
DoRA (QV) $r=8, \lambda=4$	0.90	75.45	90.82	86.76	60.32	86.98	93.81	90.33	92.97
DoRA (QV) $r=8, \lambda=8$	0.90	70.76	90.38	87.75	57.01	87.12	94.15	90.45	92.48
DoRA (QV) $r=16, \lambda=2$	1.20	69.68	90.53	87.75	59.31	87.09	93.92	89.68	92.70
DoRA (QV) $r=16, \lambda=4$	1.20	76.16	90.77	88.48	60.84	86.96	94.15	90.34	93.01
DoRA (QV) $r=16, \lambda=8$	1.20	77.26	90.83	88.96	60.32	87.10	94.17	90.46	92.80

In alignment with the experimental setup (hyperparameter setting for Llama3.1-8b) described in our Section 4.1, Figure 2, we have evaluated the RTE and MNLI task performances of our approach on Mistral-7B :

Method	Hyperparameter	RTE	MNLI
LoRA (QKV)	$r=16$，$\lambda=1$	81.28	87.80
LoRA (QV)	$r=16$, $\lambda=1$	80.51	88.87
LoRA (QV)	$r=16$, $\lambda=2$	81.59	89.04
LoRA (QV)	$r=16$, $\lambda=4$	80.87	88.64
LoRA (QV)	$r=16$, $\lambda=8$	83.75	88.78

[1] Liu, Shih-Yang, et al. Dora: Weight-decomposed low-rank adaptation. ICML Oral 2024.

[2] https://github.com/mistralai

Thank you once again to all the reviewers for your patience and invaluable comments. We hope our responses have clarified your initial concerns and questions. We are happy to provide further clarifications if necessary.