We appreciate the constructive suggestions provided by the reviewer. We provide a detailed explanation and experimental analysis as follows.

W1: the authors must reference the algorithm they use for “de-cycling” the graph, describing its steps at least in the appendix.

We provide a detailed description of the de-cycling process in the Global Response and present the algorithm's pseudocode in the PDF. This detailed description will be included in the final version of the paper.

Q1: line 143 to 146: The authors claim that the weight assigned to singular values of high loss should be small. However, from equation (2), it seems that the steps with higher loss receive the larger weight in the time window.

We sincerely apologize for the error of weight calculation formula presented in our paper. The correct formula should be $w^{(i)}=\frac{\sum^n_{j=0}R(P^{(j)}_a,Q^{(j)}_a)}{R(P^{(i)}_a,Q^{(i)}_a)}$. By calculating in this way, singular values with higher losses are assigned lower weights, thereby enhancing the effectiveness of importance assessment with magnitude in a time-series. We thank the reviewer for identifying this correction, and we will amend it in the final camera-ready version of the manuscript.

Q2: Line 148. What do the authors mean by “we normalize the weights from 0 to 1”? An equation would be helpful to clarify the operation here.

We appreciate the reviewer’s suggestion. Here, we normalize the weights of the singular values to prevent large discrepancies in weight values from affecting the evalution. We use Min-Max normalization, and the calculation formula is as follows,

$w^{(i)}_{norm}=\frac{w^{(i)}-w\_{min}}{w\_{max}-w\_{min}}$

where $w_{min}$ and $w_{max}$ represent the minimum and maximum values among $w^{(0)}$ to $w^{(n)}$, respectively. This normalization method ensures a uniform scale for all weights, thereby enhancing the robustness of our results. A comprehensive description of this normalization technique will be included in the camera-ready version of the manuscript.

Q3: Given a fixed parameter budget what is the amount of VRAM consumed by SalientLORA compared to for instance AdaLora?

We appreciate the constructive suggestions provided by the reviewer. To compare the memory usage of AdaLoRA and SalientLoRA, we fine-tune the DeBERTaV3-base and BART-large models on the CoLA and XSum tasks respectively. We set the target rank $r_t$ to 144 and the initial total rank $r_i$ to 7.5 times $r_t$. The results in the table below indicate that the training overheads for both fine-tuning methods are quite close, with SalientLoRA exhibiting a marginal increase in memory occupancy of only 0.11%-0.22%. This slight difference arises because SalientLoRA needs to store all the singular values within the time window for salience assessment. However, as each matrix retains only 2-15 singular value, the impact on memory is minimal.

Q4: When the authors state that AdaLORA has been adopted in numerous research studies (line 50) they should cite at least the most relevant ones to support the claim.

We are grateful for the reviewer's reminder. Numerous studies have shown that applying AdaLoRA achieves excellent fine-tuning performance in various scenarios, including speech models [3] and several other languages models [1, 2]. We will include these references in the final version of the paper.

[1] Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. Xu L et al., 2023. In arXiv preprint.

[2] AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning. Zhou et al., 2024. In Transactions of the Association for Computational Linguistics.

[3] Whisper-med_15k. 2024. https://huggingface.co/sin2piusc/whisper-med_15k.

Q5: What do the authors mean by “dominant role of singular values in the SVD matrix”? What is the precise meaning of “dominant role” in this context?

After performing singular value decomposition (SVD), the singular values represent the primary characteristics of a matrix. Larger singular values contain more information from the matrix and capture more of its essential features, thereby possessing greater importance. Consequently, singular values play a dominant role in the SVD matrix. In light of this, our method utilizes the magnitude of these singular values to assess their significance.

In the final version of the paper, we will include the following limitations.

The proposed mechanism of adaptive time-series window effectively accelerates and stabilizes the fine-tuning process. However, this mechanism employs a relatively rigid adaptive approach, considering only the time step information and focusing on progressively cautious pruning as the process advances. Future work will integrate the consideration of singular values' detailed dynamics, including momentum and interdependencies, to enable a more flexible and adaptive adjustment process.

Thank you very much for your feedback. We appreciate your thorough and responsible review.

Best regards.

Question: To fully evaluate the robustness of the proposed method, could you provide detailed ablation studies and analyses for the hyperparameters, including β, γ, Ti, and Tf?

We appreciate the insightful suggestions provided by the reviewer. In response, we conduct additional experimental analyses on two sets of hyperparameters to explore their impact. These experiments are conducted on the CoLA and MRPC datasets to fine-tune the DeBERTaV3-base model. The total target rank is set to 144, and all the other parameters are consistent with the main experiments.

(1) $\beta$ and $\gamma$

$\beta$ and $\gamma$ are thresholds that control the relevance and dependency relationships of singular values, respectively. To explore their effects, we keep other hyperparameters constant and change the values of $\beta$ and $\gamma$ separately to observe the timing and effects of SalientLoRA fine-tuning. The results are presented in the table below, where MCC represents Matthew’s Correlation Coefficient and Acc denotes accuracy.

The performance exhibits minimal sensitivity to variations in the values of $\beta$ and $\gamma$. This insensitivity stems from the decycling operation for the dependency graph , which effectively eliminates the majority of redundant dependency relationships. However, setting excessively low values for $\beta$ and $\gamma$​ can lead to a large number of redundant dependencies in the graph, which increases the time cost of the decycling process and thus impacts the efficiency of fine-tuning. Therefore, we select hyperparameter values that ensure high efficiency of fine-tuning, with $\beta$=0.9 and $\gamma$=2.

(2) $T_i$ and $T_f$

$T_i$ and $T_f$ control the sizes of the initial and final time windows in the adaptive time-series window mechanism, respectively. The results indicate that the impact of $T_i$ on performance is minimal, with differences only ranging from 0.02% to 0.09%. However, as $T_i$ increases, the fine-tuning time significantly lengthens. This is due to a slower reduction in rank during the early stages of fine-tuning, which impacts the efficiency of rank allocation.

Moreover, when $T_i$ remains constant and $T_f$ increases from 100 to 250, there is a slight improvement in performance, while the fine-tuning time remains relatively unchanged. This improvement can be attributed to the significance analysis in the later stages of fine-tuning, which incorporates singular values under more time steps, yielding more reliable allocation outcomes. Therefore, we set $T_i=10$ and $T_f=200$ to achieve a balance between performance and efficiency.

Dear Reviewer i7vj,

Thank you for your thorough and valuable feedback. We have carefully addressed each of your concerns. We hope that our clarifications provide satisfactory answers to your questions. We are committed to improving our work based on your insights, and we look forward to your response.

Best regards, The Authors

We appreciate the constructive feedback provided by the reviewer. We provide a detailed explanation and experimental analysis as follows.

W1: The article contains some details that are not clearly explained, such as how the R function on line 145 is calculated, and what specifically is done in the de-cycling process introduced on line 165.

(1) Function $R(\cdot)$

In our paper, we have elaborated on the function $R(\cdot)$ for calculating the orthogonalization loss in Equation 1. Specifically, $R(\cdot)$ measures the degree of orthogonality of $**P**$ and $**Q**$. The equation is as follows:

$R(**P**,**Q**)=||**P**^T**P**-**I**||^2_F+||**Q****Q**^T-**I**||^2_F$

where $**P**,**Q**$ denote the left and right singular matrices, respectively.

(2) The De-cycling Process

Due to space constraints, detailed content and analysis of the de-cycling process are provided in the Global Response.

W2：More analysis could be provided to interpret why the salience measurement works well. For example, are the average of influence domains consistent across models fine-tuned on different types of datasets?

Due to space constraints, detailed content and analysis are provided in the Global Response.

Q1: Taking the last row of Table 1 as an example, initially, the model uses a total rank that is 7.5 times the target rank, so the gpu memory usage is roughly equivalent to that of LoRA with r=8*7.5=60. Although the memory usage may decrease during the model optimization process, can you consider comparing with methods like LoRA and DoRA with r=60?

We set the average target rank of each matrix in SalientLoRA to 8, and the initial rank to 60, to compare memory usage and fine-tuning performance with $\text{LoRA}\_{r=8}, \text{DoRA}\_{r=8}, \text{LoRA}\_{r=60}, \text{DoRA}_{r=60}$. Specifically, we fine-tune the DeBERTaV3-base and BART-large models on the CoLA and XSum datasets using the aforementioned settings.

The results, as presented in the table below, demonstrate that small changes in rank settings do not significantly impact memory usage. Memory usage of SalientLoRA peaks at only 5.41GB and 23.22GB, which are merely 0.33GB and 0.86GB higher than that of $\text{LoRA}\_{r=8}$. Furthermore, the memory consumption of SalientLoRA rapidly decreases due to rank pruning, ultimately aligning closely with that of $\text{LoRA}\_{r=8}$. Notably, under this setting, our method still achieves the best results, surpassing those of $\text{LoRA}\_{r=60}$ and $\text{DoRA}\_{r=60}$ by 0.9% and 0.57%, respectively. This also underscores the superior performance of our approach.

Q2: Based on my understanding, the constructed influence domains form an undirected simple graph. If this graph forms a single cycle, how do you perform de-cycling?

In practice, the constructed dependency graph of singular values is a directed graph, where nodes represent singular values and edges represent their dependencies. This can be explained from two aspects.

(1) The dependencies are quantified through slopes between two singular values within a time-series, where the choice of independent and dependent variables significantly influences the slope calculations. For instance, consider two singular values, $\lambda_a$ and $\lambda_b$. If $\lambda_a$ is chosen as the independent variable and $\lambda_b$ as the dependent variable, the resulting slope $k_{ab}$ quantifies the influence of $\lambda_a$ on $\lambda_b$. Conversely, selecting $\lambda_b$ as the independent variable results in the slope $k_{ba}$. Since $k_{ab}$ and $k_{ba}$ are numerically distinct, the weights of the edges from $\lambda_a$ to $\lambda_b$ and $\lambda_b$ to $\lambda_a$ also differ, thus forming a directed graph.

(2) Intuitively, the dependency relationship between two singular values is unidirectional: if a singular value is of significant importance, the variations of its magnitude can impact changes in the other one, reverse is not necessarily true.

Q3：Do you calculate influence domains starting from vertices with a degree of 0, similar to topological sorting, and then update the degrees of the vertices connected to it, then repeat the process

The calculation of influence domains resembles a reverse topological sort. We calculate the influence domains of all vertices by proceeding backwards through the graph. Specifically, the process initiates from vertices with an out-degree of zero. These vertices do not influence any other nodes, so their influence domain is directly assigned a value of 1. Subsequently, the influence domain of a node that has subsequent nodes is calculated as the weighted sum of the influence domains of these downstream nodes. This iterative procedure continues until the influence domains for all nodes in the graph have been determined.