We sincerely appreciate the reviewer’s recognition of our work and the constructive feedback and insightful questions provided. Below, we address the concerns and questions raised.

---

R1. Performance improvement over prior methods like PiSSA Thank you so much for your comments. Our method consistently outperforms other approaches—including PiSSA—in terms of accuracy, often with comparable or significantly fewer trainable parameters.

• For Image Classification (IC), our method achieves over 6% higher accuracy than PiSSA on ViT-Base with a similar parameter budget.
• For Natural Language Understanding (NLU), our method achieves over 7% higher accuracy than PiSSA on RoBERTa-Large, while using much fewer parameters. In our newly added experiments (as suggested by the reviewer), we have included results for PiSSA with $r=1$ (see Tables 1–2). These results show that, under similar parameter conditions, our method performs consistently across tasks on both RoBERTa-Large and RoBERTa-Base, and achieves substantially higher average performance compared to PiSSA.
• For Natural Language Generation (NLG) (see additional results in the supplementary material), our method outperforms PiSSA on LLaMA 2-7B by: 7% higher accuracy on GSM8K, and 1.7% higher accuracy on MATH, while using fewer trainable parameters.

Below we summarize tasks where our method achieves at least 3% higher accuracy on average compared to other methods. Bold entries indicate cases where AdaMSS reduces the number of trainable parameters by more than 75% yet still achieves superior results.

---

R2. The cost of initialization of AdaMSS In our initialization phase, the most time-consuming step is the estimation of the number of subspaces. As discussed in our ablation study (see Appendix), when the number of subspaces is fixed to $K = 10$, the performance of our method with fixed $K$ is comparable to AdaMSS with subspace number estimation strategy.

In the current submitted supplementary material, we further compare AdaMSS without subspace estimation to other PEFT methods (including PiSSA, LoRA, LoRA-PRO, and LoRETTA) on LLaMA 2-7B, Mistral-7B, and Gemma-7B. In most cases, our method still achieves the best performance, even without estimating the number of subspaces. To make AdaMSS even more practical, we also apply low-rank SVD techniques to accelerate the subspace initialization process.

As shown in the current submitted supplementary material, our method initializes faster than LoRETTA and completes in just a few seconds on LLaMA 2-7B, highlighting its computational efficiency.

---

R3. Comparison with random initialization We have included a comparison between random initialization and orthogonal initialization in the current submitted supplementary material (Figure 1). As shown by our experimental results, orthogonal initialization significantly outperforms random initialization.

---

R4. Training time/memory comparisons Owing to space constraints, please see our response given in R1 for Reviewer J6fY.

---

R5. Clarifying generalization and experiments that demonstrate generalization The generalization ability of a method is reflected in its expected test error. As established in Section Analysis of AdaMSS, AdaMSS exhibits a lower upper bound on the expected test error compared to other methods, given the same number of training samples. This theoretical advantage is further supported by our experimental results, which consistently show lower average test errors for AdaMSS across a range of tasks.

---

R6. Advantages of AdaMSS As discussed in R1 and R4–R5, our method offers the following key advantages.

• (1) Superior Accuracy Across Tasks.
• (2) Reduced Training Time and Memory Usage.
• (3) Stronger Theoretical Guarantee: Theorem 1 establishes an upper bound on the expected loss of AdaMSS, providing a formal generalization guarantee that supports its theoretical soundness.

---

R7. Comparison with equivalent trainable parameter counts for LoRA and PiSSA for language models Thank you so much for the valuable suggestion. Our original comparisons on ViT models already included PiSSA $(r=1)$ as a baseline.

Following your recommendation, we have extended our experiments on language models (RoBERTa) to include LoRA $(r=1)$ and PiSSA $(r=1)$ under minimal parameter settings, in addition to our method AdaMSS$_{base}$. Please refer to Tables 1 and 2 below.

As the experimental results demonstrate, our method consistently outperforms both LoRA and PiSSA under a low-parameter budget. Furthermore, AdaMSS exhibits more stable performance across tasks, highlighting the advantage of structured subspace updates in resource-constrained settings.

Table 1. Results of LoRA $(r=1)$, PiSSA $(r=1)$, and AdaMSS$_{base}$ on RoBERTa-Base

Table 2. Results of LoRA $(r=1)$, PiSSA $(r=1)$, and AdaMSS$_{base}$ on RoBERTa-Large

---

R8. Add citation [1] and comparison with [1] We sincerely thank the reviewer for bringing this excellent work to our attention. WeGeFT is recently accepted method at ICML 2025. In the revised version of our manuscript, we have included a discussion of WeGeFT [1] and added experimental comparisons against it.

Below, we provide a summary of the differences between AdaMSS and WeGeFT, as well as empirical comparisons on Natural Language Understanding (NLU) and Image Classification (IC) tasks.

• (1) Motivation: AdaMSS is motivated by the multi-subspace structure of pretrained weight matrices. It seeks compact representations via lowest-rank representation and subspace segmentation, followed by a multi-subspace-based adaptive budget allocation strategy to reduce the number of trainable parameters while preserving model capacity.

In contrast, WeGeFT focuses on cross-layer parameter sharing, under the assumption that pretrained weights already contain useful "transferable knowledge" for downstream tasks.

• (2) Methodology: AdaMSS leverages low-rank representation and subspace segmentation to obtain a compact parameterization of the weights: $\mathbf{W}^{(k)} = \mathbf{W}_0^{(k)} + \mathbf{\hat{W}}_0^{(k)} \Delta \mathbf{Z}^{(k)}$, where $\mathbf{W}^{(k)}$ consists of the weights belonging to the $k$-th subspace, $\Delta \mathbf{Z}^{(k)}$ is low-rank matrix, and $\mathbf{\hat{W}}_0^{(k)}$ represents the principal components of the corresponding weight subspace.

  WeGeFT adopts the formulation $\mathbf{W}^l = \mathbf{W}_0^{l} + \mathbf{W}_0^{l} \Delta \mathbf{Z}$, where $\Delta \mathbf{Z}$ is a low-rank matrix shared across layers, and $\mathbf{W}^l$ denotes the weight matrix of the $l$-th layer.

Comparison with WeGeFT

Table 3 compares the performance of AdaMSS and WeGeFT on NLU tasks.
Table 4 shows their results on IC tasks.
Overall, AdaMSS outperforms WeGeFT on average for NLU (RoBERTa) and has a comparable performance for IC (ViT).

Table 3. Comparison with WeGeFT on NLU (RoBERTa)

Table 4. Comparison with WeGeFT on IC (ViT)

---

R9. Which modules are fine-tuned?

For both IC and NLU, we fine-tune only the Query and Value projection modules across all methods for a fair comparison. For Natural Language Generation, we evaluate two settings: fine-tuning only the Query and Value projections, and fine-tuning all major projection modules.

---

We thank the reviewer again for the thoughtful questions. Please let us know if any aspects remain unclear—we would be truly grateful for the opportunity to further clarify.

We sincerely appreciate the reviewer’s recognition of our work and the constructive feedback and insightful questions provided. Below, we address the concerns and questions raised.

---

R1. Clarity of presentation

We appreciate the suggestions. Following the recommendations, in the revised version, we will include a method diagram to summarize the proposed adaptation process and improve readability by simplifying the notation and derivations.

For the experimental section, we will reorganize the existing ablation studies and introduce a new discussion section to better highlight and explain the advantages of our method.

We believe that these revisions will significantly enhance the clarity of our work.

---

R2. Intuitive explanation of importance scores

The importance score used in this work is defined based on the product of $\bar{I}^{(t)}$ and $\bar{U}^{(t)}$, where:

• $\bar{I}^{(t)}$ measures the sensitivity of the weights, and it is defined on the magnitude of the gradient-weight product, which approximates the change in the loss function due to that weight.
• $\bar{U}^{(t)}$ captures the uncertainty in the sensitivity estimation.

Therefore, this importance score helps identify weights that either (1) contribute significantly to loss reduction, or (2) exhibit uncertainty, making them important candidates for training.

---

R3. Clarifying generalization

The superior generalization ability of a method refers to the following:
Given the same number of training samples and comparable training loss, the method achieves a lower expected test error than competing approaches.

To support this, Theorem 1 provides a generalization bound for AdaMSS, showing that the expected test error decreases with the Gaussian complexity of the learned function class. Since AdaMSS induces a function class with lower Gaussian complexity, it is theoretically expected to generalize better than methods with higher complexity.

This theoretical advantage is further validated by our empirical results, which consistently demonstrate that AdaMSS achieves lower average test errors across a range of tasks.

---

R4. Discussion in detail how comparing with other LoRA-based subspace methods

We thank the reviewer for the suggestion to clarify how AdaMSS compares to other LoRA variants that utilize subspaces. Following your suggestion, we will add a detailed discussion comparing our method with other LoRA-based approaches in the revised version. Besides, we will include a new Discussion Section and introduce additional experiments to better illustrate the advantages of AdaMSS.

Below is a summary of the discussion for your reference:

1. Key Differences Between AdaMSS and Other Subspace-Based LoRA Variants

While many existing PEFT methods—such as PiSSA, AdaLoRA and LoRA-GA—rely on the assumption of a single low-rank subspace, AdaMSS departs from this by modeling the pretrained weight space as composed of multiple distinct subspaces.

AdaMSS clusters the column vectors of the weight matrix into separate subgroups, each forming an independent subspace with its own low-rank representation.
Unlike AdaLoRA, which dynamically adjusts the rank of a single subspace, AdaMSS performs explicit subspace segmentation and dynamically estimates the importance of each subspace, enabling more compact expressiveness and finer adaptation.

2. Advantages of AdaMSS over other Methods

• (1) Reduced Training Time and Memory Usage:
  Thanks to its lower number of trainable parameters and the proposed adaptive budget allocation, AdaMSS achieves a better performance in both memory efficiency and training speed.
  Moreover, our budget allocation scheme offers greater flexibility in balancing accuracy and training efficiency.

• (2) Better Accuracy Across Multiple Tasks:
  AdaMSS consistently outperforms other methods—including PiSSA—on several benchmarks, only using fewer or comparable trainable parameters.
  In the following table, we summarize tasks where AdaMSS achieves at least 3% accuracy improvement over competing methods. Bold entries indicate cases where AdaMSS reduces the number of trainable parameters by more than 75% yet still achieves superior results.

In the revised version, we have extended our experiments on language models (RoBERTa) to include LoRA $(r=1)$ and PiSSA $(r=1)$ under minimal parameter settings, to better highlight and explain the advantages of our method. Please refer to Tables 1 and 2.

• (3) Theoretical Guarantee:
  Our method provides a stronger theoretical foundation. Theorem 1 formally derives an upper bound on the expected test loss of AdaMSS, offering a generalization guarantee that is not available in most other LoRA-based approaches.

Table 1. Results of LoRA $(r=1)$, PiSSA $(r=1)$, and AdaMSS$_{base}$ on RoBERTa-Base

Table 2. Results of LoRA $(r=1)$, PiSSA $(r=1)$, and AdaMSS$_{base}$ on RoBERTa-Large

---

R5. Cover as many base models and parameter scales as possible

Thank you so much for the comment. Our current experiments cover a diverse set of models, including ViT (Base/Large), RoBERTa (Base/Large), LLaMA 2-7B, Mistral-7B, and Gemma-7B, with parameter sizes ranging from 85.8M to 8.5B.
In most cases, our method achieves the best performance across these models.

Due to the limited rebuttal timeline, we were unable to include results on larger-scale models in this version. However, we are actively conducting these experiments and will incorporate a more comprehensive comparison with larger-scale models in the revised submission.

---

We thank the reviewer again for the thoughtful questions. Please let us know if any aspects remain unclear—we would be truly grateful for the opportunity to further clarify.

We sincerely appreciate the reviewer’s recognition of our work and the constructive feedback and insightful questions provided. Below, we address the concerns and questions raised.

---

R1. Sufficiency of empirical evidence for subspace structure Thank you so much for your constructive suggestions. In addition to Figure 2, we will include more visualizations of empirical evidence for multi-subspace structures across layers and tasks in the revised manuscript. Owing to space constraints, we provide partial numerical evidence for multi-subspace structures across different layers.

In addition to the approximate block-diagonal patterns shown in Figure 2, another way to observe the distribution of weight column vectors is through the singular values of the Laplacian matrix constructed from the principal components. In Table 1, we show the singular value distribution of the Laplacian matrix computed from the principal components of the query weight matrices at each layer of a pretrained ViT-Large model. From Table 1, we make the following observation: for most layers, the first 900 singular values remain significantly large, while the trailing values are very close to zero (e.g., below 0.01).

The number of near-zero singular values of the Laplacian matrix can be interpreted as an estimate of the number of disjoint subspaces within the weight space. This provides an additional numerical evidence of the multi-subspace structures within the principal components of pretrained weights.

Table 1: Singular value distribution of the Laplacian matrices constructed from the principal components of the query weight matrices at each layer of the pretrained ViT-Large model.

---

R2. Lack of comparative baselines for adaptive subspace selection

Table 2: Comparing the performance of different adaptive allocation scheme on ViT-Large model.

Thank you so much for your constructive suggestions. We agree that including comparisons against alternative strategies will help clarify the advantages of our adaptive allocation scheme.

In the revised manuscript, we include comparisons with the following two adaptive allocation strategies:

• (1) Random adaptive selection, as suggested by the reviewer.
• (2) $\ell_1$-norm-based adaptive budget allocation, where the budget is assigned based on subspaces with the highest $\ell_1$-norm, although we currently do not directly implement allocation based on principal singular directions within each cluster as suggested.

As shown in Table 2, $\ell_1$-norm-based adaptive budget allocation and random adaptive selection lead to significant performance degradation, and importance-score-based adaptive budget allocation has achieved best performance among all compared methods.

---

R3. Sensitivity to hyperparameters and preprocessing

Thank you so much for your constructive suggestions. In the current version of the manuscript, we have already compared strategies with and without subspace number estimation, as well as the effect of using different fixed values of $K$.

Following your suggestion, we have reorganized our existing ablation studies and added new ablations from the following two aspects:

1. Hyperparameters for subspace segmentation, including the thresholds $\tau$ and $K_0$ used for estimating the number of subspaces $K$;
2. Hyperparameters for adaptive budget allocation, including the target number of subspaces $K_{\text{target}}$ and the decay exponent $\rho$ used in the proposed multi-subspace adaptive budget allocation scheme.

Below we summarize the findings of these new ablation studies:

• (1) Thresholds $\tau$ and $K_0$ for subspace estimation: The threshold $\tau$ is used to detect near-zero singular values in the normalized Laplacian matrix and is set as a small value. In our experiments, we set $\tau = 0.01$. The parameter $K_0$ serves as a lower bound on the estimated number of subspaces. In Tables 3–5, we evaluate various values $\tau$ ∈ {0.001, 0.01, 0.05, 0.10, 0.15, 0.20} and $K_0$ ∈ {1, 5, 10, 15, 20} using ViT-Large with AdaMSS$_{base}$ (without adaptive budget allocation).

  The results show that when $K_0 \geq 10$, AdaMSS$_{base}$ exhibits robust performance across all three datasets for different $\tau$ and $K_0$.

• (2) Target number of subspaces $K_{\text{target}}$ and decay exponent $\rho$: In the adaptive budget allocation scheme, the number of trainable subspaces is gradually reduced to $K_{\text{target}}$ according to a smooth cubic decay schedule (default $\rho = 3$). A larger $\rho$ indicates a faster decay in the number of active subspaces. In Tables 6–8, we evaluate the sensitivity to $\rho$ ∈ {1, 2, 3, 4, 5} and $K_{target}$ ∈ {100, 200, 300, 400} using ViT-Large with AdaMSS. The results indicate that AdaMSS is robust to across a range of $\rho$ and $K_{\text{target}}$.

Table 3: Results of AdaMSS$_{base}$ under different hyperparameter settings for StanfordCars.

Table 4: Results of AdaMSS$_{base}$ under different hyperparameter settings for CIFAR100.

Table 5: Results of AdaMSS$_{base}$ under different hyperparameter settings for FGVC.

Table 6: Results of AdaMSS under different hyperparameter settings for StanfordCars.

Table 7: Results of AdaMSS under different hyperparameter settings for CIFAR100.

Table 8: Results of AdaMSS under different hyperparameter settings for FGVC.

---

R4. Writing clarity and visual explanation

Thank you for highlighting this issue. We will revise Sections 3.1 and 3.2 by simplifying the notation and derivations, and by adding visual illustrations for the proposed methods.
We hope these revisions will enhance the clarity and readability of our work.

---

We thank the reviewer again for the thoughtful questions. Please let us know if any aspects remain unclear—we would be truly grateful for the opportunity to further clarify.

We sincerely appreciate the reviewer’s recognition of our work and the constructive feedback and insightful questions provided. Below, we address the concerns and questions raised.

---

R1. Performance metrics such as GPU memory or training speed

Thank you very much for pointing this out. In the revised version, we have added a comparison of different PEFT methods in terms of both memory usage and training time. Since our method use a smaller number of trainable parameters, it demonstrates superior memory efficiency, as shown in Tables 1 and 2.

Regarding training speed, as discussed in the paper, the computational complexity of our gradient updates is of the same order as that of LoRA and PiSSA when $r = \sum_{k=1}^{K} r_k$, where $r$ denotes the hyperparameter used in LoRA and PiSSA. However, in practice, the multi-subspace-based adaptive budget allocation allows for faster training by selectively updating only the most important subspaces (see Tables 3 and 4).

Besides, the proposed multi-subspace-based adaptive budget allocation provides greater flexibility in balancing training time and performance. In our paper, the number of trainable subspaces is gradually reduced to a target value $K_{target}$ following a smooth decay schedule with decay exponent $\rho=3$. A larger $\rho$ leads to a faster reduction in the number of trainable subspaces. Table 5 presents an ablation study demonstrating how different choices of $\rho$ affect training speed and final performance.

Table 1: Optimizer memory consumption (MB) of different PEFT methods on ViT-Large using fp32 precision.

Table 2: Optimizer memory consumption (MB) of different PEFT methods on Roberta-Large using fp32 precision.

Table 3: Average training time (in seconds) of different PEFT methods on ViT-Large.

Table 4: Average training time (in seconds) of different PEFT methods on Roberta-Large for natural language understanding task (STS-B).

Table 5: Training time (in seconds) and performance (PCC) of AdaMSS with varying $\rho$ on Roberta-Large for natural language understanding task (STS-B).

---

R2. Experimental study on newer models (LLaMA3, DeepSeek, Gemini)

Thank you so much for the valuable comments. Our current experiments cover a range of models, including ViT (Base/Large), RoBERTa (Base/Large), LLaMA 2-7B, Mistral-7B, and Gemma-7B. Among them, LLaMA 2-7B and Gemma-7B were released in 2023, while Mistral-7B was released in 2024.

Due to the time constraints of the rebuttal, we were unable to include results on LLaMA 3 and DeepSeek in this version. However, we are actively working on these experiments and will include a more comprehensive comparison with newer models in the revised version.

---

R3. Scalability to long-context or multi-step reasoning tasks

Thank you for the question. Our method is scalable to long-context and multi-step reasoning tasks. This is supported by our experimental results on GSM8K and MATH — two widely recognized benchmarks for multi-step symbolic and numerical reasoning. Compared to other methods, our approach achieves higher accuracy while requiring fewer trainable parameters, demonstrating both its efficiency and effectiveness in handling complex reasoning scenarios. The detailed results are provided below for reference.

Table 6: Perfromance of diferent PEFT methods on LLaMA 2-7B.

Table 7: Perfromance of diferent PEFT methods on Mistral-7B.

Table 8: Perfromance of diferent PEFT methods on Gemma-7B.

---

R4. Compatibility with positional tuning (e.g., RoCoFT)

We thank the reviewer for this insightful question. Indeed, our framework could potentially be integrated with positional tuning methods such as RoCoFT.

One preliminary yet intuitive idea is to apply positional tuning directly to the low-rank representation $\mathbf{Z}$, especially given that $\mathbf{Z}$ often exhibits an approximately block-diagonal structure. Another promising direction, inspired by the AdaMSS, is to leverage subspace segmentation to guide RoCoFT in deciding which columns should be updated and which can be frozen, potentially enabling adaptive budget allocation during training.

We will include this discussion in the Future Work section of the revised version.

---

We thank the reviewer again for the thoughtful questions. Please let us know if any aspects remain unclear—we would be truly grateful for the opportunity to further clarify.

Dear Reviewer J6fY,

Thank you again for your valuable feedback on our submission.

We have submitted a detailed rebuttal addressing your comments, including:

• a comparison of training time and memory usage across different methods (Tables 1–5);
• an update on newer models, including coverage of Mistral-7B (2024) and Gemma-7B (2024) (Tables 7–8), with additional comparisons (e.g., LLaMA 3 (2024) and DeepSeek) currently in progress for inclusion in the revised version;
• a discussion on scalability to long-context and multi-step reasoning tasks (Tables 6–8);
• a discussion on compatibility with positional tuning approaches.

Your recognition would mean a lot to us.

Best regards,
The Authors

Hi Reviewer J6fY,

The authors have provided a detailed response; do they sufficiently address your concerns on baseline comparison and computational efficiency?

Best, AC