Thank you for your insightful and constructive review. To answer your questions thoroughly, we have conducted numerous new experiments. We hope these efforts will address your concerns.

Q1: Need to provide error bars in the results

A1: The original paper includes extensive experiments cover 12 models, 13 tasks, 1k to 9k training steps, and 1 to 128 ranks. It compares PiSSA, QPiSSA with LoRA, QLoRA, LoftQ and Full fine-tuning. Running all these experiments multiple times is very time and resource-intensive. Despite this, PiSSA consistently demonstrates significant convergence speed and performance advantages, suggesting the improvement is statistically significant. The improvements of PiSSA over LoRA are notably highlighted in Table 1 and Table 2 of General Response. The error range is very small compared to the improvement.

Q2: Comparison to Gaussian or Kaiming initialization

A2: Table 1 and Table 2 show that PiSSA outperforms both Gaussian and Kaiming initialization schemes, as they still use a zero-random initialization for the adaptor without touching the frozen $W$, while PiSSA for the first time moves the principal components out of $W$ to initialize the adaptor for directly fine-tuning the core model.

Q3: Comparison to AdaLoRA or DoRA

A3: Please refer to the response to Q1 of Reviewer x9Pi, and Q2 of Reviewer Bb8M. PiSSA outperforms both methods.

Q4: Combining AdaLoRA or DoRA with PiSSA

A4: Since PiSSA's structure is identical to LoRA's, it can be combined with other advanced LoRA variants. The specific new challenges introduced by combining PiSSA with different methods are, however, beyond the scope of this work. Nevertheless, we have evaluated the combination of PiSSA with AdaLoRA. Details can be found in Q2 of Reviewer Bb8M. PiSSA+AdaLoRA shows superior performance than other methods in Table 1 including PiSSA.

Due to time and resource constraints, we could not finish experiments combining DoRA and PiSSA. We are actively conducting these experiments, and will post the results in follow-up comment.

Q5: Comparing QPiSSA to GPTQ or int8

A5: QPiSSA is not a quantization technique, but applies NF4 quantization to PiSSA’s frozen weights; it can also be combined with GPTQ or int8. We believe PiSSA reduces quantization error due to the following reasons: 1. Reducing outlier values; 2. Making the value distribution closer to a Gaussian distribution; 3. Preserving larger values in full precision, which narrows the weight distribution of the quantized part. Int8 also focuses on the first point, and PiSSA could further improve it. The second point is friendly to NF4. The third point is also crucial, as PiSSA uses adaptor to preserve a great portion of weights in full precision. When compared to int8, QPiSSA+int8 reduces quantization error by 23.58% on LLaMA-3-8B. In Table 1, rows 4 and 8 compare QLoRA and QPiSSA using int8, showing that QPiSSA still significantly outperforms QLoRA. We will post the results of combining PiSSA and GPTQ in the follow-up comment.

Q6: LoRA gradients point in random directions

A6: Please refer to the response to Q1 of Reviewer XzvL.

Q7: S-LoRA uses principal components initialization

A7: Although the term "principal components" appears in the third section of the S-LoRA paper, it is not a method focusing on adapter initialization or PEFT. S-LoRA is actually a system designed for the scalable serving of up to 2,000 LoRA adapters. In PEFT, terms like low-rank decomposition and SVD often appear. They mostly refer to using the product of low-dimensional matrices to approximate an ideal $\Delta W$ through learning. These methods do not actually perform decomposition on any matrix. To our knowledge, PiSSA is the first study that directly performs SVD on the original model and extracts the principal singular values and vectors for fine-tuning.

Q8: Why is alpha set to r?

A8: Since PiSSA extracts the model's principal singular values and singular vectors for fine-tuning, the scaling factor (=alpha/r) in the adapter needs to be set to 1 to ensure adaptor + frozen weights equals the pre-trained weights initially. Nevertheless, it is also possible to set alpha to other values as in LoRA. In NLU tasks, we have experimented with changing this hyper-parameter to other values.

Q9: Can LoRA ultimately converges to a similar optimum after longer training?

A9: In Sections 5.2 and 5.3 of the original paper, we conducted experiments with larger datasets and longer training steps. And the results show that PiSSA/QPiSSA indeed achieve lower loss than LoRA/QLoRA, indicating convergence to a better optimum.

Q10: Meaning of ‘iter’ in Figure 2b

A10: The meaning of 'iter' is explained in Appendix E: simply put, it means iteratively executing PiSSA initialization multiple times, compressing the quantization error into the adapter each time to further reduce the quantization errors.

Q11: Convert to LoRA need 2r ranks

A11: There are three ways of using PiSSA. The first way is to save the adapter with rank $r$ and replace the original model with the residual model. However, sometimes people want to keep the original model unchanged. Thus the second way is to only save the adaptor with rank r without saving the residual model, and recover the residual model when needed by performing SVD again. Since fast singular value decomposition can complete PiSSA initialization within a few seconds to tens of seconds, the second way is also feasible in practice. The third way, which is what is discussed in Appendix C, saves a rank-2r adaptor to avoid SVD (trading-off space for time) and converts a PiSSA adaptor equivalently to a LoRA adaptor during inference. Therefore, we believe that the ability of PiSSA to equivalently convert to 2 times the rank of LoRA is not a limitation but an advantage, providing more flexibility for using PiSSA.

Dear Reviewer M1T9,

In your questions Q4 and Q5, you suggested integrating PiSSA with both AdaLoRA and DoRA, as well as combining PiSSA with llm.int8 and GPTQ. During the rebuttal period, due to time constraints, we were only able to combine PiSSA with AdaLoRA and llm.int8, and the experimental results have shown the advantages of PiSSA when integrated. Over the past few days, we have completed the experiments combining PiSSA with DoRA and GPTQ, and the results are presented in the table below:

Q4: Combining AdaLoRA or DoRA with PiSSA

A4 part 2: From lines 4, 7, and 8 of the table, it is evident that the performance of PiSSA combined with DoRA significantly surpasses that of DoRA alone and also exceeds the performance of PiSSA alone. Taking into account the combination experiments of PiSSA with AdaLoRA, it can be inferred that PiSSA benefits from the enhancement techniques of LoRA, demonstrating the potential of PiSSA when integrated with other methods.

Q5: Comparing QPiSSA to GPTQ or int8

A5 part 2: GPTQ handles each row $w$ independently, quantizing one weight at a time while updating all not-yet-quantized weights. Therefore, the method of using nuclear norm to calculate quantization error, as used in our paper, is not applicable for GPTQ. Thus, we turn to use Perplexity on WikiText-2 to calculate quantization error, where a lower Perplexity indicates reduced quantization error. The Perplexity of LLaMA-3-8B used in our experiments is 5.14. After quantization with GPTQ-4bit, the Perplexity increased to 20.79. However, using PiSSA, we were able to reduce the Perplexity to 6.23. These results validate the effectiveness of PiSSA in reducing quantization error as discussed in our paper.

From lines 11-12 of the table, it is evident that the performance of QPiSSA combined with GPTQ-4bit significantly surpasses that of QLoRA combined with GPTQ-4bit. This confirms the advantages mentioned in our paper, where QPiSSA retains the rapid convergence and superior performance characteristics of PiSSA.

We believe we have addressed all of your questions comprehensively. If you have any additional queries or require further clarification, please do not hesitate to contact us. We are committed to providing any necessary information. If you find the contributions of our paper and the supplementary experiments we have provided to be valuable, might we kindly ask you to consider adjusting your score accordingly?

Thank you for recognizing the originality, quality, and significance of our article. We also appreciate your suggestions for improving the writing. As we cannot modify the original text during the rebuttal period, we will incorporate your recommendations in the camera-ready version.

Q1: In the related work section, AdaLoRA [42, 41, 43] is mentioned, but only one of these works is AdaLoRA.

A1: Thank you for pointing this out. Due to the extensive content of PiSSA, we had to condense the text multiple times, leading to this issue. All these three papers dynamically adjust the rank of LoRA at each layer, so we used AdaLoRA as a collective term. We found similar issues in the subsequent DeltaLoRA [44, 45] citations. We will complete the citation information.

Q2: There is overlap between AdaLoRA and PiSSA in their reliance on the SVD. Discuss exactly how AdaLoRA differs from PiSSA.

A2: AdaLoRA introduces three improvements over LoRA:

1. Trainable parameters in AdaLoRA are changed to $A, B$, and $E$. $A$ and $B$ are Gaussian-initialized, and $E$ is a zero-initialized $r$-dimensional vector, making $A diag(E) B = \Delta W$, similar to singular value decomposition.
2. A regularization loss $\|AA^T-I\|+\|B^TB-I\|$ is used to make $A$ and $B$ gradually orthogonal during training, resembling the SVD of $\Delta W$.
3. An initial large rank is set, and less important E values are gradually masked during training, resulting in different final ranks for each layer, achieving better performance with the same number of parameters.

Despite the extensive use of SVD terms, AdaLoRA does not perform actual SVD on any matrix. In the PEFT domain, terms like low-rank decomposition, and singular value decomposition often appear. They generally refer to products of low-dimensional matrices approximating an ideal $\Delta W$ without actual matrix decomposition. To our knowledge, PiSSA is the first to perform SVD on the original model, fine-tuning the principal component while keeping the residual model frozen. During the rebuttal, we have evaluated PiSSA against AdaLoRA on GSM8K and GLUE tasks. The results in Table 1 and Table 2 of General Response demonstrate that PiSSA outperforms AdaLoRA.

Additionally, PiSSA and AdaLoRA represent different improvements to LoRA, making them combinable. Therefore, we additionally improved PiSSA based on AdaLoRA's three innovations:

1. After extracting the principal singular values and vectors of $W$, we use $S$ as an independent trainable vector instead of multiplying it into $U$ and $V$.
2. Since PiSSA's $U$ and $V$ are orthogonal at the beginning, maintaining their orthogonality through orthogonal regularization is very easy.
3. Although AdaLoRA claims to dynamically reduce the number of trainable parameters, the initially large number of parameters is not truly pruned, resulting in more parameters being updated during actual training. Therefore, we did not use this improvement.

PiSSA, with its intrinsic principal singular values and orthogonal singular vectors, is very suitable for combination with AdaLoRA. According to Table 1 of General Response, The performance of the improved PiSSA surpasses all the other methods including PiSSA.

Q3: Are the GSM8K evaluations 8-shot?

A3: This article uses the zero-shot evaluation prompt for GSM8K and MATH following MetaMath [1], the zero-shot evaluation prompt for humanEval and MBPP following WizardCoder [2], and the zero-shot single answer grading prompt for MT-Bench [3]. We will provide detailed settings and corresponding code in the supplementary materials.

[1] Yu, Longhui, et al. "MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models." ICLR24.

[2] Luo, Ziyang, et al. "WizardCoder: Empowering Code Large Language Models with Evol-Instruct." ICLR24.

[3] Zheng, Lianmin, et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." NeurIPS 2024.

Q4: A short, succinct list enumerating the similarities and differences early on may help readers understand.

A4: Thank you for the suggestion. PiSSA has the same structure as LoRA, making it easy for users to switch from LoRA to PiSSA for fine-tuning. However, PiSSA's different initialization leads to distinct optimization directions, faster convergence, better training results, and lower quantization errors. We will list the similarities and differences between PiSSA and LoRA in the introduction section of the camera-ready version for easier understanding.

Q5: Only looks at a single layer of a single model is hard to accept this as evidence for $W_{res}$ having a narrower distribution than that of W.

A5: To intuitively compare the distribution differences between quantized original and residual models, we took LLaMA 2-7B's first Query layer as an example to illustrate the distribution of $W$ and $W_{res}$. As you suggested, using only one layer of one model is not statistically significant. In Table 5 of General Response, we applied PiSSA initialization to LLaMA-2-7B, Mistral-7B, Gemma-7B, and LLaMA-3-8B, and fit the values in every linear layer with Gaussian distribution and calculated their mu and sigma in the table. The results show that the residual models' means are closer to 0, and the standard deviations are smaller after PiSSA initialization. Thus, $W^{res}$ indeed has a narrower distribution than W in a statistical sense. Nevertheless, the difference is not as large as that in the first layer after averaging all layers, which we suspect is because middle layers in a model tend to have more even eigenvalue distributions due to redundancy and insufficient training. We will include this statistical result in the supplementary materials and reflect it in the main text.

Q6: Minor comment: "loftQ" in Table 3 <- "LoftQ".

A6: Thank you once again for your careful reading and helpful suggestion. Your feedback is greatly appreciated, and we will incorporate all your suggestions in the camera-ready version.

Thank you for your recognition of PiSSA as a simple yet effective method, and for your appreciation of the article's quantitative and qualitative analysis. Here are responses to the concerns raised:

Q1: Provide more insight, for example, comparing the subspace of the gradient of conventional LoRA?

A1: PiSSA potentially provides many insights for the PEFT area:

1. To reduce fine-tuning parameters, LoRA and many of its variants add a low-rank adaptor $\Delta W = AB$ to $W$, and initialize $\Delta W$ with zero to ensure $W + AB$ initially does not change the model. However, PiSSA for the first time touches the frozen $W$ by decomposing $W$ into principal and residual components with SVD, which are used to initialize $\Delta W$ and $W^{res}$ respectively. This idea has already inspired many follow-up works studying the optimal distribution plans for $\Delta W$ and $W^{res}$, which contributes greatly to the community.
2. By putting the principal components of $W$ into the adaptor $AB$, PiSSA can directly fine-tune the core functions of the model instead of fine-tuning the random/zero-initialized $AB$ as in LoRA. This leads to faster convergence and higher performance. From the gradient direction perspective, we have shown in lines 50-55 and lines 123-131 of our paper that the initial gradient for LoRA is zero/random, thus could waste much time around the initial point, while PiSSA optimizes along the gradient direction of the model's principal components, thereby approximating the effect of full parameter fine-tuning. The toy experiment in Figure 2a using the MNIST dataset and an MLP with 2 linear layers visually demonstrates that PiSSA and full fine-tuning converge much faster compared to LoRA. In the supplementary materials, the loss_landscape.gif shows this process dynamically. In LLM experiments, the advantage of PiSSA over LoRA and the similarities between PiSSA and full fine-tuning can also be observed from the loss and grad norm during the training process in Figure 4a and 4b.
3. Compared to LoRA + quantization, PiSSA + quantization preserves the principal singular values and singular vectors in full precision while quantizing the residual part $W^{res}$ instead of the full $W$. Since the large singular values have been removed from $W^{res}$, its value distribution is much narrower than $W$, which makes quantization easier (the larger variance of the value distribution, the larger error quantization will introduce). This can be seen in Figure 3 and 5, where QPiSSA greatly reduces the quantization error and shows better performance than QLoRA.

Regarding your question about comparing the subspace of gradients of PiSSA and LoRA, we further validated our analysis through two additional experiments.

1. We trained LLaMA-3-8B on the MetaMath dataset 5 times, each time initializing LoRA with different random seeds while using the same batch of 128 training examples to calculate LoRA's gradients. After dimensionality reduction to two, we list the results in Table 3 of General Response. We can observe that the gradient of matrix $A$ is consistently zero, and the gradient direction of $B$ varies with each initialization. This occurs because the gradient of matrix $A$ depends on matrix $B$, which is initialized to zero in LoRA, resulting in a zero gradient for matrix $A$. Conversely, the gradient of matrix $B$ is influenced by matrix $A$, which is initialized from Gaussian, hence the varying gradient directions of matrix $B$ in each experiment. In contrast, the gradient direction in PiSSA remains consistent across all five training seeds and only depends on the original model and the training data. This experiment underscores the stability of PiSSA's optimization direction relative to LoRA.
2. Furthermore, we quantitatively compared the effects of updating using the principal singular value direction v.s. “random” direction in the early stages of fine-tuning. We trained LLaMA-3-8B on the MetaMathQA dataset using PiSSA and LoRA, saving the parameters and gradients for the first 50 iterations. At the 50th step, the losses for LoRA and PiSSA dropped to 0.3677 and 0.2899, respectively. We took the parameters at the 50th step as the target point. Using the direction and distance from the initial parameters to the final step's parameters as a reference, we compared the first 5 steps for LoRA and PiSSA. For each step, we determined how much was moved in that direction, then divided by the target distance to obtain a ratio. The ratios are recorded in Table 4 of General Response. As shown in the first 2 rows of the table, after just 5 updates, PiSSA's loss reduced from 0.8884 to 0.3346, whereas LoRA only reduced to 0.5538, reflecting the advantage of the principal singular value direction over the zero-random direction in convergence speed. In rows 3-4 of the table, matrix $A$ in LoRA had a gradient of 0 at the first step and thus did not update. In the following 4 steps, it only moved 15.94% towards the target direction. In rows 5-6 of the table, matrix $B$ in LoRA always moved less towards the endpoint in the target direction compared to PiSSA.

Q2: Besides the first r eigenvectors. How about other dimensions in the subspace?

A2: Our PiSSA puts the first $r$ singular values and vectors in the adaptor $A,B$ and the remaining ones in the frozen $W^{res}$. In Figure 8 of the Appendix, we compared the training outcomes of using the principal, middle, and minor $r$ singular values and singular vectors to initialize $A,B$. The experimental results evident that training with the principal singular values results in lower loss and higher accuracy.

Dear Reviewer XzvL,

The discussion period has passed the halfway point, and we have addressed the questions you previously raised. Do you have any additional questions or concerns that you would like to discuss?

Our paper investigates the gradient behavior of LoRA, revealing that $A$ and $B$ initially have zero gradients and random gradient directions. This leads to slow convergence and potentially suboptimal local minima. To address these issues, we propose the PiSSA initialization method. PiSSA approximates the optimization direction of full fine-tuning by fine-tuning the principal components of a model. To our knowledge, PiSSA is the first method to use SVD on the original model, employing the principal singular values and vectors to initialize the adapter and fine-tune it, while fixing the remaining parts of the model during training. Our experiments demonstrate that PiSSA not only converges faster but also achieves better final results compared to LoRA. The initialization process is efficient, taking only seconds to minutes due to the use of fast singular value decomposition.

We extend PiSSA by integrating it with NF4, llm.int8, and GPTQ quantization to create QPiSSA. This approach significantly reduces quantization error compared to QLoRA while retaining the fast convergence and strong performance of PiSSA.

PiSSA modifies only the initialization method used by LoRA, making it compatible with various LoRA enhancements. We combined PiSSA with LoftQ, AdaLoRA, and DoRA, and our results show that PiSSA+ outperforms these methods and PiSSA, demonstrating its potential for further improvements.

Our extensive experiments include 5 NLG and 8 NLU tasks using 12 different models ranging from 184M to 70B parameters. We compared performance across 1k-9k+ training steps and ranks from 1 to 128, evaluating against methods such as LoRA, QLoRA, LoftQ, DoRA, AdaLoRA, and full-parameter fine-tuning. We also examined the effects of initializing with principal, medium, and minor singular values and vectors.

If you find the content in the main paper and our rebuttal satisfactory, would you consider revising your score?

Thank you for acknowledging the originality of PiSSA, the quality of our experiments, and our contributions to the community. Here are the answers to your queries:

Q1: Comparison with advanced LoRA adapter methods, e.g., DoRA

A1: In our paper, we demonstrated PiSSA's effectiveness through large amount of experiments on 5 NLG and 8 NLU tasks, utilizing models ranging from 184M to 70B parameters. Our comparisons included 1k-9k+ training steps and ranging from 1 to 128 ranks, against methods such as LoRA, QLoRA, LoftQ, and full-parameter fine-tuning. PiSSA modifies only the initialization method used by LoRA, making it orthogonal to some LoRA enhancements. Initially, due to time and resource limitations, we did not compare PiSSA with DoRA. However, upon your request, we have now included these comparisons.

DoRA adds a learnable magnitude module to LoRA, normalizing $W + AB$ at each update step and multiplying its by the magnitude module. This allows $A, B$ to learn the direction and the magnitude module to learn the magnitude of $\Delta W$. While this approach can improve fine-tuning performance, normalizing $W + AB$ at each step results in slower fine-tuning speeds. In contrast, PiSSA only changes LoRA's initialization method, matching LoRA in training speed and converging faster, thereby reducing training costs.

We compared PiSSA with DoRA on LLaMA-3-8B for GSM8K and DeBERTa-v3-base for GLUE tasks, repeating each experiment three times and recording the results in Table 1 and Table 2.

From these tables, PiSSA significantly outperforms DoRA, as DoRA still uses zero and random initialization.

Q2: Open source schedule

A2: PiSSA has greatly benefited from the open-source community, and we are excited to contribute back.

PiSSA only modifies the initialization method of LoRA, so it incurs no additional training costs, requires no extra environmental setup, and involves no changes to existing code. This allows for straightforward training and inference, just like LoRA, given the initialized parameters. We plan to release all PiSSA-initialized models from our paper, along with the initialization of future mainstream models, to facilitate easy adoption.

For users interested in customizing PiSSA configuration or modifying its core code, we have already integrated PiSSA into HuggingFace’s PEFT library and provide the training scripts used in our paper to replicate the results. Additionally, we will release training logs and checkpoints from our experiments to aid in further comparisons and validations.