Combating the Generalization-Forgetting Trade-off in Continual Learning: A Cautious Passive Low-Rank Approach

Thank you for your appreciation and excellent summary of our work. We also appreciate the time and effort you dedicated to reviewing our research. We have addressed your questions and concerns below:

W1: About mathematical presentation

Thanks for your suggestion. We would clarify the mathematical presentation better and would improve it.

W2: About non-low-rank continual learning

Thanks for your suggestion. We agree that including a few recent low-rank continual learning methods as baselines can provide a broader performance perspective. Since we focus on parameter-efficient continual learning in our work and consider low-rank methods severing as a core aspect, we didn't choose non-parameter-efficient continual learning methods as baselines. We plan to explore non-low-rank methods in the following to make our work have a more insightful performance analysis.

Q1: About connection between Eq.7 and Eq.5

Thanks for your comment on this connection. We would like to explain: Eq.7 is defined as $\phi(U_{i}^{l},U_{j}^{l}) = \frac{\sum_{s=1}^{p}\sigma_{s}^2}{p} = \frac{1}{p}(1-d(U_{i}^{l},U_{j}^{l})^2)$, where $\sigma_{s}$ are the singular values of $(U_{i}^{l})^\top U_{j}^{l}$, and $U_{i}^{l}$ and $U_{j}^{l}$ are the left singular matrices of $(B_i^{l}A_i^{l})^\top B_i^{l}A_i^{l}$ and $B_j^{l}A_j^{l}$, respectively. The singular values of $B_j^{l}A_j^{l}$ depend on the alignment of left and right singular vectors. And $(U_{i}^{l})^\top U_{j}^{l}$ captures the cosine of the principal angles between subspaces spanned by $U_{i}^{l}$ and $U_{j}^{l}$. Thus, the singular values of $(U_{i}^{l})^\top U_{j}^{l}$ are proportional to singular values of $(B_i^{l}A_i^{l})^\top(B_j^{l}A_j^{l})$. So we have $\sum_{s=1}^{p}\sigma_{s}^2\propto\text{tr}((U_{i}^{l})^\top U_{j}^{l})\propto\text{tr}((B_i^{l}A_i^{l})^\top(B_j^{l}A_j^{l}))$. It indicates that singular values of $(B_i^{l}A_i^{l})(B_j^{l}A_j^{l})^\top$ in Eq.5 are connected to singular values of $(U_{i}^{l})^\top U_{j}^{l}$ in Eq.7. We will make detailed explanation in the revised manuscript.

Q2: About typo in Eq.(2) and presentation of Eq.(2)(3)

Thanks for your correctness. Yes, it should be $\Theta_{N}$ instead of $\Theta_{T}$. We present the modified version of Eq.(2) and Eq.(3): $F(\theta_{1},\dots,\theta_{N}) = \sum_{t=1}^{N-1}(L_{t}(\theta_{N})-L_{t}(\theta_{t}))$ and $I(\theta_{1},\dots,\theta_{N}) = \sum_{t=1}^{N}(L_{t}(\theta_{t})-L_{t}(\theta_{t}^{*}))$

Q3: About typo in Eq.(7)

Thanks for pointing out typos. Yes, there are typos in Eq.(7), and sorry for causing confusion. It should be $\phi(U_{i}^{l}, U_{j}^{l}) = \frac{\sum_{s=1}^{p}\sigma_{s}^2}{p} = \frac{1}{p}(1-d(U_{i}^{l},U_{j}^{l})^2)$

Q4: About rank updating interval $k$

Thanks for your question. The rank updating interval $k$ is used to control the frequency of subspace similarity evaluation during training, which means when to invoke Algorithm 2 for rank update. We increase the rank (randomly initialize newly added matrix elements) in Algorithm 2 and these new elements require training for a while before the next subspace similarity evaluation. The interval $k$ is the number of iterations between each subspace similarity evaluation. We also evaluate the impact of $k$ in Figure 4 on Page 10.

Q5: About superscripts in Section 2

Thank you for your suggestion. We agree that the superscripts in Section 2 should be clearer. In Algorithm 2, $A_{i}^{t}$ represents task $T_{i}$ and the current iteration number $t$, which is different from $A_{i}^{l}$ we use to denote the $l$-th layer. To avoid further confusion, we chose not to use $A_{i}^{l,t}$, as it could make the notation overly complex. Instead, as noted in line 201 "For simplicity, we use $B$ and $A$ to represent the layer-wise weight matrices $B^{l}$ and $A^{l}$."

Q6: About the difference between InfLoRA and CP-Rank

Thanks for highlighting this related work. We appreciate the opportunity to compare our method with it.

• Model: InfLoRA proposes a LoRA-based continual learning method for a pre-trained Vision Transformer (ViT), while our approach, CP-Rank, is for general large language models and not a specific LLM.

• Method: 1. InfLoRA uses fixed and same rank for each LoRA without considering rank patterns of tasks and model different layers, while CP-Rank considers rank patterns among tasks and model layers. 2. InfLoRA not only stores previous LoRA parameters but also additionally stores gradient subspaces of previous tasks, while CP-Rank initializes new task subspaces without storing extra components, which is more memory-efficient.

• Dataset: InfLoRA uses image datasets, while CP-Rank is applied to text (natural language) datasets.

InfLoRA provides CL method to vision-language models, while our method offers a scalable and effective approach to CL in LLMs, proven in NLP tasks. We are committed to including this related work and comparison in the subsequent revision.

Thank you for your valuable feedback on our work. We truly appreciate the time you invested in the review. We have carefully considered your insights and addressed the highlighted concerns. We hope our responses provide clarity on the matters raised.

W1: About the efficiency of CP-Rank

Thanks for your question about the efficiency of CP-Rank.

• Space complexity: We reduce the memory by using low-rank matrices and the memory should be $NL(d_1\times r+d_2\times r)$, where $B\in\mathbb{R}^{d_1\times r}$ and $A\in\mathbb{R}^{r\times d_2}$, where $r\ll\text{min}(d_1,d_2)$. Compared to full fine-tuning, we only need to train LoRA parameters, significantly saving memory while achieving comparable performance. Additionally, we use layerwise ranks for LoRA in each layer of each task, meaning ranks may vary across layers within a task.

• Time complexity: Performing SVD is necessary to obtain the representation of a low-rank subspace, a common method in low-rank works [1, 2]. Besides, our method randomly initializes new components of LoRAs during training, meaning $A$ or $B$ alone cannot represent low-rank subspaces. Thus, to ensure correct representation of low-rank subspaces, we obtain the left singular matrix via SVD, which is then used to calculate subspace similarity.

We will make these complexities clear in the revised version.

W2: About the first experiment

Thank you for pointing this out. This experiment serves as an example to empirically show that there exists a trade-off between low and high ranks in balancing forgetting mitigation and generalization amplification. It does not imply a straightforward linear relationship between rank size and the degree of forgetting. The motivation for our work comes from several studies [3,4] which have shown that LoRA forgets less than more regularization techniques like weight decay and dropout and also helps maintain the diversity of generations, while other studies [5,6] indicate that low-rank update mechanism limits LLMs' ability to learn as effectively as full fine-tuning, and using higher-rank update mechanism can help LoRAs achieve better performance.

W3: About Eqn(5)

Thanks for raising this comment. Eqn(5) motivates the effectiveness of Eqn(7), which measures the similarity of low-rank subspaces. Eqn(7) is defined as $\phi(U_{i}^{l},U_{j}^{l}) = \frac{\sum_{s=1}^{p}\sigma_{s}^2}{p} = \frac{1}{p}((1-d(U_{i}^{l},U_{j}^{l})^2)$, where $\sigma_{s}$ are the singular values of $(U_{i}^{l})^\top U_{j}^{l}$, and $U_{i}^{l}$ and $U_{j}^{l}$ are the left singular matrices of $(B_i^{l}A_i^{l})^\top B_i^{l}A_i^{l}$ and $B_j^{l}A_j^{l}$, respectively. The singular values of $B_j^{l}A_j^{l}$ depend on the alignment of left and right singular vectors. And $(U_{i}^{l})^\top U_{j}^{l}$ captures the cosine of the principal angles between subspaces spanned by $U_{i}^{l}$ and $U_{j}^{l}$. Hence, the singular values of $(U_{i}^{l})^\top U_{j}^{l}$ are proportional to the singular values of $(B_i^{l}A_i^{l})^\top(B_j^{l}A_j^{l})$. Thus, this relationship is expressed as $\sum_{s=1}^{p}\sigma_{s}^2\propto\text{tr}((U_{i}^{l})^\top U_{j}^{l})\propto\text{tr}((B_i^{l}A_i^{l})^\top(B_j^{l}A_j^{l}))$, showing that the singular values of $(B_i^{l}A_i^{l})(B_j^{l}A_j^{l})^\top$ in Eqn(5) are connected to the singular values of $(U_{i}^{l})^\top U_{j}^{l}$ in Eqn(7). We will provide detailed explanations in the revised manuscript.

W4: About explanation of Eqn(2)(3)(4)(7)

Thanks for your careful reading and correctness. For Eqn(2)(3)(4), we would like to show that our work aims to minimize the forgetting error via a well-designed update in the algorithm. We connect Eqn(5), the forgetting error bound, with Eqn(4), showing the impact of the forgetting error in continual learning optimization problem. For Eqn(2), the correct notation should be $N$. For Eqn(7), the codomain of Grassmann distance is $[0,\sqrt{p}]$, as mentioned in [7]. We use a reverse metric of the standard Projection Metric of Grassmann Distance, and its codomain is $[0,1]$ when the codomain of Grassmann distance is $[0,\sqrt{p}]$, as noted in [8].

Q: About correlation between new task's rank and subspace similarity of tasks

Thanks for your interest in this relationship. We use the reversed Grassmann similarity to measure the similarity between low-rank subspaces of tasks for determining if low-rank subspaces of two tasks are orthogonal. If the subspace similarity is below the threshold, we assume subspaces are orthogonal, meaning tasks do not interfere with each other. Here, as noted in our responses to Weakness2, low-rank updates may limit LLMs ability. Thus, when tasks' subspaces are orthogonal, we consider this orthogonality to give new task a chance for higher rank updates during training to improve performance. This is the reason for connecting rank-increasing updates with subspace similarity in our work, rather than connecting rank patterns with subspace similarity.

Thank you for your valuable and helpful feedback on our work. We have carefully considered your concerns, and we hope that our responses provide clear explanations for the problems raised.

W1 and W4: About experiments on larger language models

Thanks for your suggestion. We use T5-large to show main results in our manuscript since O-LoRA uses T5-large to present. Here, to show the effectiveness of our method on larger LLMs, we present the experimental results using Llama-2-7b-chat model to show the performance of CP-Rank with orthogonal projection. We use 6 NVIDIA A100 GPUs to conduct the following experiments on three task orders in standard CL benchmark.

These results indicate that our method can also work on larger LLMs. We are committed to using Llama-2-7b-chat or larger models in the revised manuscript.

W2: About explanation and analysis of rank-increasing strategy

Thanks for pointing out this point. The method of increasing ranks during training has been studied and well-supported in existing works. In theory, greedy low-rank learning [1] characterizes the trajectory of stochastic gradient descent, which performs a rank-constrained optimization and greedily increases the rank whenever it fails to reach a global minimizer. In practice, [2] proposes the InRank algorithm which increases the rank during training to find intrinsic low rank of networks. These studies show that increasing the rank during training helps models find optimized low-rank solutions. Our work differs from the above works as we focus on continual learning on a sequence of tasks, rather than training on a single task. We adapt the rank-increasing method to help LLMs find low-rank matrices that satisfy continual learning metrics for each task.

[1] Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. "Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning." arXiv preprint arXiv:2012.09839, 2020.

[2] Jiawei Zhao, Yifei Zhang, Beidi Chen, Florian Tobias Schaefer, and Anima Anandkumar. Incremental low-rank learning. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023.

W3: About hyperparameters

Thanks for your interest in practical application. In our approach, the only hyperparameters are the updating interval $k$ and subspace similarity threshold $\epsilon$. The rank-increasing mechanism increases rank by 1 when the subspace similarity satisfies the threshold $\epsilon$, without requiring additional hyperparameters. We consider $k$ and $\epsilon$ as essential conditions that cannot be omitted in the algorithm. Threshold $\epsilon$ is inherently required in rank-increasing algorithms, such as [1], to determine when to increase the rank. Updating interval $k$ is crucial to control the frequency of subspace similarity evaluations. Calculating the subspace similarity at each training step would be computationally expensive. Moreover, $k$ ensures that newly added matrix elements are sufficiently trained before the next similarity evaluation. Thus, $k$ and $\epsilon$ are indispensable for ensuring the effectiveness of our method.

[1] Jiawei Zhao, Yifei Zhang, Beidi Chen, Florian Tobias Schaefer, and Anima Anandkumar. Incremental low-rank learning. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023.

W4: About relationship between low rank and forgetting, and high rank and generalization.

Thanks for your comment. We agree that a solid theory for this relationship would significantly enhance our work. However, to the best of our knowledge, we are the first to explore and apply this relationship in the context of continual learning for LLMs. Our primary objective is to investigate whether this empirical relationship can effectively improve parameter-efficient continual learning in LLMs. To this end, we propose CP-Rank, an empirical method that dynamically increases the rank for new tasks based on subspace similarity conditions. It is also worth noting that the theoretical understanding of continual learning for LLMs remains underdeveloped, even the theory of continual learning primarily focuses on two-task settings, leaving complexities of continual learning across multiple tasks largely unexplored. Our work is to show that the empirical relationship between low rank and forgetting, and high rank and generalization, can be effectively leveraged to enhance continual learning performance. We believe that establishing a theoretical foundation for this relationship is a highly promising direction for future research.

W5: About Figure 1

Thanks for your suggestion. We will increase the font size of Figure 1 to make it clear enough for readers in the revised manuscript.

Thank you for your detailed feedback on our work. We have thoughtfully addressed your insights and concerns, and hope our responses offer the necessary clarity on the issues raised.

W1: About CP-Rank without orthogonal projection

Thanks for your question about the CP-Rank without orthogonal projection. The only difference between CP-Rank with/without orthogonal projection is the loss function. CP-Rank without orthogonal projection exactly achieved Algorithm 1 and Algorithm 2 with the loss in Eqn(1), without any orthogonal regularizer. CP-Rank (wo OP) performance only depends on the rank-increasing update. In Table 1 of our manuscript, CP-Rank (wo OP) outperforms other baselines except O-LoRA, which means only using the well-designed rank-increasing update without any explicit orthogonal methods during training can overcome forgetting and achieve generalization. Thus, CP-Rank (wo OP) reveals that leveraging low-rank property during updating can balance the trade-off in continual learning. In contrast, CP-Rank (w OP) uses the loss with the orthogonal regularizer to make subspaces of the new task gradually orthogonal to the subspaces of previous tasks, and with the help of orthogonality, CP-Rank (w OP) would perform better than CP-Rank (wo OP). To understand the effectiveness of rank-increasing update in CP-Rank (wo OP), we compare it with IncLoRA (uses a fixed rank of 8 for each new task with loss in Eqn(1)):

Across all orders, CP-Rank (wo OP) outperforms IncLoRA by about 10%. This improvement is only caused by the well-designed rank-increasing update, which means only dedicatedly increasing ranks when subspaces of tasks are assumed to be orthogonal can improve the performance greatly.

W2: About newly added matrix elements in Algorithm 2

Thanks for your interest in newly added matrix elements. The newly added matrix elements are randomly initialized when the rank is increased and are trained alongside the original elements in subsequent iterations, which means they are not required to be retrained in the current iteration. In CP-Rank, the new subspace similarity would be calculated in the following iterations, including the newly added matrix elements which are treated as original elements in the following steps after rank increase. The orthogonality assessment would be in the following iterations rather than the current iteration.

W3: About "plasticity" and "generalization"

Thanks for suggesting "plasticity". Since "generalization" also has the meaning of "the model's capacity to perform well on new, unseen data beyond the training set", we consider both "plasticity" and "generalization" to be used in learning new knowledge/tasks.

Q: About testing without task IDs

Thanks for raising this question. In our continual learning setup, we use a newly initialized LoRA for each new task to train while freezing the pre-trained model parameters and previous tasks' LoRAs during the new task training. Previous tasks' LoRAs can be treated as concatenated with pre-trained model parameters. When training the new task, previous tasks' LoRAs and the pre-trained model parameters involve the inference (forward pass) even though they are frozen. When testing tasks, we use the entire model-including the pre-trained model, previous tasks' LoRAs, and new task LoRA-to test, which reflects the model's continual learning capabilities, as it measures performance after learning new tasks. Thus, in our setting and framework, there is no task recognition and adapter selection in testing.

We sincerely appreciate your thorough review as well as constructive feedback. Your comments are extremely helpful. We have carefully addressed each of your concerns and provided detailed answers to your questions below:

W1: About experiments using larger models

Thank you for suggesting the application of our method to larger models. In SAPT[1] and O-LoRA[2], the main results (results on continual learning benchmarks) shown in their paper both use T5-large model to present. Per your suggestion, to show the effectiveness of our method on larger language models, we would like to present the experimental results using Llama-2-7b-chat model to evaluate the performance of CP-Rank with orthogonal projection. We use 6 NVIDIA A100 GPUs to conduct the main experiments on three task orders in the standard CL benchmark.

These results indicate that our method can also work with larger language models. We are committed to using Llama-2-7b-chat or even larger models in the subsequent revision.

W2: About connection between Equation 7 and Equation 5

Thank you for raising this comment. Equation 7 and Equation 5 are indeed related. Equation 7 is defined as: $\phi(U_{i}^{l},U_{j}^{l}) = \frac{\sum_{s=1}^{p}\sigma_{s}^2}{p} = \frac{1}{p}(1-d(U_{i}^{l},U_{j}^{l})^2)$, where $\sigma_{s}$ are the singular values of $(U_{i}^{l})^\top U_{j}^{l}$, and $U_{i}^{l}$ and $U_{j}^{l}$ are the left singular matrices of $(B_i^{l}A_i^{l})^\top B_i^{l}A_i^{l}$ and $B_j^{l}A_j^{l}$, respectively. The singular values of $B_j^{l}A_j^{l}$ depend on the alignment of their left and right singular vectors. And $(U_{i}^{l})^\top U_{j}^{l}$ captures the cosine of the principal angles between the subspaces spanned by $U_{i}^{l}$ and $U_{j}^{l}$. Consequently, the singular values of $(U_{i}^{l})^\top U_{j}^{l}$ are proportional to the singular values of $(B_i^{l}A_i^{l})^\top(B_j^{l}A_j^{l})$. This relationship can be expressed as: $\sum_{s=1}^{p}\sigma_{s}^2\propto\text{tr}((U_{i}^{l})^\top U_{j}^{l})\propto\text{tr}((B_i^{l}A_i^{l})^\top(B_j^{l}A_j^{l}))$. It shows that the singular values of $(B_i^{l}A_i^{l})(B_j^{l}A_j^{l})^\top$ in Equation 5 are connected to the singular values of $(U_{i}^{l})^\top U_{j}^{l}$ in Equation 7. We will provide a detailed explanation in the revised version of our manuscript.

W3: About regularization in Equation 8

Thanks for your question about this. In our approach, we already perform SVD decomposition on LoRA $B_i A_i$ at the step when determining whether to increase the rank. During this step, we obtain the singular matrices, which can be directly used in Equation 8, avoiding redundant computations. In the "Rank Bonus" section, we introduce a regularization strategy similar to O-LoRA that includes a term $\sum_{i}A_i A_j$ to encourage orthogonality between the low-rank subspaces of different tasks. The goal of this regularization is to minimize interference across tasks by making their respective subspaces as orthogonal as possible. Furthermore, when increasing the rank, we initialize the new components randomly, and thus to effectively incorporate these new elements, we should update the loss function to include them. Since $B_i A_i$ can be spanned by its left singular matrix, we use this representation within the regularization term. This ensures the new task subspace spans a broader range, allowing the model to allocate a higher rank if needed, ultimately improving the performance of tasks.

Q: About motivation of our algorithm

Thank you for suggesting methods to achieve orthogonality during the initialization of the new task LoRA components. In our algorithm, the primary objective is to dynamically determine when and how to increase the rank of the LoRA layers during training, rather than focusing on the initialization. Our approach leverages the low-rank property of LoRA to balance the trade-off between mitigating forgetting and enhancing generalization in continual learning. Specifically, we use the reversed Grassmann distance to measure the similarity between the subspace of the new task's LoRA and those of previous tasks. This metric helps assess subspace orthogonality and guides rank increases only when necessary. The reason we choose random initialization is that the goal of this work is to focus on the updating process in continual learning rather than the initialization. We consider designing an orthogonal initialization strategy as a promising direction for future research and plan to explore it further.