Continuous Subspace Optimization for Continual Learning

We sincerely appreciate your constructive feedback. We have carefully considered your concerns and our responses are provided as follows.

---

W1 & Q1. It is unclear why complete separation is necessary. Is it always desirable for new task updates to be entirely independent of prior knowledge? Could past knowledge be beneficial for the current task? If strict independence is required, why not simply train a separate model per task? Please clarify why strict orthogonality (i.e., no interference) is necessary. Could shared representations benefit learning in some cases?

A1. We thank the reviewer for raising this important point regarding the necessity of orthogonal subspaces. We clarify that the orthogonality constraint in CoSO is not meant to enforce complete independence from past knowledge, but rather to mitigate harmful gradient interference during optimization. By projecting the current task's gradients orthogonally to the subspaces important for previous tasks, CoSO prevents overwriting past knowledge while still allowing parameter updates in complementary directions.

Notably, the orthogonality constraint is applied only during optimization. CoSO maintains a single shared pre-trained model throughout the entire training process. During inference phase, all model parameters jointly contribute, ensuring that useful representations from earlier tasks are retained and leveraged in future predictions. Therefore, past knowledge can benefit the current task.

We note that our setting follows the task-agnostic inference protocol, where the task label is unavailable at test time. In this setting, even if a separate model were trained per task, it would be infeasible to determine which model to use at inference. In contrast, CoSO maintains a single unified model that achieves strong performance without requiring any task labels at test time, demonstrating its practical effectiveness and scalability under realistic continual learning scenarios.

---

W2 & Q2. CoSO and SD-LoRA appear to share several similarities. A more detailed comparison is needed, including algorithmic rationale and computational efficiency. Please elaborate on how CoSO differs from SD-LoRA in design, efficiency, and empirical performance.

A2. We appreciate the reviewer's insightful observation and agree that both CoSO and SD-LoRA share the common goal of continual learning through parameter-efficient fine-tuning. However, the two approaches differ in both design and implementation, as clarified below:

(1) Method Design: SD-LoRA assigns a separate LoRA adapter to each task, where each adapter comprises a pair of low-rank matrices. While SD-LoRA decouples magnitude and direction to enable flexible adaptation, it does not explicitly prevent task interference. In contrast, CoSO avoids task-specific modules by optimizing a shared backbone. To mitigate interference, CoSO enforces gradient orthogonality, ensuring updates remain orthogonal to prior task subspaces.

(2) Memory Efficiency: CoSO leverages gradient-based low-rank projection to optimize a single shared backbone, enabling memory-efficient training without maintaining multiple sets of adapter weights. This optimization strategy ensures that the GPU memory consumption remains constant, regardless of the number of tasks. In contrast, SD-LoRA must retain all previously learned adapters in memory during training to prevent forgetting, resulting in the GPU memory usage that grows linearly with the number of tasks. This becomes a bottleneck when handling long task sequences. As shown in Appendix E (Table 5), CoSO consumes less memory than SD-LoRA, demonstrating better scalability in both theory and practice.

(3) Empirical Performance: CoSO consistently outperforms SD-LoRA across all benchmarks. For example, on ImageNet-R (20 tasks), CoSO improves final accuracy by 2.93% and average accuracy by 3.47% (Table 1). These results highlight the effectiveness of gradient space orthogonalization and dynamic subspace composition in enhancing knowledge retention and adaptation.

In summary, CoSO differs from SD-LoRA by (i) using online gradient-derived subspaces rather than static adapters, (ii) explicitly preventing forgetting via orthogonal projection, and (iii) requiring less memory. We will revise the paper to make these distinctions clearer.

---

W3 & Q3. The method builds on existing techniques such as SVD, GaLore, and FD, with orthogonality as the key addition for continual learning. A more thorough explanation of its technical novelty would strengthen the paper. Please highlight the technical novelty of CoSO beyond combining known techniques.

A3. We thank the reviewer for raising this important point. We would like to clarify the high-level motivation behind CoSO and its key technical contributions.

Recent continual learning methods with pre-trained models often follow a common paradigm: adapting a fixed subset of parameters (e.g., prompts or low-rank adapters) to mitigate forgetting while preserving generalization ability of foundation models. Among them, LoRA-based approaches (e.g., InfLoRA) constrain updates within a fixed low-rank subspace. While this reduces interference, it also limits the model's learning capacity.

Our work takes a novel perspective: instead of introducing prompts or adapters, we tackle the limitation of learning capacity by rethinking the optimization space. Our motivation is to fully exploit the expressive potential of pre-trained models by approximating full-rank fine-tuning in a parameter-efficient way, without modifying the model architecture. To this end, we propose to optimize in a sequence of evolving subspaces following GaLore, each derived from current gradients via truncated SVD. This dynamic subspace formulation allows CoSO to retain adaptation flexibility while remaining memory efficient.

However, unconstrained subspace evolution across tasks leads to severe forgetting. To address this, we introduce a gradient orthogonality constraint that ensures updates for each task do not interfere with learned knowledge. A major technical challenge is that each task comprises multiple subspaces across training steps. To make orthogonalization feasible, we aggregate these local subspaces into a compact component using Frequent Directions, which allows us to track only the essential directions unique to each task.

We would like to emphasize that CoSO is not a simple combination of SVD, GaLore, and FD. Each component plays a distinct role in addressing specific challenges of continual learning. Together, they form a cohesive framework that improves both learning capacity and knowledge retention. As a result, CoSO consistently outperforms existing Prompt-based and LoRA-based methods across benchmarks.

---

W4 & Q4. More fine-grained analysis would help assess the method's ability to mitigate forgetting and maintain the stability-plasticity trade-off. Please include more detailed evaluation to show how well CoSO mitigates forgetting and addresses stability-plasticity.

A4. We appreciate the reviewer's suggestion to include more detailed evaluation. Our primary evaluation focuses on final task accuracy and average accuracy, which are widely adopted in existing PEFT-based continual learning works. These metrics provide a comprehensive view of both adaptation (learning new tasks) and retention (preserving previous knowledge), and allow for direct comparison with prior state-of-the-art methods.

Following the suggestion of reviewer fJcc, we have computed Backward Transfer (BWT) for all methods on ImageNet-R 20 tasks to further evaluate the effectiveness of mitigating catastrophic forgetting. The results below show that CoSO not only achieves the highest final and average accuracy, but also obtains the highest BWT (closest to zero), indicating minimal forgetting.

---

Q5. SD-LoRA includes insightful empirical analyses, such as demonstrating the importance of previously learned directions for the current task. Could the authors provide similar analyses to show the role of historical subspaces in CoSO?

A5. We thank the reviewer for this insightful point. SD-LoRA retains all task-specific adapters and is therefore able to directly analyze the importance of previously learned adapters by reusing them during training. In contrast, CoSO does not store the actual parameter updates or direction vectors, preserving only the historical subspace constructed via sketching. While this difference prevents us from conducting similar analysis as in SD-LoRA, we can design experiments to assess the contribution of the historical subspace.

Specifically, our ablation study removes the orthogonality constraint, allowing updates to overlap with past subspaces. The results show a substantial performance drop (e.g., −8.52% final accuracy in the 20-task setting), indicating that the historical subspace plays a crucial role in preserving past knowledge. This result strongly suggests that the accumulated subspace effectively mitigates interference.

We sincerely appreciate your thoughtful and encouraging feedback on our work. Below, we provide a detailed response to your concern.

---

W1. Explain the specific difference between $r_1$ and $r_2$, and whether they could potentially be merged into a single parameter?

R1. Thank you for the insightful question regarding $r_1$ and $r_2$. We fully agree that a model should ideally involve as few hyperparameters as possible to promote simplicity and ease of tuning. While these two parameters may appear similar, they are designed to serve different purposes.

The projection rank $r_1$ is used during the optimization process to control the dimensionality of the gradient subspace in which parameter updates are performed. A smaller $r_1$ ensures that the projected gradient is memory-efficient and computationally light, which is especially critical during training.

On the other hand, the Frequent Directions rank $r_2$ is used to construct the task-specific subspace by consolidating gradient directions across training steps. This subspace serves as a compact summary of the current task's critical update directions and is appended to the historical task subspace to preserve prior knowledge. To achieve a more accurate estimation of the task-specific gradient subspace, $r_2$ is set larger than $r_1$, which allows it to capture a broader spectrum of gradient variation over time.

Yes, we can merge $r_1$ and $r_2$ into a single parameter. But we find that tuning them separately leads to better performance in practice. A relatively larger $r_2$ becomes important when $r_1$ is small, as it enables better approximation of the task-specific subspace.

---

W2. The baselines are limited to Prompt-based and LoRA-based methods. It would be helpful to include comparisons with recent studies, such as RanPAC and EASE.

R2. Thank you for the valuable suggestion. Following your suggestion, we conducted additional experiments on ImageNet-R using the official implementations and matched settings. The results further demonstrate the superior performance of CoSO across diverse continual learning paradigms.

Our study focuses primarily on continual learning with pre-trained vision transformers under the parameter-efficient fine-tuning (PEFT) paradigm. The baseline methods we originally included are state-of-the-art PEFT-based approaches widely adopted in recent literature, ensuring a fair and relevant comparison in this specific setting. While RanPAC and EASE are valuable prototype-based methods that operate in the feature space via class prototypes, they follow a different paradigm from PEFT-based approaches.

---

Q1. Does Eq.(7) still satisfy Proposition 1?

A1. Thank you for the thoughtful question. Yes, Equation (7) still satisfies Proposition 1. In CoSO, we first project the raw gradient $G_{\tau,t}$ into the null space of the historical task subspace, yielding $G_{\tau,t}^{\prime}$. All subsequent steps in Equation (7), including low-rank projection, Adam update, and projection back, operate within the subspace spanned by $G_{\tau,t}^{\prime}$. As a result, the final update $\tilde{G}_{\tau,t}$ still satisfies Proposition 1.

---

Q2. Why does CoSO restrict updates to the linear projection layers and not to FFN parameters? Could this approach be extended to update the MLP layers in FFN as well?

A2. Thank you for this question. Our goal is to achieve parameter-efficient fine-tuning by updating as few parameters as possible while maintaining strong performance. Following this principle, CoSO restricts optimization to the output projection layers and keeps all other parameters (including the FFN layers) frozen. By focusing on the projection layers, we achieve effective continual adaptation while significantly reducing computational and memory overhead. Theoretically, CoSO can be extended to other modules such as FFN layers.

---

Q3. Could CoSO be applied to non-vision tasks like NLP? If so, what modifications would be needed for it to work effectively?

A3. Thank you for this interesting question. While our current work focuses on vision tasks using pre-trained vision transformers, the idea underlying CoSO, which involves projecting gradients into continuous subspaces and enforcing orthogonality across tasks, is model-agnostic and applicable to standard transformer architectures. Consequently, the algorithm can be directly adapted to popular NLP models such as LLaMA by targeting the corresponding projection layers, where gradient-based subspace adaptation remains effective. This presents a promising direction for cross-modal continual learning. We intend to explore this extension in our future research.

---

Q4. Could CoSO be combined with other continual learning approaches, such as regularization-based or prototype-based?

A4. Thank you for the insightful question. CoSO is designed as a subspace-based optimization framework that operates at the level of parameter gradients and explicitly enforces orthogonality between tasks. Due to this design, it naturally overlaps with the objective of many regularization-based methods (e.g., EWC), which also aim to prevent interference through constrained updates. As a result, combining CoSO with regularization-based approaches may offer limited additional benefit.

In contrast, prototype-based methods operate in the feature space and are conceptually orthogonal to CoSO. These methods, which store and use class prototypes to facilitate classification and knowledge retention, can be naturally integrated with CoSO by replacing or augmenting the classifier head while retaining CoSO for backbone optimization. This hybridization can potentially improve performance by leveraging both parameter-space and feature-space strategies.

We sincerely thank you for your constructive feedback on our paper. We hope our responses adequately address the concerns you raised.

---

W1. Why does the CoSO outperform NSP[1] and PGP[2] in various continual benchmarks? Is the better performance of CoSO attributed to the utilization of LoRA?

R1. Thank you for raising this insightful question. While prior works such as NSP[1] and PGP[2] leverage orthogonal gradient projection to enhance prompt-based continual learning, CoSO differs in both its update scope and optimization strategy, leading to consistently stronger performance.

• Prior methods[1, 2] constrain updates to the additional prompt tokens appended to the transformer, limiting the model's ability for task-specific adaptation. In contrast, CoSO directly updates the projection layers of the pre-trained vision transformer, which significantly expands the learning capacity of the model for continual adaptation.
• In our baselines, we also include state-of-the-art LoRA-based continual learning methods such as InfLoRA [3] and SD-LoRA [4]. InfLoRA restricts parameter updates to a fixed subspace, which inherently limits flexibility. In contrast, CoSO performs optimization in a series of dynamically evolving gradient subspaces. These subspaces are explicitly enforced to be orthogonal to the historical task subspace, enabling CoSO to continually discover new learning directions while avoiding interference.

To summarize, CoSO's superior performance stems from the novel design that jointly enables dynamic subspace exploration and orthogonal knowledge preservation, which are absent in prior Prompt-based or LoRA-based approaches.

---

W2. The paper does not discuss whether the CoSO is plug-and-play.

R2. Thank you for this valuable suggestion. CoSO is indeed a plug-and-play optimization strategy in continual learning pipelines. Unlike approaches that require model architectural augmentation (e.g., prompts or adapter), CoSO operates solely on the optimization procedure. It only requires access to the gradients of the parameters being optimized and performs low-rank projections and orthogonalization during the update. Therefore, CoSO can be directly integrated with other methods (e.g., prototype-based) that update a subset of backbone parameters without changing the model architecture. We thank the reviewer for encouraging us to make this clearer in the manuscript.

---

W3. The historical task subspace $\mathcal{M}_{\tau-1}$ should span the representation subspace of all previous tasks, rather than the gradient subspace.

R3. Thank you for this insightful question. We assume that both the feature and gradient subspaces lie in the same space and capture similar task-specific update directions. Working in the gradient subspace allows for a seamless integration with our low-rank projection-based optimization strategy. Specifically, we can directly apply orthogonality constraints to parameter updates without requiring access to intermediate feature representations. This effectively prevents interference by ensuring that current gradient updates are orthogonal to those from previous tasks. As a result, the risk of increasing the loss on prior tasks is minimized, thereby preserving previously learned knowledge.

We will revise Section 3.2 to make the above rationale clearer. We thank the reviewer again for raising this point.

---

Reference

[1] Visual prompt tuning in null space for continual learning. NeurIPS, 2024

[2] Prompt Gradient Projection for Continual Learning. ICLR, 2024.

[3] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. CVPR 2024.

[4] SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning. ICLR 2025

We sincerely appreciate your valuable feedback. We have carefully considered your concerns and provided a detailed explanation below.

---

W1. CoSO introduces several key hyperparameters, including projection rank $r_1$, FD rank $r_2$, update interval $K$, and threshold $\epsilon_{th}$. These hyperparameters vary significantly across different datasets. Does this suggest that the method requires problem-specific tuning to achieve optimal performance?

R1. We appreciate the reviewer's concern about hyperparameter sensitivity. In our experiments, we fixed $\epsilon_{\text{th}}$ = 0.98 across all datasets. This setting is found to be robust in practice. The update interval K determines how frequently the projection and sketch matrices are computed. On larger datasets like DomainNet, we use larger $K=20$ to reduce overhead, while for smaller datasets with more volatile gradients, we use $K=1$ for better performance. Many existing PEFT-based continual learning methods also involve dataset-specific hyperparameters such as prompt length or LoRA rank. We followed the standard practice by tuning all hyperparameters on validation splits, ensuring fairness in comparison.

To assess the impact of $r_1$ and $r_2$, we conducted sensitivity experiments varying their values on ImageNet-R. The results (final acc and average acc) below indicate that CoSO remains robust across a wide range of rank settings, with only minor performance fluctuations.

---

W2. Continual learning typically employs more direct measures like Backward Transfer (BWT). The absence of such metrics makes it difficult to precisely evaluate the effectiveness of mitigating catastrophic forgetting.

R2. We appreciate the reviewer's suggestion to include more direct metrics such as BWT. Our primary evaluation focuses on final accuracy and average accuracy, which are widely adopted in existing PEFT-based continual learning works. These metrics jointly reflect adaptation and retention, enabling fair comparisons with state-of-the-art methods.

Following the reviewer's suggestion, we compute BWT for all methods on ImageNet-R 20 tasks to evaluate the effectiveness of mitigating forgetting. The results below show that CoSO achieves the highest BWT, indicating minimal forgetting.

---

W3. To maintain theoretical orthogonality, CoSO restricts its optimization scope to the output projection layer, excluding the QKV layers. Compared to PEFT methods that can update QKV layers, this may limit the model's adaptability and sacrifice potential performance improvements.

R3. We thank the reviewer for raising this point. Our goal is to achieve parameter-efficient fine-tuning by updating as few parameters as possible while maintaining strong performance. Following this principle, CoSO restricts optimization to the output projection layers and keeps all other parameters (including the QKV layers) frozen. To assess the effectiveness of this selective update strategy, we compare CoSO against baselines such as InfLoRA and SD-LoRA, which fine-tune the QKV layers. CoSO consistently outperforms these methods across multiple benchmarks. This suggests that tuning the projection layers alone is sufficient for effective adaptation, validating our design choice.

---

W4. The experiments exclusively use datasets from the visual domain, such as ImageNet, the CIFAR series, and COCO for object detection. The study does not include tasks from other modalities, like natural language processing. This limits the method's generalizability within a universal fine-tuning paradigm.

R4. We thank the reviewer for pointing out the scope of our experimental setup. While our current experiments focus on the visual domain, this choice is intentional to ensure a fair comparison with strong and well-established baselines in vision-based continual learning. Notably, the core mechanisms of CoSO, namely gradient-based subspace projection and orthogonality constraints, are not limited to the visual modality. These mechanisms are inherently general and can be extended to transformers used in NLP. As part of our ongoing work, we are actively exploring extensions to vision-language settings, including the adaptation of CoSO to CLIP-based models. However, such extensions involve non-trivial re-implementations and are beyond the scope of this initial study. We intend to explore this extension in our future research.

---

Q1. Appendix D states that due to the non-linearity of Softmax, CoSO is not applied to the QKV transformation layers within the attention mechanism. Could you clarify what proportion of the total fine-tuned parameters are excluded as a result? Furthermore, how would you assess the performance loss incurred by not being able to enforce orthogonality on these critical layers?

A1. We appreciate the reviewer's question regarding the exclusion of QKV layers. In ViT-B/16, the QKV layers account for approximately 25% of the total parameters. Theoretically, CoSO can be applied to the remaining 75% of parameters, including projection layers and FFN layers, which still comprise the majority of the transformer's components.

As discussed in R3, updating more parameters does not always yield better results. By freezing most of the model parameters (including the QKV layers), we are able to better preserve and leverage the generalization capabilities of the pre-trained backbone, allowing CoSO to achieve strong performance with relatively few trainable parameters.

---

Q2. Regarding the baseline PEFT methods in the experiments, was data replay employed during fine-tuning? Have the authors considered including a fine-tuning baseline with data replay for a more comprehensive comparison?

A2. We thank the reviewer for raising the question regarding data replay. As discussed in Section 3.1, we focus on a rehearsal-free setting in which the model is not allowed to access or store data from previous tasks. This setting is widely considered more challenging, as it forgoes the benefits of revisiting past data. Our setup aligns with recent state-of-the-art PEFT-based continual learning methods. To ensure fair comparisons, all baselines included in our experiments also follow this rehearsal-free protocol.

To address your concern, we conduct additional experiments on ImageNet-R of MEMO[1], a typical replay-based method. For a fair comparison, we use 40 images per class for replay. In addition, following reviewer ndbX's suggestion, we include RanPAC[2] and EASE[3], two representative methods that adopt prototype-based replay. These additional results further validate the superior performance of CoSO.

---

Q3. Is the current comparison against baseline methods fair? When compared to methods like InfLoRA, how much of CoSO's performance gain can be attributed to its continuous subspace optimization strategy versus its larger number of trainable parameters (given that r2 > r1)? Would you consider conducting additional experiments to compare performance under a similar total parameter budget, for instance, by increasing the rank of InfLoRA to match the effective rank of CoSO?

A3. We appreciate the reviewer's insightful concern regarding fairness in baseline comparisons. We would like to clarify that the projection rank $r_1$ defines the rank of the gradient subspace, and does not add any extra trainable parameters to the model. The FD rank $r_2$ is not used to directly update model weights, but helps construct the task-specific subspace via the FD. The improved performance is stems from CoSO's ability to (1) optimize in multiple evolving subspaces rather than a fixed one, and (2) enforce orthogonality to mitigate task interference.

To further address your concern, we conduct additional experiments where we increase the rank in InfLoRA to match the rank of CoSO. As discussed in Appendix E, increasing the rank of InfLoRA would not yield improvements.

---

Reference

[1] A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning. ICLR 2023.

[2] RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS 2023.

[3] Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning. CVPR 2024.

Dear Area Chair,

Thank you very much for your kind reminder and efforts in managing our submission.

We have carefully addressed all reviewer comments in our rebuttal, including additional analyses, discussions, and new experimental results. We would be happy to provide further clarifications if needed before the discussion period concludes.

Finally, we sincerely thank the reviewers for their time and constructive feedback.

Best regards,
The Authors