LW2G: Learning Whether to Grow for Prompt-based Continual Learning

Q3.How LW2G differs from existing orthogonal continual learning methods.

We would like to clarify to the reviewer that the core contribution of this paper is not to propose a new gradient-projection method, but rather to address the issues of prompt set redundancy in prompt-based methods, which hinder cross-task facilitation and make it challenging to select the correct prompt set during testing (as reflected in the proposed PRA metric). Furthermore, we explicitly mention in the conclusion that proposing a new gradient-projection method is completely orthogonal to the LW2G approach proposed in this paper.

Besides, while [J] introduces gradient projection into prompt-based CL, its purpose is entirely different from ours. We have also compared LW2G with [J], and we kindly ask the reviewer to refer to Appendix F.4.

To further eliminate any misunderstanding from the reviewer, we will also revise the manuscript in the revised version to clarify the core contribution as follows:

1.Prompt Pool Design: The LW2G proposed in this paper is the first prompt-based CL method to suggest that task-relatedness should guide whether to expand the prompt pool. This approach results in an efficient and effective prompt pool, which not only reduces the parameter overhead of prompts but also facilitates cross-task knowledge transfer.

2.Novel Metric (HFC): This paper is the first to propose using the magnitude of gradient correction to measure the degree of hindrance to learning new tasks under orthogonal constraints and to define a concrete numerical metric, HFC (Hindrance Forward Capability). While prior works in gradient-based continual learning, such as [G], have mentioned that strict orthogonality conditions can hinder learning new tasks, no prior work has provided a clear numerical metric to quantify this hindrance. HFC is the first such metric.

3.Dynamic Prompt Pool Expansion Using HFC: Furthermore, this paper innovatively proposes using HFC to dynamically determine prompt pool expansion. Compared to methods such as [H] and [I], which manually set thresholds for deciding expansion, the proposed HFC not only eliminates the need for manually setting parameters (which often requires strong prior assumptions) but also provides a solid theoretical foundation to support this decision.

Q4.Does the proposed method require computing the representation matrix for every layer after learning each task?

We calculate the representation matrix in the same locations and using the same implementation as [I]. Specifically, the calculation of representation matrix depends on the baseline methods being compared. For example, in DualPrompt, the prompt set is only inserted into transformer blocks 0–2, so the representation matrix only needs to be calculated based on the output of transformer blocks 0–2. In S-Prompt++ and HidePrompt, the prompt set is inserted into transformer blocks 0–4, so the representation matrix needs to be calculated based on the output of transformer blocks 0–4.

Ref:

[A] Consistent Prompting for Rehearsal-Free Continual Learning

[B] CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning

[C] One-stage Prompt-based Continual Learning

[D] Rethinking Gradient Projection Continual Learning: Stability / Plasticity Feature Space Decoupling,CVPR2023

[E] Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners

[F] Divide and not forget: Ensemble of selectively trained experts in Continual Learning,ICLR2024

[G] Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners

[H] Divide and not forget: Ensemble of selectively trained experts in Continual Learning,ICLR2024

[I]Prompt Gradient Projection for Continual Learning, ICLR 2024.

Q1. Confusion due to presentation.

• Notation $l$. We apologize for the confusion caused. In the proof of Theorem 1, we have replaced the symbol $l$ with another notation. Please refer to the Appendix B.1 in our revised manuscript.
• Vague description. In our revised manuscript:
  1. We provided a precise description torwards gradient projection (see lines 160-162).
  2. We have added details about the method for obtaining the $B_{t}^{l}$ matrix to the main paper to make our proposed method easier to understand. (See lines 167–203).
  3. We followed [J] and utilized 768 samples to construct the representation matrix.

Q2.Details about baselines HidePrompt.

1.implementation Issue in HidePrompt: We kindly ask the reviewer to refer to the caption of Table 1 and Appendix E in the submitted manuscript, where we mentioned that the official code of HidePrompt has an implementation issue regarding prompt retrieval. This implementation issue caused a leakage problem, which led to the overestimation of results reported in the original paper. We asked the authors of HidePrompt for the fixed version of the code and reproduced the results.

2.Comparsion with Additional Baselines: Following the reviewers' suggestion, we have made extensive efforts over the past week to reproduce 6 other baselines for a more systematic comparison, including CPrompt[A], CODAPrompt[B], OS-Prompt[C], InfLora[D], C-Lora[E], Hide-Lora[F]. The results are provided in the following Table. LW2G demonstrates consistent improvements over each baseline by 0.19% to 2.02%, while significantly reducing the number of parameters (prompt sets or LoRA sets). For instance:

• In Hide-Lora, the number of LoRA sets is reduced from 10 to 2.
• In CPrompt, the number of prompt sets is reduced from 10 to 5.

Overall, the experimental results demonstrate that LW2G, as a plug-in module, exhibits excellent versatility. It can be flexibly integrated with many existing methods while achieving a balance between performance and parameter overhead.

3.Updates in the Revised Manuscript: We have added this discussion in our revised manuscript (refer to lines 1566-1647 and Appendix G).

Thank you for your recognition of our response and additional experiments! We sincerely appreciate the time and effort you've dedicated to our paper.

Thanks for the detailed response, which addressed most of my concerns, particularly regarding the performance of the baseline models and the clarification of the proposed contributions. I also appreciate the inclusion of additional comparison results with more baseline models, which further demonstrate the effectiveness of the proposed method. Based on these improvements, I have raised my score accordingly.

Q4.Gradient projection and whether to choose growth are not novel in the field of CL, but they are somewhat novel in the PTM-based CIL task.

We agree with the reviewer’s viewpoint, but we would like to emphasize the contributions of this paper:

1.Prompt Pool Design: The LW2G proposed in this paper is the first prompt-based CL method to suggest that task-relatedness should guide whether to expand the prompt pool. This approach results in an efficient and effective prompt pool, which not only reduces the parameter overhead of prompts but also facilitates cross-task knowledge transfer.

2.Novel Metric (HFC): This paper is the first to propose using the magnitude of gradient correction to measure the degree of hindrance to learning new tasks under orthogonal constraints and to define a concrete numerical metric, HFC (Hindrance Forward Capability). While prior works in gradient-based continual learning, such as [L], have mentioned that strict orthogonality conditions can hinder learning new tasks, no prior work has provided a clear numerical metric to quantify this hindrance. HFC is the first such metric.

3.Dynamic Prompt Pool Expansion Using HFC: Furthermore, this paper innovatively proposes using HFC to dynamically determine prompt pool expansion. Compared to methods such as [O] and [P], which manually set thresholds for deciding expansion, the proposed HFC not only eliminates the need for manually setting parameters (which often requires strong prior assumptions) but also provides a solid theoretical foundation to support this decision.

ref:

[A] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning

[B] Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA

[C] HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning

[D] Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need

[E] PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer

[F] Expandable subspace ensemble for pre-trained model-based class-incremental learning,CVPR24

[G] Generating Instance-level Prompts for Rehearsal-free Continual Learning

[H] Continual learning with pre-trained models: A survey

[I] Consistent Prompting for Rehearsal-Free Continual Learning

[J] CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning

[K] One-stage Prompt-based Continual Learning

[L] Rethinking Gradient Projection Continual Learning: Stability / Plasticity Feature Space Decoupling,CVPR2023

[O] Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners

[P] Divide and not forget: Ensemble of selectively trained experts in Continual Learning,ICLR2024

Q3.Implement comparison with more baselines.

We thank the reviewer for their suggestion to validate the effectiveness of our method on more baselines.

Since LW2G is proposed as a plug-in module, it is orthogonal to other baseline methods. Following the reviewers' suggestion, we have made extensive efforts over the past week to reproduce 6 other baselines for a more systematic comparison, including CPrompt[I], CODAPrompt[J], OS-Prompt[K], InfLora[A], C-Lora[B], Hide-Lora[C]. The results are provided in the Table 20 in our revised manuscript. LW2G demonstrates consistent improvements over each baseline by 0.19% to 2.02%, while significantly reducing the number of parameters (prompt sets or LoRA sets). For instance:

• In Hide-Lora, the number of LoRA sets is reduced from 10 to 2.
• In CPrompt, the number of prompt sets is reduced from 10 to 5.

Overall, the experimental results demonstrate that LW2G, as a plug-in module, exhibits excellent versatility. It can be flexibly integrated with many existing methods while achieving a balance between performance and parameter overhead.

Updates in the Revised Manuscript: We have added this discussion in our revised manuscript (refer to lines 1566-1647 and Appendix G).

Thank you for the valuable comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions. We hope these responses provide sufficient reasons to raise the score. We would be happy to address any further questions.

Q1.Concerns about the motivation from PRA.

We appreciate the reviewer’s constructive comments. We regret that some descriptions in our submission were not clear enough, which led to misunderstandings.

1.Clarification on Motivation: First, we agree with the reviewer’s opinion that presenting PRA improvement as the motivation for this work is not appropriate. In fact, our true motivation is simple yet intuitive: we raise concerns about the current prompt-based CL methods that "learn a new prompt set for every incremental task" and propose a solution to dynamically decide, based on task differences (via the proposed HFC metric), whether to grow a new prompt set or reuse an existing one. This ultimately forms an effective and efficient prompt set pool, which not only reduces the number of trainable parameters (fewer prompt sets), thereby improving PRA, but also facilitates cross-task knowledge transfer when similar tasks share the same prompt set. Therefore, the improvement in PRA is not the original motivation of this work but rather a by-product of the dynamic growing strategy.

2.Comparison with Existing LoRA-Based Methods: Second, we acknowledge that existing LoRA-based methods [A,B] indeed avoid the PRA issue because they aggregate all LoRA branches during inference. However, another LoRA-based method, Hide-Lora [C], does face the PRA issue. Despite this, these methods [A-C] still adhere to the approach of "learning a task-wise LoRA branch for each incremental task."

Regarding the special adapter-based method [D], it avoids the PRA issue by learning a single adapter during the $1$-st incremental task and directly using this adapter for subsequent tasks. However, the information captured in the first task cannot represent all incremental tasks, especially on benchmarks with high task dissimilarity. Consequently, its performance is relatively inferior to other prompt-extend CL methods. This has been analyzed in prior works [E,F], which categorized methods into prompt-extend and prompt-fix approaches.

3.Updates to Address Misunderstandings: Finally, we acknowledge that some of our descriptions were indeed unclear and caused misunderstandings. Following the reviewer’s kind suggestions, we have made the following revisions: 1. lines 1566-1647 and Appendix G. 2. lines 1663-1680 and Appendix I.

Q2 Comparison with DAP.

Regarding DAP[G], DAP has the problem of batch information leakage which results in comparison unfairness. As stated by [H] in the discussions on comparison fairness, "during inference,... it is equal to directly annotating the task identity. When removing the batch information... a drastic degradation in the performance ...". The reproduced results reported in [H] are shown in the following Table.

The accuracy declines by 4.8~42.6% with an average of 27.3% when eliminating the batch information leakage. The official code of DAP, line 210-236 in file vit.py, assumes that test samples are batched and all of them come from the same task, which is an unreasonable assumption.

We sincerely thank the reviewer for the time and effort spent reviewing our manuscript and communicating with us.

The contributions of this manuscript are not impressive and significant.

We deeply regret that the reviewer did not find the contributions of this manuscript to be significant. We attribute this to our failure to clearly articulate certain aspects, which may have led to some misunderstandings. We would like to take this opportunity to further explain our work to the reviewer.

We acknowledge that both prompt-based CL methods and gradient projection CL methods are either currently being explored or have already been widely studied. However, there are very few works combining these two approaches. For example:

• PGP and Inflora are two preliminary attempts, but they only perform a straightforward combination of the two methods without addressing the issue of prompt set redundancy in prompt-based methods.

• Existing gradient projection CL methods are mostly focused on a fixed model and discuss non-expansion CL methods, such as GPM and TRGP. These studies completely overlook the potential of using gradient projection as a new perspective to rethink network expansion.

In contrast, this paper delves deeper and is the first to explore the relationship and combination of gradient projection CL methods and expansion CL methods in terms of the degree of learning hindrance caused by strict orthogonality conditions (via the proposed HFC metric). This is our unique and core contribution. This exploration offers a novel perspective on expansion CL methods.

In addition, we have followed the reviewer’s first-round suggestions and added 6 more baselines to verify the broader adaptability and effectiveness of the proposed LW2G. Moreover, we are already working on extending our approach to a wider range of expansion CL methods, which will be addressed in our future work.

Finally, we sincerely ask the reviewer to reconsider their evaluation of our manuscript. We are also more than willing to address any additional concerns the reviewer may have.

Q3. Typos.

Thanks, we will fix all typos and re-check the representation in the final version.

Q4. Evaluation under other pre-trained models.

We kindly ask the reviewer to refer to Appendix F.6 in the submitted manuscript. In Appendix F.6, we validate the effectiveness of LW2G on three additional pre-trained models: IBOT21k, IBOT1k, and DINO.

The consistent improvement under different Pre-trained models illustrated that, LW2G exhibits excellent versatility.

Q5.When does the Theorem 1 hold? Any assumptions?

In Theorem 1, we prove that, for any given vector $\alpha$ in a Euclidean space and two subspaces $S_{1}$ and ​$S_{2}$ in the same Euclidean space, where the bases of $S_{1}$ are entirely contained within the bases of ​$S_{2}$, the angle between $\alpha$ and its projection onto $S_{1}$ is greater than the angle between $\alpha$ α and its projection onto $S_{2}$.

In continual learning, when using gradient projection methods, the projection subspace gradually becomes larger (as more bases are added to the subspace). Consequently, the angle between real gradient and modified gradient becomes smaller, meaning that the difference between the real gradient and the corrected gradient grows larger, which ultimately leads to increased hindrance to learning new tasks.

Therefore, HFC effectively measures the degree of hindrance to new task learning under strict orthogonality conditions.

As a result, Theorem 1 only requires the assumption that all vectors $b_{i}$ are in the Euclidean space.

ref:

[A]Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality,NIPS23

[B]OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learning,ICLR24

[C]KOPPA: Improving Prompt-based Continual Learning with Key-Query Orthogonal Projection and Prototype-based One-Versus-All

[D] Consistent Prompting for Rehearsal-Free Continual Learning

[E] CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning

[F] One-stage Prompt-based Continual Learning

[G] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning

[H] Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA

[I] HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning

Thank you for the valuable comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions. We hope these responses provide sufficient reasons to raise the score. We would be happy to address any further questions.

Q1.Performance improvement on some datasets is limited.

We appreciate the reviewer for pointing this out. In response, we provide the following clarifications:

1.Two core aspects of LW2G to the improvement of FAA: Our method (baseline + LW2G), as a plug-in module, dynamically decides whether to add a new prompt set based on the similarity between tasks. The advantages it brings over the baseline primarily stem from two aspects: (1) Prompt effectiveness: Sharing a prompt set among similar tasks facilitates cross-task knowledge transfer. (2) Prompt efficiency: Utilizing fewer prompt sets improves the accuracy of prompt retrieval (PRA). Previous works [A-C] have already mentioned that the accuracy of prompt set prediction significantly impacts FAA and FFM. Therefore, we propose PRA, which represents the average prediction accuracy of the prompt set for all test samples.

2.Trade-off between trainable parameters and performance: As shown in Table 1 and Table 2 of the submitted manuscript, SSP (prompt set parameters) demonstrates that baseline + LW2G only requires 1/5 of the learnable parameters of the baseline, yet it still consistently outperforms the baseline on FAA and FFM.

3.Limited improvement on S-Prompt++: We kindly ask the reviewer to refer to Figure 3 in the submitted manuscript. Specifically, under the CIFAR_IN10_TASK10 setting, the baseline’s PRA is already 99.52%, leaving limited potentials for LW2G to further improve it. However, even in this case, LW2G leverages cross-task knowledge through prompt effectiveness, improving FFM from 4.10% to 3.46%.

Q2.Details about baselines HidePrompt. And other prompt-based methods, e.g. CODAPrompt. And other lora-based methods.

We appreciate the reviewer for pointing this out. In response, we provide the following clarifications:

1.implementation Issue in HidePrompt: We kindly ask the reviewer to refer to the caption of Table 1 and Appendix E in the submitted manuscript, where we mentioned that the official code of HidePrompt has an implementation issue regarding prompt retrieval. This implementation issue caused a leakage problem, which led to the overestimation of results reported in the original paper. We asked the authors of HidePrompt for the fixed version of the code and reproduced the results.

2.Comparsion with Additional Baselines: Following the reviewers' suggestion, we have made extensive efforts over the past week to reproduce 6 other baselines for a more systematic comparison, including CPrompt[D], CODAPrompt[E], OS-Prompt[F], InfLora[G], C-Lora[H], Hide-Lora[I]. The results are provided in the Table 20 in our revised manuscript. LW2G demonstrates consistent improvements over each baseline by 0.19% to 2.02%, while significantly reducing the number of parameters (prompt sets or LoRA sets). For instance:

• In Hide-Lora, the number of LoRA sets is reduced from 10 to 2.
• In CPrompt, the number of prompt sets is reduced from 10 to 5.

Overall, the experimental results demonstrate that LW2G, as a plug-in module, exhibits excellent versatility. It can be flexibly integrated with many existing methods while achieving a balance between performance and parameter overhead.

3.Updates in the Revised Manuscript: We have added this discussion in our revised manuscript (refer to lines 1566-1647 and Appendix G).

We sincerely appreciate the reviewers for dedicating their valuable time and continuous effort to reviewing and engaging in the discussions of this paper.

In the discussion, reviewer 9d2R raised two main concerns:

(1) The improvement from LW2G is relatively moderate, but not huge.

(2) Whether LW2G is effective with other pre-trained models or baselines.

Regarding Concern 1:

We agree with the reviewer's point that a good strategy should achieve a balance between performance and parameter overhead. In fact, current prompt-based CL methods learn a task-specific prompt/adapter/lora module for each incremental task (e.g., the e-prompt in DualPrompt), which completely ignores the parameter overhead problem (the additional learnable modules grow linearly with the number of tasks). In contrast, LW2G is the first approach in prompt/lora/adapter-based CL methods to consider controlling the growth of learnable modules. This is driven not only by (1) concerns about parameter overhead, but also (2) the fact that learning separately for each task hinders cross-task knowledge facilitation and leads to low task-specific module selection accuracy (PRA problem).

Therefore, the improvement brought by LW2G may not be very significant in some settings, but regardless of the size of the improvement, it comes with a dramatic reduction in the number of parameters. For example, DualPrompt+LW2G reduces the prompt pool from the original 10 prompt sets to 2 prompt sets, and Hide-Lora+LW2G reduces the lora pool from the original 10 lora sets to 5 lora sets. Moreover, we also evaluated the method on long-sequence tasks; please refer to Table 2 in the submitted manuscript. In the OMNIBENCHMARK setting, DualPrompt (baseline) maintains 30 prompt sets, while DualPrompt+LW2G only uses 9 prompt sets, with a performance improvement (FAA) of 1.76.

In summary, we believe that LW2G, as the first approach in prompt/lora/adapter-based CL methods to control the growth of learnable modules, is able to achieve consistently improved performance while drastically reducing the number of parameters.

Regarding Concern 2:

In our submitted manuscript, we provided validation of the effectiveness of LW2G with other pre-trained models in Appendix F.6.

Additionally, following the reviewer's suggestion, we conducted extensive experiments. Specifically, we evaluated the effectiveness of LW2G on 9 baselines: DualPrompt, S-Prompt++, HidePrompt, CODAPrompt, OSPrompt, CPrompt, C-Lora, InfLora, and Hide-Lora. These methods represent nearly three years of state-of-the-art prompt/lora-based CL methods. The table below shows that LW2G consistently improves performance across all 9 baselines, with average improvements of 0.95 and 1.07 on CIFAR and IMR, respectively. At the same time, LW2G also significantly reduces the parameter overhead of the baselines, with reductions of 74.17% and 66.39% on CIFAR and IMR, respectively.

Thank you for the valuable comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions. We hope these responses provide sufficient reasons to raise the score. We would be happy to address any further questions.

Q1.Statistical results with mean and std.

We thank the reviewer for their kind advice and would like to provide the following clarifications:

1.In fact, we follow [B] to run our benchmarks for several different shuffles of the task class order and report the mean of these runs. We do this with a consistent seed (different for each trial) so that the results can be directly compared. We have added the standard deviation in our revised manuscript (see lines 328-343 in Table 1).

2.Additionally, we have made extensive efforts over the past week to reproduce 6 other baselines for a more systematic comparison, including CPrompt[A], CODAPrompt[B], OS-Prompt[C], InfLora[D], C-Lora[E], Hide-Lora[F]. The results are provided in the Table 20 in our revised manuscript. LW2G demonstrates consistent improvements over each baseline by 0.19% to 2.02%, while significantly reducing the number of parameters (prompt sets or LoRA sets). For instance:

• In Hide-Lora, the number of LoRA sets is reduced from 10 to 2.
• In CPrompt, the number of prompt sets is reduced from 10 to 5.

Overall, the experimental results demonstrate that LW2G, as a plug-in module, exhibits excellent versatility. It can be flexibly integrated with many existing methods while achieving a balance between performance and parameter overhead.

3.Updates in the Revised Manuscript: We have added this discussion in our revised manuscript (refer to lines 1566-1647 and Appendix G).

Q2.Ablation Study on Hyper-parameters.

We kindly ask the reviewer to refer to Appendix F.1 in the submitted manuscript, where we have conducted extensive ablation experiments on the four hyperparameters used in LW2G: $\epsilon_{\text{task}}$, $\epsilon_{\text{pre}}$, $\phi$, and $N$. As the results in Appendix F.1 demonstrate, LW2G+baseline is not dependent on carefully chosen hyperparameters to outperform the baseline. On the contrary, LW2G+baseline consistently outperforms the baseline across arbitrary choices of $\epsilon$ and $\phi$. This highlights the robustness of LW2G with respect to hyperparameters.

ref:

[A] Consistent Prompting for Rehearsal-Free Continual Learning

[B] CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning

[C] One-stage Prompt-based Continual Learning

[D] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning

[E] Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA

[F] HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning

1. Indeed, TRGP [C] allows models learning a set of new parameters (in the form of scaling matrix Q) in each new task, which help the models flexibly adapt to new knowledge. Therefore, literally, TRGP [C] and LW2G are both expansion CL methods. Besides, the proposed HFC in this work is similar to the criteria of Trust Region in TRGP [C].

2. I agree that the good point of this work is introducing a mechanism to control the growth of the prompt pool over time. However, I have a few related concerns that the authors need to address:

• The authors constantly mention "effectively leverage cross-task knowledge". However, this seems to be a vague expectation and lacks solid reasoning.

  • Theorem 1 implies "large HFC and more severe hindrance on learning new tasks". However, this does not guarantee that if HFC is controlled, the knowledge of the tasks will support each other. The reason is that even when HFC is small, there is still a high probability that the gradient corresponding to old tasks creates an obtuse angle with the gradient of the current task. That is, updating a new task on the available prompt will likely cause the old task to be forgotten. This raises the concern about the experimental efficiency of the method - why is it that occasionally sharing prompts for many tasks (in LW2G) always yields better results than using separate prompts for each task (baselines like HiDEPrompt, Sprompt, DualPrompt, etc., )?

  • The fact that LW2G's BWTs are higher than those of baselines (as reported by the authors) cannot be evidence that LW2G facilitate positive knowledge transfer between tasks. The simple reason is that the reported BWTs are negative.

  • In addition, in the manuscript, the authors claim about forward transfer (FWT), but the rebuttal response only provides results related to backward transfer (BWT), without FWT? It seems to be unnecessary, because the numbers of BWTs are quite similar to FFM?

• The storage cost for LW2G is obviously larger than the related work PGP?

Q4.storage overhead in basis vectors.

We kindly ask the reviewer to refer to Appendix F.3 in the submitted manuscript, where we have already discussed the overhead introduced by storing the bases required in LW2G. Specifically, the extra overhead is divided into two parts: (1) a set of bases for the pre-trained knowledge space and (2) a set of bases for the old task feature space. The size of these two sets depends on the choice of $\epsilon$ during the SVD. In the following Table II, we analyze the memory introduced by Gradient Projection as $\epsilon$ varies. The "Bases" indicates the total number of bases for the two sets; "Extra Memory" represents the additional memory required. Specifically, we calculate the memory by considering each base as a tensor of length 768, stored as float32.

The results from the following Table II demonstrate that the overhead to store bases in LW2G is moderate comparing to the improvement on FAA.

Table II: Discussion of the effects of memory on IMR_INC20_TASK10.

ref:

[A]Prompt Gradient Projection for Continual Learning, ICLR 2024.

[B]Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer, NeurIPS 2022.

[C]TRGP: Trust Region Gradient Projection for Continual Learning, ICLR 2022.

[D]Rethinking Gradient Projection Continual Learning: Stability / Plasticity Feature Space Decoupling,CVPR2023

[E]Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners

[F]Divide and not forget: Ensemble of selectively trained experts in Continual Learning,ICLR2024

[G]Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters. CVPR,2024

[H] Consistent Prompting for Rehearsal-Free Continual Learning

[I] CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning

[J] One-stage Prompt-based Continual Learning

[K] InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning

[L] Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA

[M] HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning

[N]A Comprehensive Survey of Continual Learning: Theory, Method and Application, TPAMI,2024

Q2.Details about baselines HidePrompt.

We appreciate the reviewer for pointing this out. In response, we provide the following clarifications:

1.implementation Issue in HidePrompt: We kindly ask the reviewer to refer to the caption of Table 1 and Appendix E in the submitted manuscript, where we mentioned that the official code of HidePrompt has an implementation issue regarding prompt retrieval. This implementation issue caused a leakage problem, which led to the overestimation of results reported in the original paper. We asked the authors of HidePrompt for the fixed version of the code and reproduced the results.

2.Comparsion with Additional Baselines: Following the reviewers' suggestion, we have made extensive efforts over the past week to reproduce 6 other baselines for a more systematic comparison, including CPrompt[H], CODAPrompt[I], OS-Prompt[J], InfLora[K], C-Lora[L], Hide-Lora[M]. The results are provided in the Table 20 in our revised manuscript. LW2G demonstrates consistent improvements over each baseline by 0.19% to 2.02%, while significantly reducing the number of parameters (prompt sets or LoRA sets). For instance:

• In Hide-Lora, the number of LoRA sets is reduced from 10 to 2.
• In CPrompt, the number of prompt sets is reduced from 10 to 5.

Overall, the experimental results demonstrate that LW2G, as a plug-in module, exhibits excellent versatility. It can be flexibly integrated with many existing methods while achieving a balance between performance and parameter overhead.

3.Updates in the Revised Manuscript: We have added this discussion in our revised manuscript (refer to lines 1566-1647 and Appendix G).

Q3.Experiments of positive knowledge transfer.

Following [N], we provide a comparison of the BWT (Backward Transfer) results between LW2G+CPrompt and CPrompt. BWT measures the influence of learning task $j$ on previously learned tasks $i = 1,2,...,j-1$. A larger BWT indicates that learning new tasks has a stronger positive impact on previously learned tasks. The results are presented in the following Table I, which shows that LW2G can effectively facilitate positive knowledge transfer.

Table I: Results of BWT.

Thank you for the valuable comments. Below, we provide a point-to-point response to these comments and summarize the corresponding revisions. We hope these responses provide sufficient reasons to raise the score. We would be happy to address any further questions.

Q1.Comparison with relevant Work.

We would like to clarify the innovative contributions of this paper relative to [A-C] as follows:

1.The first key difference is that [B, C] are non-expansion CL methods, whereas LW2G is a plug-in method for expansion prompt-based CL methods. Expansion CL methods do not change the structure or capacity of the neural network; therefore, the total number of network parameters remains unchanged [C]. Prompt-based CL methods, which add a new set of PEFTs for each task, clearly fall under expansion CL methods. The LW2G proposed in this paper is a plug-in method specifically designed for prompt-based CL.

2.Although both LW2G and [A] introduce gradient projection into prompt-based CL, their objectives are entirely different.

• The goal of [A] is to combine gradient projection with pre-trained model-based CL methods.
• The goal of LW2G is to dynamically decide whether to learn a new prompt set for each new task based on the relationships between tasks, thus maintaining an efficient and effective prompt pool.

In current prompt-based CL methods, the size of the prompt pool is linearly related to the number of tasks, in that a new prompt set is learned for each incremental task. Some studies have already shown that learning a separate set of parameters for each incremental task hinders the potential for cross-task knowledge sharing. For example, in [G], the author claimed that "the use of independent adapters neglects the potential for inter-task knowledge sharing and cooperation, resulting in a limited representation capability and efficacy." Similarly, in [F], it was found that "the ensemble of multi-experts outperforms the best individual expert." Therefore, exploring an efficient and effective prompt pool can not only achieve a sub-linear increase in learnable parameters with respect to the number of tasks, but also effectively leverage cross-task knowledge.

3.Even though LW2G is clearly different from [A], we have also compared LW2G with [A], which was included in the original manuscript. We kindly ask the reviewer to refer to Appendix F.4.

• Regarding whether TRGP is an expansion/non-expansion CL method:

  • The code you showed is not comprehensive. Through the link you provided, we can see that during training, for each task, they initialize a "memory" (Lines 843-851), which will store the trained/computed value of scaling matrix Q, which is specific for each task (Lines 956-962). When in inference, they retrieve the corresponding value of Q, for each task (Lines 987, 995), to predict the final result.

  • Therefore, it is evident that TRGP is an expansion CL method, which is similar to prompt-based CL method (LW2G), which do not directly change the original architecture, but inject into model the specific components which aid in improving model performance. And thus, the contributions/novelties of HFC and also this work seems to be not significant.

• Regarding Theorem 1 and "knowledge transfer" effect, I still concern that the paper does not ensure the positive knowledge transfer between task.

  • Normally, when using gradient projection, the performance of the following task will be hinder to some extent, compare to training independently (i.e, using distinct prompt set for tasks).

  • Therefore, the superior performance over the baselines as observe can result from having fewer prompts means the probability of selecting the correct prompt when testing is higher, leading to higher performance. Not because of knowledge transfer.

  • In addition, negative BWT means negative backward transfer. Achieving higher BWT, but negative ---> Only means that you prevent forgetting better, cannot be sure about "promote mutual assistance between tasks".

Overall, the reviewer has made a good effort to address the concerns raised. However, the responses provided do not fully satisfy the concerns, especially regarding the novelty of the method as well as the author’s inaccurate claims regarding the method’s effectiveness. Finally, I would like not to change my rating to a positive one.

1. The reason I discussed the essence of TRGP (not just the claims in their paper) is that I find there are quite many similarities between your method and TRGP, which makes your method seem incremental.

2. Regarding your extended experiment you have provided, please let me know how you apply LW2G into DualP. Because this method has 2 kinds of prompts, including G-prompts (shared), and E-prompts (task-specific). This is important to evaluate the role of knowledge transfer in model performance.

Thank you for your response.

I am afraid that the experimental results you provided are not convincing for me. I will point out the reasons below - regarding the knowledge transferability between tasks of this method:

• Regarding Backward transfer (BWT): It is very hard to transfer knowledge from new task to previous ones. The reason is that when you apply gradient projection, the behavior of model w.r.t old tasks will be almost unchanged.

• Regarding Forward transfer (FWT):

  • In the post (https://openreview.net/forum?id=QZuZmfLLRG&noteId=OjthBJVEZs), you agreed with me that "Gradient Projection Hinders Learning of New Tasks".
  • Besides, in another post (https://openreview.net/forum?id=QZuZmfLLRG&noteId=aeCyGhrh3z), you admitted that "the use of the term "forward knowledge transfer" in the submitted manuscript was imprecise".

  ----> These arguments mean there is no positive FWT

Based on these, the experimental results you provided, and also your state about "cross task knowledge transfer" are unbelievable. It seems to me that the authors are trying to polish their paper, but it is not reasonable.

We sincerely appreciate the reviewers for dedicating their valuable time and continuous effort to reviewing and engaging in the discussions of this paper.

Below, we summarize Reviewer eZ2K’s main concerns during the rebuttal process and provide a point-by-point response.

Main Concern1: Is LW2G very similar to TRGP or other gradient projection CL methods, thus making the contribution of LW2G incremental?

Reply: No.

LW2G is fundamentally different from previous gradient projection CL methods. The evidence is as follows:

1.Different Types of Continual Learning Methods.

Existing gradient projection CL methods focus on fixed models and non-expansion CL methods, such as GPM and TRGP. In contrast, LW2G focus on expansion CL methods (prompt-based CL methods). Specifically, LW2G is the first to explore the relationship and combination of gradient projection CL methods with expansion CL methods. It examines the degree of learning hindrance caused by strict orthogonality conditions (via the proposed HFC metric). Current prompt-based CL methods learn a task-specific prompt/adapter/lora module for each incremental task (e.g., e-prompt in DualPrompt), which hinders cross-task knowledge facilitation and leads to a low task-specific module selection accuracy (PRA problem). The core contribution of LW2G is the design of a dynamic mechanism to control the growth of the prompt set, which brings two key advantages:

• Prompt effectiveness: "Gradually consolidating knowledge from multiple tasks into one prompt set" is more effective than using "task-specific prompt sets," thus promoting cross-task knowledge transfer.
• Prompt efficiency: "Fewer prompt sets" improve PRA and reduce parameter overhead.

Therefore, LW2G enables the formation of an efficient and effective prompt pool for expansion CL methods(prompt/lora/adapter-based CL methods)

2.Different Continual Learning Scenarios.

Existing gradient projection CL methods mainly focus on task-incremental learning (with task IDs provided during testing). However, current prompt-based CL methods consider the more challenging class-incremental learning scenario (without task IDs during testing). LW2G is designed as a plug-in module for prompt-based CL methods, addressing this more difficult scenario.

3.Some details.

Regarding the reviewer's mention of the TRPG official code ((Lines 956-962)), we conducted a line-by-line inspection, and the fact is that the scaling matrix Q does not expand with the task, but rather, it is a fixed-size matrix Q that learns from scratch for each task. This differs from the definition of expansion CL methods and therefore belongs to non-expansion CL methods.

Regarding the reviewer's mention of the TRPG official code ((Lines 987, 995)), we also performed a line-by-line check. The fact is that since TRGP is designed for task-incremental learning, during inference, the scaling matrix Q is retrieved with the task-id. Therefore, if TRGP is used to address class-incremental learning, it will also face the issue of scaling matrix Q retrieval accuracy (PRA problem). In this case, the scaling matrix Q is similar to the prompt set in prompt-based CL methods, which learns specifically for each task. Since the accuracy of selecting the scaling matrix Q also affects the performance of TRGP, LW2G+TRGP can also be used to control the growth of the scaling matrix Q in TRGP.

In summary, LW2G has a wider range of applications than TRGP and can also serve as an optional enhancement choice when TRGP is used in class-incremental learning scenarios.