Class-Incremental Learning with Parameter-Efficient Cross-Task Prompts

Thank you for your detailed review of our paper and constructive feedback! We have revised our paper and addressed your concerns. We summarize below the changes we made.

Q1. "A potential data leakage concern: as the pre-trained model has already been exposed to ImageNet21k during its initial training, which may violate the foundational principles of class incremental learning. Therefore, A missing baseline: I strongly recommend that the author includes performance metrics without the prompt, thereby delineating the actual gains derived from this approach."

A1: PTM-based CIL methods do exhibit data leakage in Usual CIL benchmarks, as shown in [4]. However, [4] also introduced four new benchmarks where Data leakage phenomenon does not exist. Notably, [4] introduces a novel baseline for PTM-based CIL methods called SimpleCIL. SimpleCIL performs directly on incremental tasks without any prompting or fine-tuning. The corresponding results can be found in the third row of the tables below.

The total results can be found above. Specifically, CIFAR, CUB, and ImageNet-R are widely adopted benchmark datasets for CIL. However, due to the data overlap between ImageNet-based benchmarks and the pre-trained dataset, ImageNet is unsuitable for evaluating PTM-based CIL methods. Therefore, we follow [4] to evaluate our method on four new benchmarks: ImageNet-A, ObjectNet, OmniBenchmark, and VTAB. These new benchmarks not only have no overlap with ImageNet but also exhibit a large domain gap between tasks, making it challenging for PTMs to handle.

Notably, despite training on an extremely large-scale dataset, SimpleCIL fails to achieve high accuracy on certain tasks, such as ImageNet-R and ImageNet-A. In contrast, PECTP consistently outperforms SimpleCIL on seven benchmarks, especially on those without overlap.

Thank you for your diligent review of our paper. With your valuable comments and suggestions, especially for the Questions about paper revision, we have significantly improved the clarity, fluency, and details of our paper. We sincerely appreciate you and your advice.

Q3. "According to Table 3, utilizing both IPG and OPG has the good synergy to enhance the performance. To the best of my understanding, the roles of IPG and OPG are to prevent forgetting knowledge from the previous task and pretrained model, respectively. "

A3: We agree with the reviewer's perspective that there exists two catastrophic forgetting issues in PTM-based methods:

(a) forgetting knowledge of previously learned incremental tasks and,

(b) forgetting knowledge obtained from the pretraining stage of PTM.

However, we would like to clarify that the PRM proposed in our paper aims to minimize the former issue by reducing it at two granularity, specifically categorized as OPG and IPG. As for the latter one, we follows the approach in [4], utilizing 'feature fusion module' to mitigate the impact of continual learning on the knowledge gained during pretraining.

Finally, we identify a mistake where we mistakenly quoted the results of Baseline on IN-A Inc10 from [4]. It has been corrected in the latest version of the paper, and we have included additional results from ablation experiments on CUB and ImageNet-R. Here is a summary of the results:

The experimental results demonstrate that IPG exerts a stronger constraint on the prompt itself. When the domain gap between incremental learning tasks is small, such as in CIFAR and ImageNet-R, it leads to a greater improvement in performance. Additionally, the results in the last row indicate that OPG and IPG are complementary. Using them together can significantly reduce the forgetting of knowledge acquired from previously learned incremental tasks.

Q4. "The results in large number of tasks setup would be helpful to validate the effectiveness of their method."

A4: Thank you for your suggestions. Learning tasks with long sequences have consistently been a challenging research area in CIL. We followed [5] to partition tasks based on the number of classes, and evaluated our approach on different configurations of ImageNet-R: 20 tasks (10 classes/task), 40 tasks (5 classes/task), 50 tasks (4 classes/task), and 100 tasks (2 classes/task).

The specific implementation results demonstrate the consistent superiority of the PECTP across task sequences of varying lengths. However, we also observed significant performance degradation for L2P, DualPrompt, and PECTP when learning extremely long task sequences. For instance, when continuously learning 100 tasks, PECTP only achieved an 8% improvement over SimpleCIL[4], despite incurring a substantial increase in computational overhead. We acknowledge this as an area for future exploration and investigation. Once again, thank you for bringing this issue to our attention.

If you have any further questions, we are willing to clarify any misunderstandings and address your inquiries to the best of our ability.

Refs:

[1] Learning to Prompt for Continual Learning, 2022 CVPR

[2] DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning, 2022 ECCV

[3] CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning, 2023 CVPR

[4] Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need, 2023 Arxiv

[5] iCaRL: Incremental Classifier and Representation Learning, 2017 CVPR

[6] Visual Prompt Tuning, 2022 ECCV

Thank you for the review, comments, and constructive feedback! We are encouraged that you found our idea promising, our method effective and our writing clear. We provide answers to your comments and questions below.

Q1. "While L2P has the prompt pool, the number of prompts does not depend on the number of tasks. As such, L2P should be prompt-fixed, not prompt-extending. "

A1: We have reviewed the description of L2P [1] in the original paper and found an error in our understanding. We categorize L2P as a prompt-fix method and will make the corrections in the latest version.

Q2. "Results on comparison of PECTP and the other baselines with same prompt number."

A2: We would like to provide a detailed explanation of the specific calculation method for the number of prompts in PTM-based CIL methods.

Firstly, in existing PTM-based CIL methods, the description of 'prompt' refers to a set of prompts instead of a single prompt. For example, in L2P, $P_{i} \in \mathbb R^{L_{P}\times D}$, where $L_{P}$ is the number of single prompts, and each $P_{i}$ is stored in a prompt pool. Therefore, for L2P, the total number of prompts is $L_{P} \times$ the number of prompts .

Furthermore, current PTM-based CIL methods follow VPT[6] for prompt handling, which has two variants: VPT-Deep and VPT-shallow. For instance, in DualPrompt[2], e-prompts: $e_{i} \in \mathbb R^{L_{e}\times D}$ are inserted into the 3-5 layers of the VIT encoder, while g-prompt: $g_{j} \in \mathbb R^{L_{g}\times D}$ is inserted into the 1-2 layers of the VIT encoder. Therefore, for DualPrompt, the total number of prompts is: the number of e-prompts $\times$ $L_{e}\times$ inserted layers + the number of g-prompts $\times$ $L_{g}\times$ inserted layers.

Finally, we strictly reimplement the baseline method according to the description in the original paper [1-3]. Additionally, we agree with the reviewer's perspective that the performance of PTM-based methods depends significantly on the number of trainable parameters. Therefore, maintaining an equal number of parameters for comparison will provide meaningful results. Considering that PECTP introduces fewer trainable parameters compared to the baseline methods, we have included three additional variants: PECTP-L, PECTP-D, and PECTP-C, each roughly equivalent in the number of trainable parameters to L2P, DualPrompt, and CODAPrompt[3], respectively. Here is a summary of the results:

The results illustrate that the proposed PECTP demonstrates a significant performance improvement, even with a much smaller parameters compared to baseline methods (DualPrompt, L2P, and CODA are 10, 44, and 72 times larger than PECTP, respectively). Furthermore, as the number of prompts in PECTP increases the performance continues to improve and approaches the UPPER bound. This further emphasizes the superiority of our approach.

We wanted to follow up and gently ask if we are able to address your concerns in our response. As the discussion period is nearing an end, we would appreciate any updates or further questions you may have, and if not, we hope you might consider raising your score. We thank you for your time in advance!

Thank you for the review, comments, and constructive feedback! We are encouraged that you found our method effective and our writing clear. We provide answers to your comments and questions below.

Q1. "What is the intuition behind limiting the update of prompts?"

A1: The intuition behind limiting the update of prompts is to maintain knowledge related to previous tasks within the prompts. In existing PTM-based CIL methods [1-5], task-specific knowledge is encoded into prompts. The task-specific knowledge is associated with each incremental tasks, such as visual features or task information like the task ID, and is updated continually for each incremental task. Due to considerations of storage and computational overhead, we adopt prompt-fix methods that use only a fixed set of prompts.

However, if prompts are continually trained on each incremental task using cross-entropy loss without any constraints, prompts are prone to losing knowledge of previous tasks. For instance, training a set of prompts $\mathcal P$ on task 1 results in $\mathcal P_{1}$, which acquires knowledge about task 1. This implies that utilizing $\mathcal P_{1}$ effectively guides the PTM in classifying the first task. However, when we continue training $\mathcal P_{1}$ on task 2, resulting in $\mathcal P_{2}$, the parameters of $\mathcal P_{2}$ have changed compared to $\mathcal P_{1}$. This means that $\mathcal P_{2}$ cannot effectively guide the PTM in correctly classifying task 1. Therefore, without adding additional constraints during the training，$\mathcal P_{i}$ is prone to losing information about previous tasks: $1,2,...,i-1$.

To verify this point, we provide an additional set of experimental results, where PlainCIL represents continuously learning new task knowledge without adding any constraints to prompts. SimpleCIL [4] represents directly using a PTM to address incremental learning tasks without any prompting or finetuning. Here is a summary of the results:

Q2. "Wouldn't limiting the update also limits the ability of learning new tasks?"

A2: We also concur with the reviewer's perspective that constraining the update of prompts may impede learning for new tasks, consequently limiting the acquisition of new knowledge. However, even in usual regularization-based CIL methods [7], there exists a trade-off between new and old knowledge. Specifically, in usual classic incremental learning approaches, there exists weakening the forgetting of old knowledge by constraining learning for new tasks.

In summary, we are actively seeking a more reasonable balance between new and old knowledge. Additionally, although the motivation behind the proposed method is to efficiently utilize prompt parameters to adapt to highly memory-constrained scenarios, our experimental results indicate that even for some methods with parameters far exceeding PECTP, the proposed method remains comparable. For specific results, please refer to https://openreview.net/forum?id=4lqo5Jwfnq&noteId=Iv3lvYIpt9.

Dear Reviewer 2DHh,

We thank the reviewer for their engagement in the rebuttal process. As the discussion period is nearing an end, we would appreciate any updates or further questions you may have, and if not, we hope you might consider raising your score. We thank you for your time in advance!

Best,

Paper 2478 Authors