Prompt Gradient Projection for Continual Learning

For Weakness 3 about ''the scope of the proposed PGP method'':

To further validate our method on the vision-language model, we introduce it to the CLIP model. The experimental setting is that, in the vision side, we only set a single trainable image prompt shared by all tasks. As for the text side, we follow the operation as [3], we set trainable text prompt for each class, which is only trained at the corresponding task. We conduct our experiments on two datasets: 10-Split-CIFAR100 and 10-Split-TinyImageNet, as shown in Table 3. Besides that, we also verify the effectiveness of our method under the task incremental setting, as shown in Table 4. Results show that our method has improved the performance a lot for all the above settings, which we think proves that our method is also useful in the vision-language models and further enlarges the scope of our method.

We have noticed that, compared with your mentioned method, our method does not exhibit comparable accuracy performance, despite conducting experiments on entirely different datasets with [4]. The reasons are as follows: Firstly, in comparison to its complex method and network structure, we employ a very simple setting, utilizing only a single image prompt, which restricts the performance of our method. Secondly, it employs the data rehearsal method to further enhance performance. However, our method is rehearsal-free.

In our design, we want to exclude all of the external influential factors and only verify the PGP method. Thus, we choose the simplest experiment setting of a single image prompt. Experiments show that our method works well.

Table 3: Comparison to CLIP model without/with gradient projection method under class incremental setting (without task identifier in the test phase).

Table 4: Comparison to CLIP model without/with gradient projection method under task incremental setting (with task identifier in the test phase).

As for the latest methods, such as LoRA, K-Adapter, and other parameter-expansion methods. Since they mainly focus on fine-tuning a large model, whose goal is to adapt the model to the down-stream tasks, we are afraid this is beyond the scope of this paper, which aims to learn new concepts while not forgetting the prompts. Also, the combination of gradient projection and fine-tuning methods would be interesting. We would like to study this in the future.

[3] Zhou, Kaiyang, et al. "Learning to prompt for vision-language models." International Journal of Computer Vision 130.9 (2022): 2337-2348.

[4] Zhou, Da-Wei et al. “Learning without Forgetting for Vision-Language Models.” ArXiv abs/2305.19270 (2023): n. pag.

To answer your concerns about our method of the limited efficacy and the scope of the reported experimental baselines, we conclude our method as follows:

(1) We are the first one to propose the preventing forgetting method from the view of prompt gradient. As continual learning is not a fine-tuning task, the data is coming as a sequence of tasks, and data from previous tasks would not be seen in future tasks, which brings a heavy decline in performance of models. Thus, how to overcome catastrophic forgetting is the prime aim of continual learning. To the best of our known, we are the first one to deduct the orthogonal condition to prevent forgetting for prompt-based methods and implement the gradient projection on prompt. We believe that our method owns a huge potential, which is still uncovered. In future works, we will further explore the potential of our method to achieve better results, and also release the code.

(2) We conduct our method on multiple and various models, e,g, ViT, CLIP, distinct prompt-tuning paradigms, e,g, prompt-tuning, prefix-tuning, different pre-trained backbones, e.g. ImageNet-1K, SAM, and DINO, under three continual learning settings: class incremental learning, task incremental learning, and online class incremental learning. All experiments show that our method can effectively reduce forgetting and improve the average accuracy. Besides that, our method is a plug-in, and can serve in different prompt-based continual learning methods. Especially nowadays, as the tendency of large models become hotter and hotter, we think our method can have a promising application future.

(3) We prove that our method has little time and memory consumption through experiments. Besides that, our method does out introduce other models or save any exemplars. Based on these facts, we believe that our method can have advantages in training speed and memory, which could be beneficial when transferring our method to large vision foundation models (VFMs).

(4) Finally, we reveal the mechanism of trade-off between the plasticity and stability in the perspective of prompt gradient, and well balance the plasticity and stability of model by adjusting the threshold \epsilon, which brings a new insight as the above trade-off dilemma is a core problem in continual learning.

For minor comments about ''Definition of V_t and U_t missing'':

Thanks for your valuable suggestions. We have added the missing definition of V_t and U_t.

For Weakness 2 about ''the training time consumption'':

To answer your question, we reconducted experiments about L2P with data rehearsal on the 10-Split-CIFAR100 dataset. we randomly sampled the exemplars and achieved a result akin to the report results in the paper. Then, we measured the time consumption of our reproduced L2P-R and compared it with our method as shown in Table 2. The time consumption of our method is 0.756 h and the time consumption of L2P-R is 0.787 h. The reason why our method is faster than L2P-R, we hold the view is that L2P-R needs to train the extra rehearsal data in every epoch, which spends a lot of time. The more rehearsal data is saved, the more training time is consumed. However, our method is rehearsal-free.

Table 2: Time Complexity Comparison, Accuracy, Forgetting between our approach and L2P, L2P-R. L2P-R indicates L2P with data rehearsal.

To sum up, our method not only exhibits superior performance and requires less storage memory but also consumes less training time.

Dear reviewer LEcv, thanks for your valuable suggestions. Here are our responses:

For Weakness 1 about ''use a backbone trained on a different dataset and use other CL datasets'':

As your suggestion, we re-evaluated the effectiveness of our method by extending two distinct pre-trained models, namely ViT-DINO and ViT-SAM. The results are shown in the Table below. Additionally, we tested our method on the CUB-200 dataset based on three ViT pre-trained backbones: ImageNet-1k, DINO, and SAM, further validating the effectiveness of our method on non-ImageNet datasets.

Table 1: Comparison to distinct pretrained-dataset backbones with/without prompt gradient projection.

[1] DINO pre-trained models (https://arxiv.org/abs/2104.14294)

[2] SAM pre-trained models (https://arxiv.org/abs/2106.01548)

For Question 4 about ''Figure 3 Clarification'':

Our explanation for this issue is as follows: Firstly, as you can see, Figure 3 is set in the background of online class-incremental learning, strictly limiting each image appearing only once. That is to say, each task is only trained for one epoch. The limited number of epochs results in fewer gradient update processes and fewer gradient projection processes, leading to insufficient gradient projection. Here, we emphasize that it is precisely due to the insufficient gradient projection that the phenomenon of "Dual-Prompt appearing to be shifted up (accuracy) or down (forgetting) by a certain scale" occurs.

Our rationale is that each projection serves as a correction to the direction of gradient, thereby reducing forgetting. The more projections, the more sufficient the correction, and the more stable and effective the mitigation of forgetting. Conversely, fewer projections lead to insufficient correction, causing the mitigation of forgetting to be highly unstable, oscillating randomly between good and bad states of suppressing forgetting. The term "a certain scale" represents this manifestation of random oscillation. To demonstrate our conclusion, we conducted experiments under two settings: epoch=1 and epoch=3 on the same 10-Split-TinyImageNet dataset, as shown in Table 5 below:

The "Diff" in the Table represents the difference between FOR (forgetting) and ACC (accuracy) after projection and before projection. It can be observed that, at epoch=1, the differences in ACC and FOR across different tasks fluctuate more noticeably, aligning with the oscillating process described in the theoretical explanation, showing an oscillating pattern between good and bad states. However, at epoch=3, the differences in ACC and FOR increase steadily with the increasing number of tasks, and the variations become highly stable.

Table 5: Comparison to the differences in ACC(Diff_ACC), the differences in FOR(Diff_FOR) at different tasks under settings of distinct epoch.

For Question 1 about ''Reference Request'':

We have inserted the following two pertinent references into the statement location as advised.

[1] Saha, Gobinda, Isha Garg, and Kaushik Roy. "Gradient Projection Memory for Continual Learning." International Conference on Learning Representations. 2020.

[2] LiLin, Sen, et al. "TRGP: Trust Region Gradient Projection for Continual Learning." International Conference on Learning Representations. 2021.

For Weakness 4 about ''Terminology and Consistency'':

In the revised version, we have replaced V_t, V_1, and V_2 with new symbolic representations and added detailed explanatory text to ensure consistency with the preceding context and eliminate any potential confusion.

For Weakness 5 about ''Notation Clarification'':

Thank you for your question. It has been addressed along with the previous one.

For Weakness 3 about ''Proof Presentation'':

We have expanded the content in the last two paragraphs of page 5, providing a more compact connection with the preceding and following sections. This adjustment is intended to enhance the clarity and comprehensibility of the paper.

For Weakness 2 about ''Figure 2'':

We have enhanced the clarity of Figure 2 by providing a more detailed explanation in the caption, elucidating the content and essence of our method, and further improving the comprehensibility of the image.

Dear reviewer epe6, thanks for your valuable suggestions. Here are our responses:

For Weakness 1 about ''Readability and Clarity'':

We have rewritten the abstract and the last paragraph of page 3, and further revised several sections of the paper where language ambiguity exists. We thank your suggestion which makes our paper clearer and more readable than before. The following are two examples of revised version:

For Abstract

Before revised: Prompt-tuning has demonstrated impressive performance in continual learning by querying relevant prompts for novel classes training.

After revised: Prompt-tuning has demonstrated impressive performance in continual learning by querying relevant prompts for each input instance, which can avoid the introduction of task identifier.

Before revised: In PGP, we deduce the orthogonal condition for prompt gradient via the self-attention mechanism in vision-transformer.

After revised: In PGP, we deduce that reaching the orthogonal condition for prompt gradient can effectively prevent forgetting via the self-attention mechanism in vision-transformer.

For the last paragraph of page 3

Before revised: one limitation of gradient projection methods is which is only applicable for task incremental learning but will fail in the class-incremental inference, as the projected gradient is a strict constraint for novel classes training.

After revised: one limitation of gradient projection methods fail in the class-incremental inference is that the projected gradient needs task identifier to find relevant network parameters.

For Question 3 about ''Performance vs. Time Complexity Trade-off'':

Thanks for your valuable suggestions.

(1) Discussion of the promising performance of the proposed algorithm and its associated time complexity:

i) For Time: In the answer to time complexity, we have shown detailed information about time consumption in our method. In fact, both the time consumption on calculation of projection matrix and gradient projection process is little, only adding 2% extra training time on the original baseline.

ii) For Performance: Our approach demonstrates a high generalization ability, could be combined with the almost prompt-based continual learning methods. Besides that, from the perspective of gradient projection, we further realize a well-done balance of the model's plasticity and stability. This balance is crucial for continual learning, causing a favorable balance often results in enhanced performance.

(2) Explaining why this algorithm should be prioritized over other existing algorithms:

To the best of our knowledge, we are the first to propose the gradient projection for reducing forgetting in the context of prompt-tuning. Note that the tendency of vision foundation model has recently caused significant changes in computer vision. Our approach first studies the gradient of prompt-learning and we believe related applications, like fine-tuning a VFM, could benefit from our paper. We have additionally tested the large vision foundation model for continual learning, and the results turn out our approach is also effective in this scenario. Please kindly refer to Table 2. Here, for the vision side, we set a single trainable image prompt shared by each task. For the text side, we follow the operation with [3], we set trainable text prompt for each class, only trained at the related task.

Table 2: Comparison to CLIP model without/with gradient projection method on 10-Split-CIFAR100 with class incremental setting (without task identifier in the test phase).

Here, we also compare our method with other two non-gradient and prompt-based methods, one is data rehearsal, and the other is a paper titled "Introducing Language Guidance in Prompt-based Continual Learning", referred to as LGCL, presented at ICCV’23 [4]. LGCL is slightly lower than our method but with complicated designs, as shown in Table 3. Compared with baseline and IGCL, we have found we own the following three advantages.

1. For memory, we only need to store a tiny projection matrix. While for data rehearsal, saving exemplars would spend far more memory space than us. For LGCL, although it is a rehearsal-free method, it has to introduce another network, a pre-trained text encoder: CLIP L/14 into the method, which also needs to occupy a significant amount of memory space.

2. For time, extra time in our method, mainly costs on calculation of projection matrix and gradient projection process, which is shorter than the retraining process of exemplars in the data rehearsal method, explained in the answer of time complexity. For LGCL, due to the non-open source code, we cannot reproduce the experiments. However, in the paper, it adds an additional text encoding process and adopts extra loss functions to train the model. Therefore, compared to LGCL, our method also exhibits shorter time complexity.

3. Our method could serve as a plug-in to reduce forgetting of the models in almost all of the prompt-based continual learning methods. With the hot tendency of the large models nowadays, our method could be transferred to the large Vision Foundation Models (VFMs) and have a promising application future in the fine-tuning paradigm.

Table 3: Comparison to Accuracy, Forgetting of L2P-LGCL and L2P-PGP.

[3] Zhou, Kaiyang, et al. "Learning to prompt for vision-language models." International Journal of Computer Vision 130.9 (2022): 2337-2348.

[4] Khan, Muhammad Gul Zain Ali, et al. "Introducing Language Guidance in Prompt-based Continual Learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

For Question 2 about ''Time Complexity'':

Thanks for pointing out our omission.

Under the same hardware platform, experimental setting, and backbone, we compared our method with the original L2P and L2P with data rehearsal on the 10-Split-CIFAR100 dataset. As L2P does not mention detailed information about how to make data rehearsal, we adopted a simple data rehearsal process which randomly sampled a set of exemplars and achieved a result similar to the reported ones.

The results indicate that, in comparison with the original L2P, our approach reduces forgetting by 1.04%, improves accuracy by 0.57%, and extends training time by 0.016 hours. In comparison with L2P with data rehearsal, our method not only reduces forgetting by 2.13%, improves accuracy by 0.13%, but also shortens training time by 0.031 hours.

Table 1: Time Complexity Comparison between our approach and L2P, L2P-R. L2P-R indicates L2P with data rehearsal.

For further analysis, we use “time.time()” function to coarse read the time consumption in each sub-process.

For data rehearsal, it concludes in three steps: 1. Collect exemplars (4*10-5~26*10-5 s), which can be ignored. 2. Store exemplars (Simplified in our reimplement method). 3. Exemplars join in the training process. We mainly discuss the time consumption in the third step: In the batch size of 20, one iteration that only exemplars take part in costs a mean time of: 0.196 s. Here, we set epochs as 5 in the experiment. Thus, on the 10-Split-CIFAR100, training with exemplars needs extra (5+10+15+20+25+30+35+40+45) * 5 = 1125 iterations and costs 1125 * 0.196 = 220.50 s. To sum up, theoretical analysis shows that data rehearsal would add an extra 220.50 s in the training process.

For our method, it also concludes three steps: 1. Collect samples (10-6*33*5*10 s), which can be ignored. 2. Calculate the projection matrix, consumption time is (10.65 + 8.78 + 9.42 + 8.72 + 8.68 + 8.68 + 8.53 + 8.81 + 8.72 + 8.69 = 81 s). 3. Gradient projection process, consumption time is (2.79 + 3.22 + 2.95 + 2.73 + 3.11 + 3.28 + 2.72 + 3.01 + 2.78 + 2.98 = 29.57 s). To sum up, theoretical analysis shows that gradient projection would add an extra 81 + 29.57 = 110.57 s in the training process.

Notice that, we set 5 epochs in the training process. If we increase the epoch number, the consumption time of L2P-R would also increase due to more iterations of training exemplars. However, our method is unrelated to the epoch number and can be free of increasing epochs.

Thank you for your diligent review of our paper. With your valuable comments and suggestions, especially for the Questions about paper revision, we have significantly improved the clarity, fluency, and details of our paper. We sincerely appreciate you and your advice.

For Question 2 about ''modified with gradients for the classifier attached to the backbone'':

In fact, in prompt-tuning CL, all follow a special configuration in the classifier. The details are as follows: This framework employs a unified classifier, using the 10-Split-CIFAR-100 dataset as an example. In the first task (including classes from 0 to 9), we train the classifier, and obtain the weights on dimensions of 0-9. In the second task (including classes from 10 to 19), we freeze the weights on dimensions of 0-9 in the classifier and continue to train the weights on dimensions of 10-19. We do the same thing in the subsequent tasks. It is obvious that, in this setup, the classifier weights on dimensions of the old classes are not updated (frozen). Therefore, no gradients are involved, eliminating the need for gradient update.

Therefore, prompt-tuning CL, eliminates the need for any projection or alteration of gradients in the classifier. We have some new results when adopting prototype-based classifiers, but this is beyond the scope of this paper.

For Question 1 about ''Figure 1 is a bit unintuitive as normally'':

We have reconfigured Figure 1 according to your suggestion and made it more intuitive. We appreciate your valuable feedback which has enhanced the quality of our paper.

Dear reviewer Q1LH, thanks for your valuable suggestions. Here are our responses:

For Weakness 1 about ''The comparison of prompt-tuning based CL methods with other CL methods'':

Thanks for the advice. The backbone is actually pre-trained on ImageNet, and therefore it is somewhat unfair to compare with non-prompt-tuning based CL methods.

However, the tendency of the pre-trained large models, or vision foundation model has recently caused significant changes in computer vision. This paper focuses on reducing the forgetting of prompt-tuning paradigm, which, we believe, would bring new insights for continual learning. We have additionally tested a lot of other baselines, especially including large vision foundation model, and the results turn out our approach is also effective in this scenario. Please kindly refer to Table 1. Here, the methods in Table 1 are from 1) Replacing the pre-trained backbone of ImageNet-1K in L2P and DualPrompt with SAM and DINO. 2) Introducing prompt gradient projection method to CLIP model. For the vision side, we set a single trainable image prompt shared by each task. For the text side, following the operation with [1], we set trainable text prompt for each class, which is only trained at the corresponding task.

For the unfair comparison, please kindly note that: 1), in our baseline (L2P and DualPrompt [2, 3]), they have already compared their methods (adopting pre-trained backbones) with non-prompt-tuning-based CL methods (e,g. LwF, EWC, BiC, DER). We just simply follow their existing practice. We would denote the backbone comparison in our revised paper. 2), Our goal is to analyze the gradient space for prompt-tuning-based CL methods and propose a gradient projection method that effectively reduces forgetting. Therefore, our focus is mainly on observing performance improvements compared to the baselines (L2P and DualPrompt), non-prompt-tuning based CL methods are used as references.

Table 1: Comparison to distinct baselines with/without prompt gradient projection method on 10-Split-CIFAR100.

[1] Zhou, Kaiyang, et al. "Learning to prompt for vision-language models." International Journal of Computer Vision 130.9 (2022): 2337-2348.

[2] Wang, Zifeng, et al. "Learning to prompt for continual learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Wang, Zifeng, et al. "Dualprompt: Complementary prompting for rehearsal-free continual learning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[4] DINO pre-trained models (https://arxiv.org/abs/2104.14294).

[5] SAM pre-trained models (https://arxiv.org/abs/2106.01548).