One-stage Prompt-based Continual Learning

We appreciate your positive feedback on our work. Please check our response to your questions and concerns.

Q1: Does this method still work without using supervised pretraining? For instance, what if replacing the supervised pretraining model with an unsupervised pretraining model (e.g. DINO [1])?

A1: Thank you for the valuable feedback. We think the reviewer's suggestion is interesting and important for PCL works. In response to this suggestion, we use DINO pre-trained weights instead of ImageNet-1k pre-trained weights, and compare the performance across ER, LwF, and PCL works. We set the other experimental settings as identical.

The non-PCL methods (ER and LwF) demonstrate similar or even superior performance compared to DualPropmt and L2P. This contrasts with the results obtained using the ImageNet-1k pre-trained model, where PCL methods outperformed non-PCL methods. This observation suggests that the backbone model plays a crucial role in PCL. However, CodaPrompt and our OS-Prompt outperform the other methods by a significant margin. This indicates that our method continues to perform well even without utilizing supervised pertaining.

We have included the experiments and analysis related to the above observations in the Appendix.

Q2: Lack of clarity in experimental results. In particular, Figure 2 is not very clear. Is the token embedding distance depicted in this Figure related to the current task? What are the distances between token embeddings for tasks 1 to t-1 when task t is learned?

A2: Thanks for the insightful suggestion. In response to the reviewer's feedback, we have restructured Fig. 2 in our manuscript. This figure depicts the measurement of distances between token embeddings of previous tasks when a new task is learned. We conducted experiments on 9 pairs, assessing the distance between token embeddings of task 1 and those after tasks {3, 5, 7, 9}; the distance between token embeddings of tasks 3 and those after tasks {5, 7, 9}; and the distance between token embeddings of tasks 5 and those after tasks {7, 9}. The results reveal that token embeddings from early layers show minimal changes during training. Consequently, utilizing token embeddings from these earlier layers (layers 1~5) provides a stable and consistent representation throughout the continual learning stages.

Thank you for your efforts in reviewing our article and providing constructive feedback. We’d like to reply to your concerns in detail.

Q1: Potential Scalability Concerns. The paper does not provide insights into how this method scales with larger, more complex datasets.

A1: We appreciate the insightful suggestion from the reviewer. In response to their input, we conducted experiments on DomainNet [R1], a large-scale domain adaptation dataset consisting of around 0.6 million images distributed across 345 categories. Each domain involves 40,000 images for training and 8,000 images for testing. Our continual domain adaptation task was created using five diverse domains from DomainNet: Clipart → Real → Infograph → Sketch → Painting. The experimental methodology closely followed the protocol established in prior PCL research. The outcomes of these experiments are presented in the table below.

Our method (OS-prompt and OS-propmt++) achieves comparable performance compared to the previous PCL methods with ~50% inference computational cost. These results affirm the practical viability of our approach, indicating its applicability in real-world scenarios. We have added the results in the Appendix.

[R1] Peng et al. Moment matching for multi-source domain adaptation. ICCV19

Q2: Lack of Exploration on QR Loss Limitations. A deeper dive into the trade-offs associated with the QR loss would provide a more balanced view of its utility.

A2: Thank you for your suggestion. During the training of OS-Prompt++, in comparison to OS-Prompt, the inclusion of QR loss does enhance accuracy. However, QR loss requires a reference ViT feedforward step (which is different ViT from the backbone ViT), resulting in additional training computational cost compared to OS-Prompt. It's worth highlighting that while OS-Prompt++ has a high training computational cost compared to OS-Prompt, it remains consistent with the training cost of prior PCL works (refer to Table 3).

This computational efficiency during training becomes important in the context of addressing the online continual learning problem [R2]. This problem assumes the model needs rapid training on the given streaming data. If the training cost requires a high computational cost, the stream does not wait for the model to complete training before revealing the next data for predictions, leading to performance degradation. Therefore, an analysis of the training efficiency of OS-Prompt++ is required.

To address this issue, in Fig. 6, we provide the trade-off between FLOPs and accuracy within a reference ViT used for QR Loss. In our OS-Prompt++ default setting, we extract the reference prompt query r from the final layer of the reference ViT. This requires additional training complexity compared to OS-Prompt. To improve efficiency, instead of using the final layer’s [CLS] token embedding for prompt query r, we utilize the intermediate token embeddings from layers deeper than 5 within the reference ViT. The results show there is an accuracy and training efficiency trade-off in our QR Loss design.

More importantly, as per the reviewer's suggestion, we have added content in the Conclusion section to point out the potential limitations of the proposed method.

[R2] Real-Time Evaluation in Online Continual Learning: A New Hope, Y. Ghunaim et al., CVPR2023

We appreciate your positive feedback on our work. Please check our response to your questions and concerns.

Q1: What is the difference between Query function forward and reference forward? Why the (training) computational cost still reduce 50% when QR loss with reference forward existing?

A1: Sorry for the confusion. We would like to clarify QR loss is only used in OS-prompt++, which is an advanced version of our OS-Prompt. The OS-Prompt does not employ QR loss and is more efficient than existing two-stage methods during both training and inference. The users can choose to use OS-prompt if they have fewer computing resources in training, but we highlight that OS-prompt can achieve comparable performance with existing SOTA. We summarize the difference between OS-Prompt and OS-Prompt++ in our general response.

The Query function (used by previous PCL methods) is leveraged in training and inference, but the Reference function (proposed in this work) is only used in training, not inference. Therefore, with QR loss (i.e. OS-prompt++), the training computational cost does not reduce by 50%. We report the relative computational cost of both training and inference in Table 3. Please refer to the training cost for OS-prompt++ in Table 3.

Q2: From the description, seems that how to select query that different from previous PCL is not clear, both of them apply the [CLS] token embedding.

A2: We apologize for the unclear description. To obtain a query in layer l, prior methods utilize the [CLS] token from the 'last layer' of the Query ViT (Fig. 1, Left). In contrast, our approach involves using the [CLS] token from the 'layer l,' as depicted in Fig. 3 (arrow ②). Consequently, traditional PCL methods necessitate two feedforward steps: one for obtaining the query prompt and the second for the backbone ViT. In our OS-Prompt, we directly leverage the intermediate [CLS] token feature, which requires only one feedforward step. The difference in query selection strategy is also elaborated in Eq.(1) and Eq.(6) in our manuscript.

Q3: Is QR without query ViT could perform better generalization on unseen task/domain?

A3: We appreciate the insightful suggestion from the reviewer. This is an interesting and important concern because we use the intermediate layer’s [CLS] feature as a query, which might have less generalization power compared to the [CLS] feature from the last layer of query ViT.

To this end, we conducted experiments on DomainNet [R1], a large-scale domain adaptation dataset consisting of around 0.6 million images distributed across 345 categories. Our continual domain adaptation task was created using five diverse domains from DomainNet: Clipart → Real → Infograph → Sketch → Painting. Thus every task has a different domain image, which is a proper experiment to check the generalization ability of our method. The experimental results are presented in the table below.

Our method (OS-prompt and OS-propmt++) achieves comparable performance compared to the previous PCL methods with ~50% inference computational cost. These findings suggest the adaptability of our method across various domains in different stages of continual learning.

We have added the results in the Appendix.

[R1] Peng et al. Moment matching for multi-source domain adaptation. ICCV19

We appreciate your constructive feedback on our work. Please check our response to your questions and concerns.

Q1: The proposed method needs to discuss the computation cost of the training phase more thoroughly.

A1: Thank you for the valuable feedback. We want to clarify that our OS-Prompt++ (with query pool regularization) does not result in increased training costs compared to previous Prompt Continual Learning (PCL) methods, such as CodaPrompt, DualPrompt, and L2P (refer to Table 3). Instead, OS-Prompt++ maintains the same training cost as the earlier approaches. On the other hand, our efficient version of PCL, named OS-prompt, minimizes the forward computation of the PCL framework, improving computational efficiency for both training and inference.

In our work, we aim to solve 'Rehearsal-Free' continual learning, aiming to address both privacy and memory efficiency concerns, aligning with prior PCL research. Although our problem setting differs from online continual learning like [A], we agree with the reviewer that our proposed method needs a more thorough discussion of the training computation cost to provide a better understanding of the proposed approach. Following Table 1 provided in [A], we compute the relative training complexity with respect to ER or Experience Replay, which is the simple baseline with standard gradient-based training.

Here, we would like to clarify the training computational cost of prompt learning. In general neural network training, the forward-backward computational cost ratio is approximately 1:2. This is due to gradient backpropagation ($\frac{dL}{da_l}=W_{l+1}\frac{dL}{da_{l+1}}$) and weight-updating ($\frac{dL}{dW_l}=\frac{dL}{da_l+1}a_l$), where $L$ represents the loss, $W$ denotes the weight parameter, $a$ is the activation, and $l$ is the layer index. In prompt learning, the forward-backward computational cost ratio is approximately 1:1. This is in contrast to general neural network training, as only a small fraction of the weight parameters (less than 1%) are updated.

Bearing this in mind, we present the observations derived from the table, assuming that all methods employ the same architecture.

• Previous PCL (L2P, DualPrompt, CodaPrompt) consists of two steps; (1) the query ViT requires only feedforward without backpropagation; (2) the backbone ViT feedforward-backward training with prompt tuning. This results in the relative training computational cost is 1 ($= \frac{Query ViT forward (1) + Backbone ViT forward (1) + Backbone ViT backward (1)}{ER forward (1) + ER backward (2)}$)
• Similarly, our OS-Prompt++ also consists of two steps; (1) the reference ViT with only a feedforward step to calculate QR loss; (2) the backbone ViT feedforward-backward training with prompt tuning. This results in a relative training computational cost 1 ($= \frac{Reference ViT forward (1) + Backbone ViT forward (1) + Backbone ViT backward (1)}{ER forward (1) + ER backward (2)}$) with respect to ER.
• On the other hand, our OS-Prompt requires one feedforward-backward step with prompt tuning. Therefore, relative training computaional cost becomes $\frac{2}{3}$ ($= \frac{Backbone ViT forward (1) + Backbone ViT backward (1)}{ER forward (1) + ER backward (2)}$). This shows the potential of our OS-Prompt on online continual learning.

Overall, we propose two versions of the one-stage PCL method (i.e., OS-prompt and OS-prompt++), and these two options enable the users to select a suitable method depending on the problem setting. For example, for online continual learning, OS-prompt is a better option since it requires less training cost. On the other hand, for offline continual learning, one can use OS-prompt++ to maximize performance while spending more training energy.

We have added this discussion in the Appendix.

[A] Real-Time Evaluation in Online Continual Learning: A New Hope, Y. Ghunaim et al., CVPR2023

We sincerely appreciate the constructive feedback and valuable suggestions provided by the reviewers for our work. We are committed to addressing each of the reviewers' questions and concerns in a point-to-point manner.

To facilitate this process, we have revised our manuscript and have attached the modified PDF file based on the reviewers' comments. Since we have a day for Author/Reviewer discussion, we welcome any further discussion and would be grateful.