Continual Learning via Learning a Continual Memory in Vision Transformer

Purpose of using the VDD benchmark over other benchmarks

The primary motivation for utilizing the VDD benchmark is to address three critical aspects of continual learning:

• The necessity for a natural benchmark in continual learning.
• The need for a dynamic backbone to handle tasks that differ significantly from the base task.
• The importance of explicit task ID inference in class-incremental learning.

Elaboration on Key Points

Previous benchmarks have often focused on constructing tasks within artificial settings. For instance, in the widely used Split ImageNet-R benchmark, tasks are typically created by randomly shuffling classes, with each task containing an equal number of classes. This design does not reflect real-world scenarios, where the number of classes within each task may vary significantly. The randomized classes in a task also do not reflect a natural setting: real world tasks can be coherent but still show a large variance in distribution. The VDD benchmark effectively addresses these limitations by providing a more realistic framework for evaluating continual learning models.

To illustrate this concept from a model design perspective, we have included results on the Split ImageNet-R benchmark, a standard benchmark in prior research, within our revised manuscript. As shown in Table 3, methods such as Learn to Prompt, Dual Prompt, and CODAPrompt perform well under standard conditions, where each task contains an equal number of classes. However, their performance degrades substantially when the number of classes per task varies. This decline is primarily due to inconsistencies in how the classification head is utilized during training and testing. This effect is even more pronounced with the VDD benchmark, which exhibits a larger imbalance in class distribution across tasks.

Under a class-incremental learning framework, Learn to Prompt, Dual Prompt, and CODAPrompt demonstrate significant accuracy drops on the VDD benchmark. Conversely, S-Prompts, which employs explicit task ID inference, performs considerably better. Building upon this strategy, we integrated a similar task ID inference approach into our proposed CHEEM model. Our findings show that CHEEM outperforms S-Prompts, underscoring the importance of employing a dynamic backbone to effectively handle tasks that differ substantially from the base task.

Choice of the Projection layer as CHEEM

Our choice of the projection layer on the MHSA block is based on the following considerations:

Forward transfer ability Fine-tuning the projection layer results in high average accuracy across all tasks in the VDD benchmark. While fine-tuning the entire MHSA block achieves slightly higher accuracy, the projection layer offers a more efficient, lightweight alternative.

Forgetting Although fine-tuning the projection layer exhibits higher average forgetting compared to fine-tuning the Query and Key components, it significantly outperforms them in terms of forward transfer. Fine-tuning the entire MHSA block leads to significant forgetting, while the projection layer has a good balance between forward transfer and forgetting.

Simplicity and minimally invasive implementation The Query, Key, and Value components are typically implemented as a single, monolithic linear layer in standard architectures. This structure limits the flexibility of these components to be modified with respect to the four basic operations proposed in our paper. In contrast, the projection layer is lightweight and a standalone component, making it easier to modify.

More comparisons with prior works

We have added comparisons with DualPrompt and CODAPrompt in the revised manuscript (Table 2) under the class-incremental setting. Table 2 gives a holistic overview of the problem we address, as mentioneed above. We leave comparisons with models at the same scale of parameters for future work, as it is not the focus of this paper.

Thank you for your valuable feedback. We address your comments as follows:

Most important update in revision: our proposed CHEEM works with Exemplar-free Class-Incremental Continual Learning (ExfCCL). Please refer to our global response and revised manuscript for details.

Improvements in presentation

We have revised the manuscript to improve the writing and presentation, and provided a brief summary of the changes in the global comment. Please refer to the revised manuscript at you convenience, as well as the global comment which summarizes the changes.

Addressing the class incremental setting

We have modified the method to handle class-incremental learning. The modifications required to handle class-incremental learning are minor and do not change the core of the method. Section 2.1 and 2.2 in the revised manuscript describe the modifications in detail. We have reproduced them below for your convenience:

"To handle class-incremental setting, we explicitely infer task IDs for test data, and adopt the method proposed in S-Prompts. For a task $t$, we leverage $K$-mean clustering of the CLS tokens of its training images computed by the query function $q(\cdot)$ (i.e., the base model $f_0$). Denote by $Z_t=\{z^1_t, \cdots, z^K_t\}$ the clustered task centroids for task $T_t$. Werefer to this as the external memory, $\Psi=\cup_{t=1}^N Z_t$ after training (as opposed to the internal parameter memory). In inference, for a testing sample $x$, we first compute its {\tt CLS} token using the same query function $q(x)$, denoted by $z$. The task ID is then inferred via $K$-NN retrieval, $KNN(z, \Psi)$, either by retrieving the class ID of the Top-1 NN centroid in the external memory $\Psi$, or by majority voting of the class ID from the $K$-NN centroids."

We have also added comparisons to Learn to Prompt, DualPrompt and CODAPrompt under the class incremental setting in Table 2 of the revised manuscript. The revised Table 2 gives a holistic overview of the advantages of CHEEM:

• Comparing S-Prompts and CHEEM shows the advantage of using a dynamic backbone over a fixed one.
• Learn to Prompt, DualPrompt and CODAPrompt show significant drops in accuracy due to discrepancy in the way the classification head is used in training and testing.
• Baseline regularization-based methods EWC and simple L2 regularization show a strong baseline on VDD benchmark, even outperforming recent prompt-based method (due to the discrepancy in handling the classification head). However, they still suffer from significant catastrophic forgetting, highlighting the balanced stability and plasticity of CHEEM.

Purpose of Figure 4 (Figure 2 in the revised manuscript)

Figure 4 in the original manuscript is meant to illustrate that the proposed HEE based NAS is effective in reusing the existing parameters (internal memory) when learning a new task. This is evidenced by the large number of Reuse and Adapt operations learned by the search. In contrast, the pure exploration sampling scheme learns a large number New operations adding to the parameter cost, and learns Skip operations in a manner that harms the overall performance (Figure 8 in the revised manuscript). This shows that the proposed hierarchical exploration-exploitation sampling is effective in learning a model that can reuse the existing parameters when learning a new task.

Dear Reviewer wv9n,

Thank you for your reply. We are sorry to hear that our rebuttal did not address your concerns.

We would like to address your concerns in the reply as follows.

About the writing: The authors responded in a really strange way. I expect the authors to polish the writing, .e.g, fix typos. However, the authors almost re-write the manuscript, in my view.

We significantly revised introduction and experiments due to the switch from task-incremental settings to class-incremental settings. For the introduction, we revised it paragraph by paragraph to better reflect the update. For experiments, we re-organized and rewrote the experiments. However, we keep the core component (section 2) largely the same, except for some rewording and a new subsection on task ID inference. By analogy, we keep the "shape" of the submission, while significantly updating its "appearance". The underlying reason is that we aimed to seize the opportunity of this review process (all valuable feedback) to try our best to make the submission better.

About the class-incremental setting: I expect the authors to explain the illustration in Line 97-102 in the original manuscript. However, the authors modify their method to fit class-incremental settings. It is a very significant change since the two settings are different.

We agree that the changes are significant. But those changes are still based on the exactly same ideas proposed in the original submission, with a newly adopted task ID inference module. More specifically, we did not need to retrain our models, and only change the inference settings by replacing given task IDs to inferred ones, and then rerun the evaluation scripts to compute the average accuracy and forgetting. Motivated by this review process, after we observed that we can indeed integrate task ID inference (adopted from S-Prompts) and our original dynamic backbone learning method, we see this as a big opportunity since it reveals very interesting observations under the CCL settings. E.g., the state-of-the-art prompting-based methods (L2P++, DualPrompts and CODA-Prompt) catastrophically failed on VDD.

About the benchmark and model design: I am still confused about why the proposed method is specific to the proposed benchmark. The better performance cannot explain why.

Our choice of sticking to the VDD benchmark is two-fold as follows: (i) It is the benchmark we claimed contributions in our original submission, so we want to keep it. (ii) In the literature of continual learning, after the Learn-to-Grow paper (Li, et al, ICML'19), we do not see other continual learning methods have tested this challenging benchmark. Since our method is motivated by Learn-to-Grow, we chose to focus on VDD. We observed very interesting results in our revision, e.g. the state-of-the-art prompting-based methods (L2P++, DualPrompts and CODA-Prompt) catastrophically failed on VDD. We also added ImageNet-R in the revision.

In terms of "The better performance cannot explain why", we think we have figured out why. We added the analyses in the appendix of the revision which explained the experimental observations (why our method works well on VDD and why it does not work well on ImageNet-R, similarly for other prompting based methods).

Appendix Section A: Analysis of Task-Specific Head selection vs. Shared head. We analyze the behavior of the final logits in prompt-based methods, which use a shared classification head, and investigate under which conditions the predicted class remains unchanged when the full head is included as opposed to when only the task-specific head segment is used.

Appendix Section B: Pros and Cons of task ID inference using centroids. We plot tSNE visualizations of the task centroids on both, the VDD and Split ImageNet-R benchmarks. We find that the centroids are well separated in the VDD benchmark, but not in the Split ImageNet-R benchmark. This is the reason for the lower performance of CHEEM in the balanced Split ImageNet-R setting compared to prompt-based methods.

About more comparison: I think more comparison is needed.

We agree with you that evaluating our method on more datasets and comparing with more traditional methods will be helpful. We request you to re-think if our current comparisons are meaningful and/or interesting to the community or not. Comparing with more traditional methods in a fair manner will need to adapt them to work with ViTs, for which we have done for three methods, Supermasks in Superposition (SupSup) (Wortsman et al., 2020), Efficient Feature Transformation (EFT) (Verma et al., 2021) and Lightweight Learner (LL) (Ge et al., 2023) in the appendix (Table 6). Comparing on more benchmarks will take more time for experiments beyond what we have during the rebuttal period, we will try our best to add more in another round of revision.

Lines 92-102: Common classification space

The meaning of lines 97 and 98 in the original manuscript was to simply state that current approaches assume the same number of classes per task and use a shared classification head, which is not a natural setting. We apologise for the confusion and we have carefully revised the entire section in the revised manuscript. Lines 96 to 130 in the revised manuscript state our motivation more clearly.

We still argue that using a common classification head, where head segments from the previous tasks are masked out when learning the current task leads to performance degradation due to the discrepancy in the way the classification head is used in training and testing. This setup is used in most of the prior work like L2P, DualPrompt and CODA-Prompt. The drop is especially severe when the number of classes in a task varies, as shown in Table 3 in the revised manuscript.

Use of co-variances of features to guide the search

This is an interesting question and worth future investigation. For the purpose of this work, we found that using the mean is sufficient to guide the search. We plan to address your point in future work.

Experimental results of Table 5 (Table 6 in the revised manuscript)

We have clarified the purpose of these experiments in the Appendix Section D.3 of the revised manuscript. We reproduce the explanation below for your convenience:

"Table 6 demonstrates that CHEEM achieves the highest average accuracy on the VDD benchmark. Although its performance on individual tasks may be lower, CHEEM exhibits robustness to domain variations, as reflected in its overall average accuracy. In contrast, other methods show varying performance depending on the domain. SupSup performs well on tasks that are significantly different from the base ImageNet task, such as UCF 101, Omniglot, and SVHN, but performs poorly on tasks closely related to ImageNet. Conversely, EFT and LL excel on tasks similar to ImageNet but struggle with out-of-distribution tasks. Despite its lower individual task performance, CHEEM's consistent performance across diverse downstream tasks highlights its adaptability and makes it more general."

FLOPs comparison with L2P

The FLOPS increase in L2P is due to the additional prompt tokens added to the input sequence, which results in additional computations in the MHSA and FFN blocks. L2P adds a fixed number of prompts to every input sequence, and hence the increase in FLOPS is constant per task as you have stated. However, to avoid confusion, and since the increase is not significant enough to cause problems in pranctice, we have removed this in the revised version of the manuscript.

Application to Vision-Langiage Models

Thank you for you suggestions and pointing out the potential of applying CHEEM to vision-language models. We will explore this in future work.

Clarity of writing

We have revised the manuscript to improve the writing and presentation. Please refer to the revised manuscript at you convenience, along with the global coment highlighting the changes made.

Lack of citations

Thank you for pointing out these excellent works to us. We have included citations to all the works mentioned.

Thank you for your valuable feedback. Below, we provide clarifications on the points raised:

Most important update in revision: our proposed CHEEM works with Exemplar-free Class-Incremental Continual Learning (ExfCCL). Please refer to our global response and revised manuscript for details.

Applicabiltiy to class incremental setting

We have modified the method to handle class-incremental learning. The modifications required to handle class-incremental learning are minor and do not change the core of the method. Section 2.1 and 2.2 in the revised manuscript describe the modifications in detail. We have reproduced them below for your convenience:

"To handle class-incremental setting, we explicitely infer task IDs for test data, and adopt the method proposed in S-Prompts. For a task $t$, we leverage $K$-mean clustering of the CLS tokens of its training images computed by the query function $q(\cdot)$ (i.e., the base model $f_0$). Denote by $Z_t=\{z^1_t, \cdots, z^K_t\}$ the clustered task centroids for task $T_t$. Werefer to this as the external memory, $\Psi=\cup_{t=1}^N Z_t$ after training (as opposed to the internal parameter memory). In inference, for a testing sample $x$, we first compute its {\tt CLS} token using the same query function $q(x)$, denoted by $z$. The task ID is then inferred via $K$-NN retrieval, $KNN(z, \Psi)$, either by retrieving the class ID of the Top-1 NN centroid in the external memory $\Psi$, or by majority voting of the class ID from the $K$-NN centroids."

We have added comparisons with Learn to Prompt, DualPrompt, and CODAPrompt under the class-incremental learning setting, as presented in Table 2 of the revised manuscript. This updated table gives a comprehensive view of the strengths of CHEEM:

• Advantage of a dynamic backbone: Comparing S-Prompts with CHEEM demonstrates the benefits of utilizing a dynamic backbone rather than a fixed one, leading to improved adaptability across tasks.
• Methods such as Learn to Prompt, DualPrompt, and CODAPrompt exhibit significant accuracy drops due to inconsistencies in how the classification head behaves during training versus testing. This shows the importance of explicit task ID inference.
• Elastic Weight Consolidation (EWC) and simple L2 regularization serve as strong baselines on the VDD benchmark, outperforming recent prompt-based methods. This may be due to their more consistent handling of the classification head. However, these regularization methods still experience notable catastrophic forgetting, underscoring the balanced stability and plasticity achieved by CHEEM.

We have also included experiments with the Split ImageNet-R benchmark (Section 3.2). We experiment with two settings: balanced setting, where ImageNet-R is divided into 10 tasks with 20 classes each, and an imbalanced setting, where ImageNet-R is divided into 6 tasks with uneven number of classes.

• Under task-incremental setting, CHEEM outperforms all the baselines in both settings, showing the expressive power of the dynamic backbone.
• Under class-incremental setting, prior prompt-based methods perform better than CHEEM in the balanced setting, which is caused due to the low average precision of the iask ID inference
• In the imbalanced setting, prompt-based methods show a sharp drop in performance (e.g., from 76.48% to 67.66% by CODA-Prompt) due to the classification head discrepancy described above. Under the imbalanced setting, CHEEMperforms on par with the prompt-based methods.

Applicability to domain-incremental setting and online continual learning

The VDD benchmark can be considered as a domain-incremental setting, the only difference being that the space of the outputs is also different. Hence, CHEEM can easily be extended to a domain-incremental setting if a separate classifier head is used for each domain. Moreover, the explicit task ID inference in the updated version of CHEEM can infer the domain, as shown by the performance on the VDD benchmark. We agree that the applicability of CHEEM to online continual learning is more difficult, and we plan to address this in future work. We also note that prior works using the ViT backbone have not addressed online continual learning, and as such it remains an open problem.

Dear AC and authors of paper #8730:

I have read the rebuttal and the revised version of the paper carefully and have further comments listed below.

Yours, Reviewer 874z

Comments:

1. I have noticed that the main claim of this work has undergone significant changes compared to both the original and revised versions. For example, in the original version (lines 103-104), it was stated that: “So, we choose to take one step forward by studying task-incremental learning to gain insights on how to structurally and dynamically update Transformer models at streaming tasks in the VDD benchmark.” However, in the revised version, the focus has shifted to addressing “the exemplar-free class-incremental (ExfCCL) setting.” In my original review, I suggested demonstrating the potential of the proposed method in a more challenging continual learning scenario rather than completely changing the main claim of the paper. Such a drastic change not only increases the reviewers’ burden but also causes confusion regarding the paper’s core contribution. Moreover, as per ICLR’s policy: “Area chairs and reviewers reserve the right to ignore changes that are significantly different from the original paper.”

2. Despite these issues, I have still carefully reviewed both the rebuttal and the revised paper. While I acknowledge the effort, I believe there is still room for improvement, and I do not recommend acceptance of the paper in its current state. Since the revised version now focuses on tackling exemplar-free class-incremental learning, I question the choice of sticking to the VDD benchmark. Why not follow prior works[3,4] to conduct experiments on datasets CIFAR-100, CUB-200, ImageNet-R, ImageNet-A, Omni-Benchmark, Object, and VTAB? or follow the MTIL benchmark [A, B] to conduct experiments on Aircraft, Caltech101, CIFAR100, EuorSAT, Flowers, Food, MNIST, OxfordPet, Cars and Sun397 sequentially? What makes the VDD dataset particularly unique or crucial for this study?

   If the primary claim is now centered around class-incremental learning with ViTs, the experimental scope is not sufficiently extensive to convincingly validate the effectiveness of the proposed method.

3. The current approach leverages techniques from S-prompts for task ID inference. If class-incremental learning is now the main focus, more methods for task-ID inference should be explored and discussed. This is also related to my original review, where I raised concerns about how to measure task similarity—a point I believe the authors have not addressed thoroughly in either the rebuttal or the revised version.

Overall, the experimental validation is not comprehensive enough, and critical discussions regarding task similarity and alternative methods are lacking.

Based on the above points, I believe the current version of the paper is not ready for publication, and I have downgraded my score to 3.

Thank your for your valuable feedback. Following are the clarifications on your comments:

More comparisons with prior works

We have adapted the method to accommodate class-incremental learning. These modifications are minimal and do not alter the core framework of the original method. Section 2.1 and 2.2 in the revised manuscript describe the modifications in detail. We have reproduced them below for your convenience:

"To handle class-incremental setting, we explicitely infer task IDs for test data, and adopt the method proposed in S-Prompts. For a task $t$, we leverage $K$-mean clustering of the CLS tokens of its training images computed by the query function $q(\cdot)$ (i.e., the base model $f_0$). Denote by $Z_t=\{z^1_t, \cdots, z^K_t\}$ the clustered task centroids for task $T_t$. Werefer to this as the external memory, $\Psi=\cup_{t=1}^N Z_t$ after training (as opposed to the internal parameter memory). In inference, for a testing sample $x$, we first compute its {\tt CLS} token using the same query function $q(x)$, denoted by $z$. The task ID is then inferred via $K$-NN retrieval, $KNN(z, \Psi)$, either by retrieving the class ID of the Top-1 NN centroid in the external memory $\Psi$, or by majority voting of the class ID from the $K$-NN centroids."

We have added comparisons with Learn to Prompt, DualPrompt, and CODAPrompt under the class-incremental learning setting, as presented in Table 2 of the revised manuscript. This updated table gives a comprehensive view of the strengths of CHEEM:

• Advantage of a dynamic backbone: Comparing S-Prompts with CHEEM demonstrates the benefits of utilizing a dynamic backbone rather than a fixed one, leading to improved adaptability across tasks.
• Methods such as Learn to Prompt, DualPrompt, and CODAPrompt exhibit significant accuracy drops due to inconsistencies in how the classification head behaves during training versus testing.
• Elastic Weight Consolidation (EWC) and simple L2 regularization serve as strong baselines on the VDD benchmark, outperforming recent prompt-based methods. This may be due to their more consistent handling of the classification head. However, these regularization methods still experience notable catastrophic forgetting, underscoring the balanced stability and plasticity achieved by CHEEM.

We have also included experiments with the Split ImageNet-R benchmark (Section 3.2). We experiment with two settings: balanced setting, where ImageNet-R is divided into 10 tasks with 20 classes each, and an imbalanced setting, where ImageNet-R is divided into 6 tasks with uneven number of classes.

• Under task-incremental setting, CHEEM outperforms all the baselines in both settings, showing the expressive power of the dynamic backbone.
• Under class-incremental setting, prior prompt-based methods perform better than CHEEM in the balanced setting, which is caused due to the low average precision of the iask ID inference
• In the imbalanced setting, prompt-based methods show a sharp drop in performance (e.g., from 76.48% to 67.66% by CODA-Prompt) due to the classification head discrepancy described above. Under the imbalanced setting, CHEEMperforms on par with the prompt-based methods.

With the new resutls and comparisons, we believe that the revised manuscript provides a more comprehensive comparison with prior works highlighting the advantages of CHEEM.

Additional training time for the proposed method

The training time for the proposed method is longer compared to other methods, which is a drawback of the method. However, the proposed hierarchical exploration-exploitaiton sampling enables the method to explicitly reuse much of the parameters (internal memory) of the prior tasks, resulting in a small addition in the number of parameters. The search also enables learning the Skip operation, which reduces the number of FLOPs for a downstream task.

Comparison with PEFT techniques

The core focus of the paper is to formulate a method that enables dyynamic allocation of parameters to tasks based on their difficulty, including the ability to skip modules when the task is easy. A key drawback of PEFT methods used for continual learning is that they do not enable the model to dynamically allocate lesser compute to easier tasks. In the proposed CHEEM, the Skip operation enables this. Furthermore, the Adapt operation in the proposed method can be replaced by other adapter methods such as LoRA. We leave this study for future work, as it is not the focus of this paper.

Thank your for your valuable feedback. Following are the clarifications on your comments:

Most important update in revision: our proposed CHEEM works with Exemplar-free Class-Incremental Continual Learning (ExfCCL). Please refer to our global response and revised manuscript for details.

Improvements to the presentation

We have revised the manuscript to improve the presentation. We have simplified Figure 4 (Figure 2 in the revised manuscript) according to your suggestions. We have also made sure that Figure 5 in the original manuscript is closer to the corresponding description (Figure 3 in the revised manuscript).

The overall designs are not focus to the continual learning. (i) For example, in line 244 "How to Adapt in a sustainable way" is related to the gradient vanishing or exploding when learning deep network. And the design of residual connection is not novel. (ii) Using the cls-token to guide the task-specific learning is not a significant contribution because previous works also use the cls-token to select and learn task specific prompts as in L2P and DualPrompt. (iii) The proposed HEE-BASED NAS considers the task similarity but the novelty is limited due to (ii).

The core focus of the paper is to formulate a method that enables dynamic allocation of parameters to tasks based on their difficulty, including the ability to skip modules when the task is easy. We have revised the manuscript to convey the motivations and contributions better in Section 1. We use the insights from prior works to enable us to implement our CHEEM framework, and our methodology explains how the different components are integrated effectively.

The methods compared in Table 2 and Table 3 are limited and somewhat outdated; the latest papers in class-incremental learning should be included with necessary modifications.

We have adapted the method to accommodate class-incremental learning. These modifications are minimal and do not alter the core framework of the original method. Section 2.1 and 2.2 in the revised manuscript describe the modifications in detail. We have reproduced them below for your convenience:

"To handle class-incremental setting, we explicitely infer task IDs for test data, and adopt the method proposed in S-Prompts. For a task $t$, we leverage $K$-mean clustering of the CLS tokens of its training images computed by the query function $q(\cdot)$ (i.e., the base model $f_0$). Denote by $Z_t=\{z^1_t, \cdots, z^K_t\}$ the clustered task centroids for task $T_t$. Werefer to this as the external memory, $\Psi=\cup_{t=1}^N Z_t$ after training (as opposed to the internal parameter memory). In inference, for a testing sample $x$, we first compute its {\tt CLS} token using the same query function $q(x)$, denoted by $z$. The task ID is then inferred via $K$-NN retrieval, $KNN(z, \Psi)$, either by retrieving the class ID of the Top-1 NN centroid in the external memory $\Psi$, or by majority voting of the class ID from the $K$-NN centroids."

We have added comparisons with Learn to Prompt, DualPrompt, and CODAPrompt under the class-incremental learning setting, as presented in Table 2 of the revised manuscript. This updated table gives a comprehensive view of the strengths of CHEEM:

• Advantage of a dynamic backbone: Comparing S-Prompts with CHEEM demonstrates the benefits of utilizing a dynamic backbone rather than a fixed one, leading to improved adaptability across tasks.
• Methods such as Learn to Prompt, DualPrompt, and CODAPrompt exhibit significant accuracy drops due to inconsistencies in how the classification head behaves during training versus testing. This shows the importance of explicit task ID inference.
• Elastic Weight Consolidation (EWC) and simple L2 regularization serve as strong baselines on the VDD benchmark, outperforming recent prompt-based methods. This may be due to their more consistent handling of the classification head. However, these regularization methods still experience notable catastrophic forgetting, underscoring the balanced stability and plasticity achieved by CHEEM. We have also included experiments with the Split ImageNet-R benchmark (Section 3.2). We experiment with two settings: balanced setting, where ImageNet-R is divided into 10 tasks with 20 classes each, and an imbalanced setting, where ImageNet-R is divided into 6 tasks with uneven number of classes.
• Under task-incremental setting, CHEEM outperforms all the baselines in both settings, showing the expressive power of the dynamic backbone.
• Under class-incremental setting, prior prompt-based methods perform better than CHEEM in the balanced setting, which is caused due to the low average precision of the iask ID inference
• In the imbalanced setting, prompt-based methods show a sharp drop in performance (e.g., from 76.48% to 67.66% by CODA-Prompt) due to the classification head discrepancy described above. Under the imbalanced setting, CHEEMperforms on par with the prompt-based methods.

Appendix Section A: Analysis of Task-Specific Head selection vs. Shared head

• We analyze the behavior of the final logits in prompt-based methods, which use a shared classification head, and investigate under which conditions the predicted class remains unchanged when the full head is included as opposed to when only the task-specific head segment is used.

Appendix Section B: Pros and Cons of task ID inference using centroids

• We plot tSNE visualizations of the task centroids on both, the VDD and Split ImageNet-R benchmarks. We find that the centroids are well separated in the VDD benchmark, but not in the Split ImageNet-R benchmark. This is the reason for the lower performance of CHEEM in the balanced Split ImageNet-R setting compared to prompt-based methods.

Results on Split ImageNet-R benchmark

The ImageNet-R dataset is created to challenge models trained using ImageNet. So, by design, the external memory of our CHEEM that is based on ImageNet-1k trained base model will not work well. The above Table shows the results which we analyze as follows:

i) Randomly Assigned and Imbalanced Classes in Streaming Task Challenge All of ExfCCL Methods We Test. Under CCL, the three prompting based methods suffer from catastrophic drop of performance from balanced to imbalanced settings (e.g., from 76.48 $\pm$ 0.25% to 67.66 $\pm$ 3.09% by CODA-Prompt, see Fig.5 and analyses in Appendix A ). Our CHEEM is on-par with DualPrompt and CODA-Prompt. Our CHEEM is also stable from balanced to imbalanced scenarios. Under TCL, all methods obtain significant performance boost, and our CHEEM is better than all others. Similarly, we envision that leveraging the dynamically learned backbone by our CHEEM in constructing the external memory will be a promising direction to be studied in future work.

ii) Randomly Assigned Yet Balanced Classes in Streaming Tasks Challenge the Task-Centroid based Task ID Inference. On the one hand, for streaming tasks with balanced but randomly sampled classes per task (without replacement), our CHEEM under CCL has the worst performance (64.72% vs 76.48% by CODA-Prompt), which was caused by the low average precision of task ID inference (see Fig.6 and analyses in Appendix B). Our CHEEM under TCL obtains the best performance (92.47%), which shows that the learned task-aware backbone is expressive. Overall, task centroids computed using the CLS tokens in the base model are not able to distinguish tasks from each other with high accuracy. One potential solution is to leverage the dynamically learned backbone by our CHEEM in constructing the external memory, considering its superior performance under TCL, at the expense of more costly task ID inference. On the other hand, although it is interesting to test continual learning approaches using the Split ImageNet-R benchmark, the random composition of classes in a task is not natural in comparison with scenarios in natural human learning. Unlike the Split ImageNet-R, the VDD benchmark may suit continual learning better from perspective the perspective of real-world scenarios, for which our CHEEM works the best.

Dear reviewers and AC, thank you for you efforts in reviewing our submission. We have reproduced the updated results in Table 2 and Table 3 for your convenince. Please let us know if you have further questions.

Results on VDD benchmark

Some analyses of the results:

i) The Advantage of Task-Aware Dynamic Backbone Over Frozen Backbone: CHEEM vs S-Prompts. CHEEM adopts the same task ID inference method (i.e., the external memory) proposed by S-Prompts. The improvement by CHEEM with above 3% absolute average accuracy increase clearly shows the advantage of our proposed internal parameter memory for maintaining task-aware dynamic backbones, against the method of prepending retrieved task-specific prompts in S-Prompts. The above Table also shows a potential for harnessing prompt-based methods in CHEEM. S-Prompts performs better on tasks that are similar to the first ImageNet task and have less training data such as Flwr (918 training images) and DTD (1692 training images).

ii) Explicit Task ID Inference: With vs Without. The three baselines, L2P, DualPrompt and CODA-Prompt, use a shared head classifier in inference, which, as aforementioned in the introduction, suffers from the discrepancy between training and testing. Due to the significant imbalance class distributions in the VDD, their average accuracy undergo catastrophic drops (19.42% by L2P, 26.86% by DualPrompt and 34.01% by CODA-Prompt, vs 78.95% by S-Prompts and 82.74% by our CHEEM).

iii) Stability vs Plasticity of the Backbone. To further show the advantage of our CHEEM learning to update the feature backbone from the base model, we compare with EWC, L2 Regularization and Experience Replay. The three methods regularize the weights, including the shared head classifier, in learning new tasks using different formulations. Comparing with L2P, DualPromt and CODA-Prompt, under CCL, the regularization based methods work better, showing the negative effects of the discrepancy of handling the shared head in training and testing by the three prompting based methods, as well as a potential of integrating them. Comparing with S-Prompts, the plasticity of updating the entire backbone with regularization is less effective than the plasticity introduced by prompts while keeping the backbone frozen in S-Prompts. Comparing with our CHEEM, the three regularization based methods suffer from significant forgetting, highlighting the balanced stability and plasticity via our CHEEM.