Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective

We appreciate the reviewer's recognition of the simple yet effective approach, promising performance results, detailed experiments and analysis, and the clarity of our paper. To further address the reviewer's concern, we have:

• included an analysis of FLOPs (for Weakness 1).
• provided an explanation of why iCaRL is chosen in Section 3 (for Weakness 2).
• provided further clarification to highlight the unique value of Section 3 (for Weakness 3).
• included experiments on a more realistic scenario and ViTs (for Question 1,3).
• provided an analysis of the impact of resetting the plastic learner (for Question 2).
• proofread the paper carefully to correct typos (for Question 4).

Weakness1: Details about FLOPs.

We appreciate the reviewer's valuable comment regarding the computational cost. We note that despite Dual-Arch involving the training of two models, the total FLOPs remain lower than those of the baselines. For instance, on CIFAR100, the FLOPs for Sta-Net and Pla-Net are 255M and 241M, respectively, resulting in a combined total of 496M. In contrast, the FLOPs for ResNet-18 and ResAC are 558M and 1383M, respectively. Actually, we have previously reported the FLOPs results for a batch (128) of samples in A.1 of the appendix of our initial submission, and we have revised these results to reflect the FLOPs per single sample, as reported above.

Weakness2: Concern about the use of iCaRL in Section 3.

We appreciate the reviewer's thoughtful consideration and acknowledge Experience Replay (ER) is a more intuitive choice. However, we argue that vanilla ER might not be suitable for evaluating the CL performance (especially stability) of architectures. Specifically, when using ER without other CL strategies, a model's performance on old tasks might be mainly enhanced by learning from the replay buffer rather than preserving existing knowledge.

To illustrate this point, we introduced an ablation baseline for both vanilla ER and iCaRL, denoted as 'w/ buffer only'. This baseline involves training the models exclusively on the last 5 tasks, with an initial replay buffer constructed from the first 5 tasks. We then evaluate their performance on the first 5 tasks after the last task is learned. Here are the results on CIFAR100/10:

The results show that the ablation baseline achieves comparable (for such a weak baseline) performance on tasks 1-5 as the original ER, even though it does not train on these tasks. In contrast, the performance difference in iCaRL is significantly larger than that in ER (7.14% vs. 2.10%). This demonstrates that iCaRL enables the model to better preserve existing knowledge beyond learning from the buffer, making it more effective for evaluating stability.

We hope that this additional analysis highlights the limitations of ER in evaluating the CL performance, particularly stability, of architectures and provides a clearer understanding of why we did not choose it.

Weakness3: Concern about the unique findings in Section 3.

While it is true that our results align with existing findings about the impact of architecture on stability, we believe they contribute to the field by extending the research to a more detailed understanding of the balance between stability and plasticity. Unlike previous works that primarily focus on overall CL performance or stability, our study delves into the separate aspects of stability and plasticity. This distinction allows us to highlight the trade-off between these two critical factors at the architectural level, which we believe adds a new dimension to the existing body of knowledge.

Question4: typo in Section 5.3

We have corrected the typo, changing "ABALATION STUDY" to "ABLATION STUDY." We have thoroughly proofread the manuscript to address every other issue we found. Thank you for pointing this out.

References.

[1] Arani et al. Learning fast, learning slow: A general continual learning method based on complementary learning system.

[2] Sarfraz et al. SYNERgy between SYNaptic Consolidation and Experience Replay for General Continual Learning

[3] Buzzega et al. Dark experience for general continual learning: a strong, simple baseline.

[4] Rostam et al. Complementary learning for overcoming catastrophic forgetting using experience replay.

[6] Wang et al. Learning to prompt for continual learning.

[7] Wang et al. Dualprompt: Complementary prompting for rehearsal-free continual learning.

Question1: Experiments on more general continual learning settings.

Thank you for your insightful comments. While the focus of this study is on class-incremental learning, we recognize the importance of more realistic scenarios. We have conducted additional experiments on general continual learning [1] [2] [3], a more challenging and realistic continual learning scenario where task boundaries are not explicitly available. These results indicate that Dual-Arch consistently enhances CL performance in this scenario, underscoring its broad applicability. We have also included these results in Section A.6 of this revision. Additionally, we have revised our Related Work to extend its scope by including a discussion of this scenario.

Question2: Concern about whether resetting weights benefits plasticity in Dual-Arch.

We appreciate the reviewer's insightful comments regarding the potential benefits of periodically resetting the weights of the plastic learner. We did explore this strategy but found that it did not enhance performance within our proposed framework. While we acknowledge the valuable insights from existing works that suggest that resetting weights can improve model plasticity in certain scenarios, we believe that preserving the existing weights allows the plastic learner to leverage prior knowledge for acquiring new knowledge more effectively.

Here are the performance results (%AIA) of resetting the plastic learner before learning each task, denoted as 'w/ resetting,' on CIFAR100/10:

Question3:Exploration and insights for vision transformer.

While our study focuses on ResNet, we believe that the architectural insights presented in our paper could potentially extend to Vision Transformers (ViTs). Validation of fully transferring our solution to the pre-trained ViTs is beyond our current resources, but we have conducted preliminary experiments to validate the partial transfer of our insights to vision transformer architectures.

Firstly, regarding the methods that use a frozen pretrained backbone, the trainable part of the architectures is solely a single layer known as prompt, which might make the architecture very stable but lacking in plasticity. Therefore, applying our solution by employing the pretrained ViT as the stable learner and a plastic architecture as the plastic learner may benefit continual learning. Here, we combine our proposed framework with these methods by using the ResNet-18 as the plastic learner, and the results are as follows.

Moreover, we note that our proposed framework can also benefit ViTs within the setting of training from scratch. Here are the validation results on SepViT [4].

Finally, a possible approach for fully transferring our proposed framework to pretrained ViTs might involve designing two distinct vision transformer architectures—one wide and shallow, the other deep and thin—and pre-training them on a large dataset. These pretrained models could then be utilized in Dual-Arch, similar to what we have explored in our study. However, the computational cost associated with pre-training these models is substantial and currently beyond our resources.

Dear Reviewer a2rA:

Thanks for your careful comments and your time spent on our work. We have revised our paper and added the discussion and experiments concerning.

Currently, all of your concerns can be resolved in the responses and revised version of the paper. However, as the revision deadline is approaching, we kindly request your feedback to confirm that our responses and revisions have effectively resolved your concerns. If there are any remaining issues, we would be grateful for the opportunity to address them to ensure the quality of our work.

Yours sincerely,

Authors of Paper 6928

Dear Reviewer a2rA,

Hope you had a wonderful Thanksgiving holiday week! We would like to sincerely thank you once again for your valuable comments and your time spent on our work. We believe that all your questions can be well addressed in the responses and revised version of the paper.

As the extended discussion period ends tomorrow, we would greatly appreciate your feedback soon. If our rebuttal adequately addresses your concerns, we kindly request an update on your evaluations. Once again, thank you very much for your reviews and feedback!

Yours sincerely,

Authors of Paper 6928

We appreciate the reviewer's recognition of our interesting findings, simple yet effective approach, and the clarity of our paper. To further address the reviewer's concern, we have:

• revised Table 1 to better strengthen our motivation (for Weakness 1).
• provided more detailed hyper-parameters settings and error bars for Table 3 (for Weakness 2).
• included a more detailed comparison with related works (for Weakness 3).
• provided a more detailed workflow of our approach (for Question 1,3)
• included forgetting results for Table 2 (for Question 2).

Weakness1: Presentation of Table 1.

Thank you for your suggestion to include the total parameter count in Table 1 to enhance clarity. The revised table is as follows.

Note that the parameter counts of the variant differ only slightly (less than $\pm$3%) from the original ResNet-18.

The reason we vary both width and depth simultaneously is to maintain a consistent number of parameters. This ensures that any observed differences in performance are mainly attributable to architectural designs rather than parameter counts. Changing the width or depth individually would significantly alter the parameter counts, introducing an additional variable. Our study's motivation stems from the observation that, with similar numbers of parameters, wider and shallower networks tend to exhibit greater stability but lower plasticity, while deeper and narrower networks show the opposite trend. The experiments in Table 1 effectively demonstrate this trade-off.

Weakness2: Details on hyper-parameters.

We highly recognize the significant impact of hyper-parameter tuning on Continual Learning (CL). To ensure a fair comparison and avoid over-tuning, we have adhered to the following practices:

• For parameter $\alpha$, we set it to 0.5 as mentioned in Section 4.3, which aligns with the earliest work on knowledge distillation [1]. This setting is consistent across experiments that use dual architectures, including the ablation study, to ensure a fair comparison. We have further highlighted this content in Section 4.3 of this revision.
• For parameter $\beta$, it is determined by the CL methods in combination with our work, as detailed in Section 4.3. We followed the default settings in the open-source library PyCIL [2] for both Dual-Arch and all baselines to ensure a fair comparison, as mentioned in the Implementation Setup in Section 5.1. We have further highlighted the importance of hyper-parameters in Section 5.1 of this revision.

Moreover, considering that $\beta$ is actually not related to our study, we have removed it from Eq.(2), now presented as Eq. (4) in the current revision, to avoid misunderstanding. Below are the performance results (%AIA) with different values of $\alpha$.

The results show that a smaller $\alpha$ may result in better performance.

To further address the reviewer's concern, we have revised the result in Table 3, reporting the mean and std over three runs with different initializations.

Moreover, we note that we have evaluated our proposed framework by combining it with a lot of existing CL methods to fully validate its effectiveness. For instance, if we employ the Wilcoxon Signed Rank Test to calculate the confidence level at which Dual-Arch significantly outperforms using two Sta-Net across CL methods, the results yield a p-value of 0.03125, which is less than the significance level of 0.05. This indicates that Dual-Arch is significantly better than using two Sta-Nets at a confidence level of 95%.

For reference, the Wilcoxon Signed Rank Test can be implemented in R using the command:

wilcox.test(c(70.21 ,71.53 ,75.26 ,73.18 ,74.44), c(69.27 ,71.38 ,74.30 ,72.65 ,73.77), paired=TRUE, alternative = "greater")

Dear Reviewer e7KZ:

Thanks for your careful comments and your time spent on our work. We have revised the paper accordingly and incorporated additional discussions and experiments addressing your concerns.

Currently, all of your concerns can be resolved in the responses and revised version of the paper. We want to leave a gentle reminder that the revision deadline is approaching. We would appreciate your feedback to make sure that our responses and revisions have resolved your concerns, or whether there is a leftover concern that we can address to ensure the quality of our work.

Yours sincerely,

Authors of Paper 6928

We appreciate the reviewer's recognition of our innovative research perspective, findings, and approach. To further address the reviewer's concern, we have:

• revised the Introduction and Related Works to better emphasize the limitations of previous methods and the advantages of our approach (for Weakness 1).
• included detailed mathematical descriptions (for Weakness 1, Question 1,2).
• revised Table 1 follows the suggestion (for Question 3).
• included additional ablation experiments (for Question 4).

Weakness1: Better presentation of this study's value.

Thank you for your valuable comments. While we have highlighted the unique contributions of our study by emphasizing its distinct research perspective, we acknowledge the need to further emphasize the limitations of previous methods and the advantages of our approach. To address this, we have revised the related content in the Introduction Section and expanded the discussion in the Related Works Section to clearly delineate the advantages of our approach beyond existing methods.

Weakness2: Mathematical Descriptions.

We appreciate the reviewer's suggestion for detailed mathematical descriptions. In this revision, we have incorporated these descriptions in Section 4.3, including the definitions of continual learning, the cross-entropy loss ($L_{CE}$), and the knowledge distillation loss ($L_{KD}$). Additionally, we would like to clarify that the continual learning loss ($L_{CL}$) is an abstract term that is specifically designed by the continual learning methods in combination with our approach.

Weakness3: Experiments on other architectures

While our study primarily focuses on ResNet, the insights presented in our paper could extend to other architectures, such as Vision Transformers (ViTs). Here, we report the results of transferring our method to SepViT [1] on ImageNet100/10. The results demonstrate that Dual-Arch consistently enhances the CL performance of SepViT, indicating its generality to ViTs. We have also included these results in Section A.7 of this revision.

Question1: Workflow of the testing process.

Following [2], after the learning of each task, the (main) model is evaluated on the test data of the current task as well as all previously learned tasks. We have detailed this content in Section 4.3 of this revision.

Question2: Implementation of replay.

Following [2] [3], to implement replay, parts of data from previous tasks are preserved and subsequently incorporated into the training data for all models. We have detailed this content in Section 4.3 of this revision.

Question3: Indicating the original ResNet in Table 1.

We appreciate the suggestion and have highlighted this aspect to enhance readability in this revision.

Question4: Experiment involving Pla-Net---StaNet.

We have incorporated such an experiment in this revision, and the results are as follows.

The configuration 'Pla-Net---StaNet' demonstrates inferior performance compared to our original setup, 'Sta-Net---PlaNet'. Additionally, it marginally outperforms 'Pla-Net---Pla-Net', which may be attributed to the potential benefits derived from the integration of different architectures.

[1] Li, et al. Sepvit: Separable vision transformer.

[2] Zhou et al. Pycil: A python toolbox for class-incremental learning.

[3] Rebuffi et al. icarl: Incremental classifier and representation learning.

We appreciate the reviewer's acknowledgment of our sound observation and idea, as well as detailed experiments and analysis. To further address the reviewer's concern, we have:

• provided further clarification and experiment to support our argument of a stability-plasticity tradeoff at the architecture level (for Weakness 1).
• explained why Resnet18 is selected as the baseline (for Weakness 2, Question 2, 3)
• included a further experimental evaluation on ViTs (for Question 1).
• provided an explanation of the enhanced plasticity within our proposed framework (for Question 4).

Weakness1: Concern about the stability-plasticity tradeoff at the architecture level.

We acknowledge that stability and plasticity can vary in different training environments, which is due to the impact of model parameters on the stability-plasticity tradeoff. However, it is crucial to emphasize that the architecture itself also inherently influences these properties. For instance, under the same training settings, ResNet typically exhibits greater plasticity compared to a deep MLP in most scenarios. Our study, conducted under consistent training settings, reveals that certain architectures are inherently more plastic, while others are more stable. This suggests that the architecture plays a fundamental role in stability and plasticity. We believe that while training settings and model parameters (i.e., weights) are important, they do not diminish the existence of the stability-plasticity tradeoff at the architecture level. Instead, they highlight the need for considering both architectural design and learning methods to achieve optimal performance.

Moreover, we note that the experimental results in our responses to Question 4 also support our argument of a stability-plasticity tradeoff at the architecture level.

Weakness2: Concern about the using of ResNet18 on CIFAR-100.

We appreciate the reviewer's valuable feedback. However, we believe it is not entirely accurate to conclude that ResNet-18 is over-parameterized for CIFAR100 in the context of Continual Learning (CL). First, ResNet-18 is widely adopted by many existing CL methods and benchmarks for the CIFAR100 dataset, including [1-4]. These existing works support the use of ResNet-18 for CIFAR-100 in CL, indicating that its parameter counts are suitable for achieving competitive performance. Moreover, expansion-based methods that continually expand network parameters often outperform methods with fixed parameters when using ResNet-18 as the backbone in CL. For instance, DER achieves significantly superior CL performance compared to WA and Foster on CIFAR-100, as demonstrated in Table 2 of our paper. This observation further suggests that ResNet-18 might not be over-parameterized for CIFAR-100 in CL.

Moreover, regarding Question 2, it is important to clarify that the limited performance drop in Figure 3a might be due to the significant increase in parameter counts introduced by DER, which expands from 11.2M to 112M parameters. This substantial parameter expansion sightly masks the inherent parameter efficiency of ResNet-18. To further illustrate this point, we can observe from Figure 3b of our paper that when combining ResNet-18 with Foster, an expansion-free method, the performance drops considerably with a reduction in parameter counts. This indicates that ResNet-18 is not inherently over-parameterized for the CIFAR-100/10 task. We believe that reporting the results of both representative expansion-based and expansion-free methods makes our parameter efficiency analysis comprehensive.

[1] Yan et al. Der: Dynamically expandable representation for class incremental learning.

[2] Arani et al. Learning fast, learning slow: A general continual learning method based on complementary learning system.

[3] Madaan et al. Heterogeneous Continual Learning.

[4] Sun et al. Decoupling Learning and Remembering: a Bilevel Memory Framework with Knowledge Projection for Task-Incremental Learning.

Question1: Experiments on other architectures.

While our study primarily focuses on ResNet, the insights presented in our paper could extend to other architectures, such as Vision Transformers (ViTs). Here, we report the results of transferring our method to SepViT [5] on ImageNet100/10. The results demonstrate that Dual-Arch consistently enhances the CL performance of SepViT, indicating its generality to ViTs. We have also included these results in Section A.7 of this revision.

[5] Li, et al. Sepvit: Separable vision transformer.

We appreciate the reviewer's recognition of our novel and interesting findings, the fair performance improvements, and the clarity of our paper. To further address the reviewer's concern, we have:

• provided further clarification to highlight the unique contribution of our work (for Weakness 1).
• included further experimental evaluation with additional baseline, incremental step, continual learning scenario (for Weakness 2, Question 1, 2, 3).
• proofread the paper carefully to correct typos (for Weakness 3).

Weakness1: Level of novelty.

While existing observations and techniques have indeed motivated this work, our study offers unique contributions beyond merely combining them. Specifically, this paper is the first to explicitly identify that a stable architecture limits plasticity, while a plastic one limits stability, reflecting the stability-plasticity trade-off at the architectural level within continual learning (CL). Moreover, this novel issue is addressed through our innovative approach of combining two distinct and specialized architectures, which has not been explored in prior CL works, to the best of our knowledge. This exploration and method innovation distinguish our work and contribute to the advancement of CL research.

We have also revised the Introduction to highlight the unique contribution of our work. We hope this clarification underscores the distinctiveness of our contributions and the value of our intuitive yet novel approach.

Weakness2: More thorough evaluation.

Please see our responses to the Questions.

Weakness3: Proofread.

We have corrected the typo, changing "ANN" to "AAN." We have thoroughly proofread the manuscript to address every other issue we found. Thank you for pointing this out.

Question1:Evaluation in combination with other methods

We acknowledge the importance of a more thorough evaluation. To further address this concern, we have conducted an empirical evaluation in combination with ADC[1], a state-of-the-art replay-free CL method. The results on CIFAR100/10 are presented below.

These results also suggest that, in comparison with existing architectural designs, the wider and shallower variants may offer superior stability at the cost of reduced plasticity. This trend mirrors that of Table 3 in our paper, thereby supporting our claim.

Question2:Experiments on settings with more tasks.

We are pleased to report the results on settings with a greater number of tasks (e.g., 50). Below are the results on CIFAR100/50. These results show that our method still outperforms the baselines in this more realistic setting. We have also included these results in Section A.5 of this revision.

Dear Reviewer uYip:

Thanks for your careful comments and your time for our work. We have revised our paper and added the discussion and experiments concerning.

Currently, all of your concerns can be resolved in the responses and revised version of the paper. However, as the revision deadline is approaching, we kindly request your feedback to confirm that our responses and revisions have effectively resolved your concerns. If there are any remaining issues, we would be grateful for the opportunity to address them to ensure the quality of our work.

Yours sincerely,

Authors of Paper 6928