Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective

We sincerely thank Reviewer 2FFZ for the recognition of our insightful empirical findings and interesting idea. We are also grateful for the valuable and constructive feedback.

Conclusions regarding deeper network

Our main conclusion is that "existing architectural designs typically exhibit good plasticity but poor stability, while their wider and shallower variants exhibit the opposite traits" (contribution 1). We acknowledge that further increasing depth further does not significantly boost plasticity, so our plastic network's depth aligns with ResNet-18 (Sec. 4.3). We'll revise Sec. 3.2 to clarify this and avoid ambiguity.

Relative forgetting

While the Average Forgetting is widely used, we agree that relative forgetting (RF) [1] offers a fairer stability measure. We've added RF to our analysis. Results on Tab. 1 (partial) support our original claims.

Essential References

We thank the reviewer for highlighting these relevant works. We've updated Sec. 2.3 to include a discussion of [2] and [3]. Specifically: (italicized indicates new content):

"..., certain works utilize two learners (known as slow and fast learners) with different functions to achieve CL. Among them, MKD [2] and Hare & Tortoise[3] employ techniques such as exponential moving averages to integrate knowledge across two models, effectively balancing stability and plasticity. Our proposed solution ..."

Computation time

Training: Despite reduced FLOPs (A.1), training time may increase due to non-parallelism. We note that this common trade-off in multi-model methods (e.g., [1]) is justified by the performance gains. As shown in the below (CIFAR100/10, minutes), our methods require 1.39× to 1.77× time cost for training. We'll include this analysis in this revision.

Inference: Using only the lightweight main model during inference, our method reduces computational cost, showing a 76% faster speed.

Training procedure

As suggested, we'll include pseudo-code to clarify the training and testing procedure. Brief process is as follows:

1. Initialize the stable main learner $\theta_0$ and plastic auxiliary learner $\phi_0$).
2. For each new task t (1 to N):
   • Train plastic learner: Update $\phi\_{t-1}$ for E epochs using normal classification loss to get $\phi_{t}$, then freeze and save it.
   • Train stable learner: Update $\theta_{t-1}$ for E epochs with the plastic learner's assistance and CL method (loss Eq. 1) to get $\theta_t$.
   • Test stable learner: Evaluate $\theta_t$ on all previous tasks (1 to t).

We clarify that our method is fully transparent, though some details may need elaboration:

1. Sequential training of two models is mentioned in Sec. 4.4 (lines 257-258);

2. Fairness is ensured by comparing total FLOPs of both models against single-model baselines in A.1.

Model ensemble

We tried ensemble techniques (e.g., exponential moving average) but found them challenging to implement across differing architectures. We believe they hold promise for integrating knowledge and will explore them in our future work.

Performance of swapping sta and pla networks

The stable and plastic learners have distinct roles (main and auxiliary) in learning, so swapping their architectures degrades performance. This shows the need for dedicated architectures suited to each learner's function.

Parameters count in Sec. 5.4

"87%" and "81%" refer to Fig. 3, where our method remain effective under stricter parameter constraints.

Dynamic Architecture in Fig. 2

"Dynamic Architecture" refers to integrating our method with dynamic architecture techniques by expanding the stable learner’s network only. This has been clarified in the revision.

Hyper-parameters (HP)

For fairness and reproducibility, we use the same HP for both baseline and baseline+DualArch without specific tuning. HPs for CL methods follow the PyCIL library [4], with epochs and optimizer scheduler adjusted for consistency across all methods, as noted in Sec. 5.1.

References:

[1] Improving plasticity in online continual learning via collaborative learning. CVPR 2024.

[2] Rethinking Momentum Knowledge Distillation in Online Continual Learning. ICML 2024.

[3] Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks. ICML 2024.

[4] Pycil: a python toolbox for class-incremental learning. 2023

We sincerely thank Reviewer Kz16 for the recognition of our intriguing research perspective and interesting idea. We are also grateful for the valuable and constructive feedback.

Computation overhead.

We would like to clarify that the total FLOPs of two models (Sta-Net and Pla-Net) in Dual-Arch is lower than the baseline, as detailed in Appendix A.1 (lines 614-617). For instance, on CIFAR100, Sta-Net and Pla-Net require 255M and 241M FLOPs, respectively, totaling 496M—less than the 558M FLOPs of ResNet-18. We thank the reviewer for the valuable suggestion and will emphasize this point in the revised main text for clarity.

Training time: While the total FLOPs are reduced as shown in A.1, we acknowledge that the actual training time may increase due to non-parallel training. We note that this trade-off is common in multi-model methods (e.g., [1]) and is justified by the performance gains. As shown in the below CIFAR100/10 experiments, our methods require 1.39× to 1.77× time cost for training. We will include this analysis in this revision.

Inference time: Since our approach utilizes only the main model (a stable learner) during inference, which is more lightweight compared to the baseline ResNet-18, the computational cost during inference is significantly reduced. Below are the results on ImageNet, demonstrating that our method achieves a 76% increase in inference speed.

Long task sequences and large datasets.

Large datasets: CIFAR100 and ImageNet100 are common benchmarks in continual learning (CL) research[1-3]. To further address the reviewer's concern, we extend our evaluation to large-scale ImageNet-1K/10 tasks. Following [4], we use the 32×32 downsampled ImageNet-1K for efficiency, with a replay buffer size of 20,000 as in [5]. Results show our method remains effective, achieving gains of +3.37% and 5.11%, indicating its strong scalability.

Long task sequences: Tab. 7 (Appendix A.5) demonstrates our method’s superior performance on CIFAR100/50, a benchmark with long task sequences (50 tasks). The results below highlight consistent improvements of our method over baselines.

Forgetting metrics.

We would like to clarify that (final) average forgetting (FAF) is a well-established metric in continual learning for measuring stability [4, 5]. Regarding the reviewer's suggestion to compare the model's performance against a universal upper bound, we respectfully note that such an evaluation may represent overall performance rather than forgetting. Since the upper bound is shared across methods, the resulting metric would primarily reflect final/last accuracy (LA), which inherently combines both plasticity (AAN) and forgetting (FAF), as approximated by $LA \gtrapprox AAN - FAF$.

Initialization of networks.

All models are randomly initialized, consistent with baseline methods. We will add a pseudo-code in this revision to clarify this point. Here is a brief process description:

1. Randomly initialize the stable learner (main model, $\theta_0$) and plastic learner (auxiliary model, $\phi_0$).
2. For each new task t (1 to N):
   • Train plastic learner: Update $\phi\_{t-1}$ for E epochs using normal classification loss to get $\phi_{t}$, then freeze and save it.
   • Train stable learner: Update $\theta_{t-1}$ for E epochs with the plastic learner's assistance and CL method (loss Eq. 1) to get $\theta_t$.
   • Test stable learner: Evaluate $\theta_t$ on all previous tasks (1 to t).

References:

[1] Rethinking Momentum Knowledge Distillation in Online Continual Learning. ICML 2024.

[2] Resurrecting old classes with new data for exemplar-free continual learning. CVPR 2024.

[3] Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence 2023

[4] Loss of plasticity in deep continual learning. Nature 2024.

[5] Pycil: a python toolbox for class-incremental learning. 2023.

[6] A comprehensive survey of continual learning: Theory, method and application. IEEE TPAMI 2024.

[7] Class-incremental learning: A survey. IEEE TPAMI 2024.

We sincerely thank Reviewer xKgH for the valuable and constructive feedback.

Generalizability of architectural insight / Concern about supplementary material.

There may be a misunderstanding regarding Tab. 5 and 6 results:

• Shallower yet wider ViT (5×49, Tab. 5): Lower AAN/FAF value → reduced plasticity, improved stability .
• Shallower yet wider MLP (3×1050, Tab. 6): Lower AAN/FAF value → reduced plasticity, improved stability .
• Deeper yet narrower MLP (5×680, Tab. 6): Higher AAN/FAF value → improved plasticity, reduced stability .

These align with ResNet trends (Table 1), supporting our architectural insights (depth -> plasticity, width -> stability).

Performance gains in Tab. 4.

We believe that the performance gains on ViT (minimum +2.47%, maximum +5.88%) is not very modest, especially considering the fact that our method reduces 30% parameter counts.

Concern about dataset limitations.

CIFAR100 and ImageNet100 are common benchmarks in continual learning (CL) research[1-3]. To further address the reviewer's concern, we extend our evaluation to large-scale ImageNet-1K/10 tasks. Following [4], we use the 32×32 downsampled ImageNet-1K for efficiency, with a replay buffer size of 20,000 as in [5]. Results show our method remains effective, achieving gains of +3.37% and 5.11%, indicating its strong scalability.

Concern about primary performance driver.

We clarify two potential misunderstandings regarding the performance drivers of Dual-Arch.

a) The auxiliary (plastic) model’s weights ($\phi_t$) are not reinitialized but are updated incrementally from the previous task’s weights ($\phi_{t-1}$), as shown in Fig. 2. So the performance gains could not stem from the plasticity of a freshly initialized model. We will further clarify this in Sec. 4.4 of this revision.

b) Dual-Arch's efficacys stems from two key components as mentioned in Sec. 5.3 (lines 324-325):

1. Dual networks (stable and plastic learner)
2. Dedicated architectures (Sta-Net as stable learner, Pla-Net as plastic learner)

While dual Sta-Nets perform strongly (still worse than Dual-Arch), this does not weaken our core proposition. Instead, it highlights the importance of:

• Dual networks (dual Sta-Nets still involve stable and plastic learner)
• Dedicated architectures (Sta-Net as the stable learner)

To further validate the importance of dedicated architectures, we compare against dual ResNet-18 networks. Results (AIA %) show that our specialized architectures achieve better performance with fewer parameters.

Why iCaRL was chosen?

Simpler baselines like Experience Replay (ER) may not fully suit CL evaluation—particularly stability. A key limitation is that without additional CL strategies, performance on old tasks might not primarily rely on model's knowledge preservation.

To investigate this, we introduce an ablation on CIFAR100/10: "w/ buffer only", where models:

• Train only on the last five tasks,
• Use an initial replay buffer from the first five tasks,
• Evaluate performance on the first five tasks after learning the final task.

Results show the ablation achieves relatively comparable performance on tasks 1–5 to original ER, despite no direct learning. In contrast, iCaRL shows a larger gap (7.24% vs. 2.10%), indicating the model preserves knowledge beyond buffer memorization. This highlights iCaRL’s suitability for evaluating CL architectures beyond ER.

Theoretical explanation.

While theoretical analysis (e.g., NTK or feature learning) could offer deeper insights, it is beyond this work’s scope. Existing theories often rely on idealized assumptions (e.g., infinite/very large width or two-layer networks), making extensions to our empirical width/depth trade-offs highly non-trivial.

References:

[1] Rethinking Momentum Knowledge Distillation in Online Continual Learning. ICML 2024.

[2] Resurrecting old classes with new data for exemplar-free continual learning. CVPR 2024.

[3] Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence 2023.

[4] Loss of plasticity in deep continual learning. Nature 2024.

[5] Pycil: a python toolbox for class-incremental learning. 2023.

We sincerely thank Reviewer MGyK for the recognition of our insightful research perspective, simple yet effective method, and impressive performance improvements. We are also grateful for the valuable and constructive feedback.

Validation on replay-free settings.

While our current experiments primarily focus on replay-based continual learning settings, our approach is also compatible with state-of-the-art replay-free methods. To demonstrate this, we evaluate our approach in conjunction with three representative replay-free continual learning methods: LWF [1], ADC [2], and LDC [3] as representative replay-free methods. The results (LA /%) consistently show that our Dual-Arch approach enhances the performance of all baseline methods, underscoring its versatility and effectiveness in replay-free continual learning scenarios.

Architecture-based methods.

We appreciate the reviewer’s valuable comments and will revise Sections 2.1 and 2.2 to clarify the distinction between two key concepts. To clarify, architecture-based methods primarily focus on dynamically expanding or allocating networks for each task based on a given basic architecture. In contrast, "Neural Architecture for CL" refers to studies that explore and optimize the basic architecture itself to better suit continual learning objectives.

Parameter counts rules.

We thank the reviewer for raising this point. To avoid any confusion, we will update the description to explicitly state that the reported parameter counts include all used networks (not just the main network).

Typo issues and Figure presentation.

We sincerely appreciate the reviewer’s attention to detail. All mentioned typos have been corrected, and we have increased the font size in Figure 3 for better readability. We will thoroughly proofread the entire paper to ensure clarity and correctness.

References:

[1] Li, Zhizhong, and Derek Hoiem. Learning without forgetting. IEEE TPAMI 2017.

[2] Goswami, Dipam, et al. Resurrecting old classes with new data for exemplar-free continual learning. CVPR 2024.

[3] Gomez-Villa, Alex, et al. Exemplar-free continual representation learning via learnable drift compensation. ECCV 2024.