Adaptive Continual Learning: Rapid Adaptation and Knowledge Refinement

Thank you for your comprehensive feedback and insightful suggestions. We appreciate your suggestions to improve clarity, and suggestions on experiments. In this rebuttal, we aim to address all your concerns comprehensively.

Experiments

Q: “compare … very related setting where data of previous tasks/classes may appear again in CL…”

A: We appreciate the concerns. During the rebuttal stage, we add methods CLS-ER[3], CLIB [5], and DualNet [6]. In summary, these methods exibit less satisfactory performance in the results attached at the end. While this setting seems to be similar, we hope to emphasize that papers such as [1,2] focus on changing class distribution, where the key challenge is to handle possible class imbalance. However, our proposed scenario considers possibly recurring tasks, highlighting the need to rapidly recognize seen tasks and refine corresponding information.

Q: “Inappropriate choice of baselines in the experiments.”

A: To better clarify our choice of baseline methods, we want to mention that our selected methods (EWC++, MIR, ER, GDumb, AGEM) align with recent task-free research conventions [4,5,6,7]. These baselines are especially suitable for task-free scenarios because they do not require explicit task identity. For example, [7] used MIR, ER with GMED [8] variant; Similarly, the mentioned [4] used ER, MIR, GDUMB, and GMED, supplemented with methods designed for class imbalance – CBRS (Class-balancing reservoir sampling), ACE.

Q: “What are Average Accuracy and Knowledge Accuracy…”

“in Table 2, the only task-free baseline outperforms…”

A: Thanks for the question. For better clarity, we have included the formulas for Average Accuracy and Knowledge Accuracy both in this rebuttal and the revised manuscript Average Accuracy measures the cumulative performance over time, and Knowledge Accuracy measures the information retained only at the last timestep, assuming task identity is known. Therefore, the method such as ExpVAE, which separately trains a classifier per task, has a high Knowledge Accuracy, while poor Average Accuracy since inference by VAEs is inaccurate. $\text{AverageAccuracy} = \frac{1}{T}\sum_{t=1}^T acc_t, \quad \text{Knowledge Accuracy} = \frac{1}{N} \sum_{i=1}^N acc_{T,i},$ where $acc_t$ is the accuracy at time $t$, and $acc_{T,i}$ is the test accuracy of task $i$ assuming task identity is known at the final time.

We want to emphasize that as indicated in Compared methods, Section 4.1, the baseline ExpVAE requires task identity during training, which separately trains a pair of generator and classifier for each task. In the inference stage, the generator is used to choose the correct classifier. ExpVAE serves as an upper bound on how such a structure of pairs of generators and classifiers will behave, even if task identities are known during training. As seen from the average accuracy in Table 1, inference from generator is inaccurate. Table 2 evaluates the final average test accuracy of the best classifier per task. Therefore the Knowledge accuracy of ExpVAE behaves similarly to Oracle which employs a multi-head ResNet with shared feature extractor, while during inference generators fail to correctly recall the correct classifier.

Q: “random shuffling of 200 segments can have a significant impact on experimental performance.”

A: Thanks for the insightful point. In order to avoid performance being significantly affected by randomness, we repeated experiments for 10 runs and reported the standard error. Since the standard error is moderate, we believe the reported results reflect the performance.

Number of slow learners.

Q: “Algorithm 2 needs to maintain $1+m_t$ models…almost the same number of trainable parameters as the other methods”

A: We appreciate your observation. As mentioned in Architecture, Section 4.1, our proposed method and Oracle employ a reduced ResNet18 with a shared feature extractor. Therefore the excessive $m_t$ output heads contribute a small number of parameters compared with the feature extractor.

Q: “How is $m_t$ obtained in Algorithm 2”

A: Thanks for the excellent question. We want to clarify that slow learners are sequentially added once patience is above a threshold, and $m_t$ is the number of slow learners maintained at time $t$, which is not the underlying truth. To avoid confusion, we have revised the notation to $\hat{m}_t$ in the manuscript.

Q: “How many slower learners are there at the end of training”

A: Thanks for the helpful suggestion. Along with the maximum trainable parameters of our proposed algorithm reported in Table 4 (Section 4.1), the final count of slow learners at the end of training is detailed in Table 4, attached to the rebuttal.

Thank you for your insightful comments. We are encouraged that you think the new scenario is practical and the proposed algorithm is technically sound and empirically verified. In the response, we aim to address the concerns adequately.

Regression task

Q: “whether it is also applicable for regression problem”

A: Thanks for the insightful point. Although our experiments focused on classification tasks, a primary focus in computer vision, the proposed algorithm is applicable to any supervised learning problems including regression. In the regression case, the label is output which is usually a scalar and loss is typically the mean square error. The exact theoretical guarantees for the supervised case in Section 3 directly hold for such a scenario.

Ablation study

Q: “...considering each process only”

A: Thanks for the helpful suggestion. We acknowledge that the exploration-only version serves as a useful baseline. In response to your query about considering each process only, we have updated the original Figure 5 in Section 4.2. From the figure, we find exploration-only version behaves worse than no-refinement and no-recall. Such a phenomenon is reasonable because while fast learner explores without memory the same as Finetune, slow learners never refine their knowledge and fast learner does not recall any knowledge from slow learners. Therefore, the exploration-only version quickly forgets learned tasks similar to Finetune.

Choice of hyperparameters

Q: “...constructive guideline to set the two hyper-parameters”

A: Thanks for the excellent feedback. From the insights in theoretical guarantees, $\alpha$ is the proportion of slow learners mixed with fast learner, and $Q$ is the threshold to detect whether to create a new slow learner. Empirically, we find that setting $Q$ between $5$ and $20$, and $\alpha$ between $0.1$ and $0.2$, are good starting choices. For further optimization of the hyperparameters, conducting a grid search on the initial batches can yield more refined settings.

Warm regards,

Authors of Adaptive CL

As shown in Table 1 and 3, CLS-ER performs slightly worse than ER, likely due to its reliance on an EMA model that may not adapt as swiftly to task switches. CLIB, designed for changing class distributions, maintains a memory buffer with sample importance, which is less applicable in our scenario where class imbalance is not a primary concern. While CLIB outperforms GDumb which also updates solely on memory, the average accuracy is not comparable to other methods. DualNet trains a fast model with features from prior tasks learned with semisupervised learning. From Table 1 and 3, we find the fast model cannot adapt to seen tasks as fast as our proposed light router structure. Table 2, focusing on knowledge accuracy, reveals that both CLS-ER and CLIB exhibit better information retention than Finetune, but are not comparable to our proposed method and Oracle with multiple models.

Tabel 1: Final average accuracy from 10 runs.

Table 2: Knowledge accuracy from 10 runs

Table 3: Adaptiveness from 10 runs

Table 4: Number of slow learners (output heads) at the end from 10 runs.

[1] Koh H, Seo M, Bang J, et al. Online Boundary-Free Continual Learning by Scheduled Data Prior, ICLR, 2022.

[2] Xu, Zhenbo, et al. Revealing the real-world applicable setting of online continual learning. MMSP, IEEE, 2022.

[3] Arani, Elahe, et al. Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System. In ICLR 2021.

[4] Chrysakis, Aristotelis, et al. Simulating Task-Free Continual Learning Streams From Existing Datasets. In CVPR 2023.

[5] Koh et al. Online continual learning on class incremental blurry task configuration with anytime inference. In ICLR 2022

[6] Pham et al. DualNet: Continual Learning, Fast and Slow. In NeurIPS 2021

[7] Kumari, Lilly, et al. Retrospective adversarial replay for continual learning. In NeurIPS 2022

[8] Jin, Xisen, et al. Gradient-based editing of memory examples for online task-free continual learning. In NeurIPS 2021

Thank you for your comprehensive and constructive feedback. We endeavor to address each of your concerns thoroughly in the subsequent response.

Novelty

Q: “...already some papers proposed similar periodic/recurring CL tasks [1,2]”

A: Thanks for bringing up the relevant papers. Although these papers present scenarios that appear similar, the underlying challenges and corresponding solutions they address differ significantly from ours.

These papers consider the scenario where class distributions change gradually or periodically over time. Thus the main challenge of this scenario is to learn without paying too much attention to the classes that frequently occur recently. Therefore, [1,2] focus on methods that can balance class distribution, such as an EMA (exponential moving average) of working models [1] and asymmetric cross entropy (ACE) to handle class imbalance [2].

Our proposed Adaptive CL tackles a sequence of possibly recurring tasks without known task identities. The key challenge is to quickly recall relevant information when a prior task reoccurs, and accumulate knowledge over time. This challenge motivates us to develop a unique structure combining a single fast model, multiple slow models, and a router, specifically designed to rapidly recall relevant information for recurring tasks.

Experiments

Q: “the methods chosen for comparison are outdated.” “compared with closely related methods such as Cls-ER[3] and recent task-free methods like [4]”

A: Thanks for your excellent suggestions. During the rebuttal stage, we add methods CLS-ER[3], CLIB [5], and DualNet [6]. Since the focus of [4] is to propose a way to simulate a scenario with changing class distribution, [4] does not propose a new task-free algorithm. We show that these 3 new methods have less satisfactory performance. The results are discussed and attached at the end.

To better clarify our choice of baseline methods, we want to mention that our selected methods (EWC++, MIR, ER, GDumb, AGEM) align with recent task-free research conventions [4,5,6,7]. These baselines are especially suitable for task-free scenarios because they do not require explicit task identity. For example, [7] used MIR, ER with GMED [8] variant; Similarly, the mentioned [4] used ER, MIR, GDUMB, and GMED, supplemented with methods designed for class imbalance – CBRS (Class-balancing reservoir sampling), ACE.

Q: “address the storage cost”

A: Thanks for your feedback. As indicated in Table 4, Section 4.1, the additional storage cost incurred by our algorithm is minimal. Since we adopt a multi-head ResNet18 with a shared feature extractor, adding more heads introduces a small additional cost compared with the large shared feature extractor

Clarity

Q: “In Figure 1, the meaning … unclear”

A1: Thanks for the helpful suggestion. To address this, we have revised Figure 1 in Section 1 to clearly depict our proposed network structure, showcasing the integration of fast/slow models and the router.

Q: “scalability challenges refer to?”.

A: Thanks for pointing out. The term 'scalability challenges' in our context refers to the computational difficulties of managing probability distributions over a high-dimensional parameter space in previous algorithms. We have updated the expression in Section 3.3 to “However, this approach, which requires density over a high dimensional space, faces scalability challenges in large-scale deep learning tasks.”

Q: “In Algorithm 2, $\beta$ is also not explained.” “not clear why the slow learners have the correct $m_t$ models” “...more detailed explanation of how the GMM model is updated”

Thanks for the excellent critic. Algorithm 2 involves three components at time $t$: (1) fast model $\theta_t$ (2) slow models $\beta_{t-1,i}$ for $i=1,\dots,m_{t-1}$ (3) mixing weights $w_{t,i}$ to combine fast/slow models. The $m_t$ is the estimated number of models in Algorithm 2. In Algorithm 2, when a fast learner consistently outperforms slow learners, it signals the emergence of a new task, prompting the addition of the fast learner as a new slow model. To improve clarity, in the updated manuscript, we will (1) move the key GMM approximation steps in the Appendix to Section 3.3, and (2) change $m_t$ to $\hat{m}_t$ to avoid confusion.

Thank you for your insightful suggestions and comments. We are pleased that you find the paper well-written and proposed Adaptive CL interesting. In this rebuttal, we aim to address all your concerns comprehensively.

Clarity.

Q: “The actual network of the proposed model is not clear.”

A: Thanks for your suggestions. We have updated Figure 1 to clearly illustrate the actual network architecture, comprising a fast model, multiple slow models, and a router that keeps a mixing weight to combine these models. Our algorithm guarantees that the router can (1) assign a high weight to the relevant slow model when a prior task reoccurs, or to the fast model when a new task arrives; (2) update the corresponding model without information overwrite, avoiding the case where a model already mastered one task continue to learn another and forget mastered one.

Choice of Regret bound.

Q: “Why use the Regret Bound to explain the proposed approach instead of other theories?”

A: Thanks for mentioning the choice of regret instead of test accuracy in the end. In the online scenario, it's crucial to assess cumulative performance at each timestep, rather than just the final outcome. Therefore, regret, which measures the difference between the cumulative performance and the cumulative best performance when given the truth, is a standard metric in the literature of online learning.

Computation cost.

Q: “The proposed approach requires three steps in each time”

A: Thanks for the excellent question. The router update, involving merely a one-dimensional vector, is computationally light. The major computations are the backpropagation of the fast model and several slow models. Similar to other methods involving multiple models [1,2], in the experiment, we keep the fast and slow models light – a multi-head ResNet18 with a shared feature extractor, so that the trainable parameters align with the method involving a single network. Given that each model is designed to handle a specific task, employing light models is both efficient and effective.

[1] Lee, Soochan, et al. "A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning." International Conference on Learning Representations. 2019.

[2] Ye, Fei, and Adrian G. Bors. "Task-free continual learning via online discrepancy distance learning." Advances in Neural Information Processing Systems 35 (2022): 23675-23688.

The following 4 tables report mean $\pm$ standard error from 10 runs.

Tabel 1: Final average accuracy from 10 runs.

Table 2: Knowledge accuracy from 10 runs

Table 3: Adaptiveness from 10 runs

Table 4: Number of slow learners (output heads) at the end from 10 runs.