Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning

Respected Reviewer abk8, We first thank you for your valuable and insightful feedback, and for recognizing our analysis from a Pac-Bayes perspective and proposed method. Below, we address your concerns in a point-by-point manner and welcome further discussion if anything remains unclear.

Q: Discussion on more real-world scenarios where such restrictions are applicable and realistic.
A: We agree with your perspective on the importance of realistic experimental settings and hope to address your concerns from two perspectives.

1. Evaluations on real world settings: In fact, we have included two real-world continual image classification benchmark datasets with a natural temporal evolution of visual concepts in the real world that spans a decade (2004-2014) into the experiments to include both class incremental and domain incremental settings.
2. Real-world applications: Similar to answer to reviewer u1kw, our initial intuition for limiting request frequency to the memory buffer is based on practical considerations. In real-world scenarios, such as autonomous vehicles and sensor network data classification where OCL could be applied, ensuring real-time accessibility of the memory buffer without incurring training delays is typically impractical.

Q: More analysis on how to improve the model throughput and the relationship with pre-trained models.
A: To further improve model throughput, we provided a NsCE Lite version in Appendix C.2.2, which enhances throughput by not fine-tuning the entire network. We found that, in most cases, this lightweight framework achieves comparable results to NsCE, particularly on relatively simple datasets. However, we must acknowledge that such approaches heavily rely on the selection of pre-trained models. For detailed guidelines on selecting pre-trained models, please refer to Appendix C.3.

Q: Some typos, missing citations and experimental results in main text.
A: We feel sorry for any confusion or inconvenience for you. We will cite the mentioned paper in the main text and carefully proof-read our text to ensure that no grammar mistakes or typos still exist.

Q: Reasons behind the discrepancy between $A_{AUC}$ and Last accuracy.
A: It should be noted that $A_{AUC}$ and last accuracy actually reflect different aspects of the model's performance. $A_{AUC}$ mainly assesses the real-time performance of the model, while the last accuracy measurement reflects the model's performance after processing the entire data stream. Our focus is on achieving a high-performance, anytime inference OCL model with minimal time and memory costs. Therefore, last accuracy is not our first priority. But even without employing data augmentation and knowledge distillation, our NsCE framework still achieves comparable results in last accuracy with SOTA OCL methods.

Respected Reviewer u1kw, We first thank you for your valuable and insightful feedback, and for recognizing our empirical evaluation and theoretical insights. Below, we address your concerns in a point-by-point manner and welcome further discussion if anything remains unclear.

Q: How the pre-trained model improve the learning speed
A: We first appreciate your insightful question. As stated in Section 2, we highlighted that there is a significant trade-off between effective learning and model throughput. Meanwhile, Figure 1 clearly shows that a proper pre-trained initialization enables the model to rapidly achieve good performance on continual tasks. This, in turn, reduces the need for extensive training steps to mitigate ignorance, thereby increasing the learning speed. Certainly, fully exploiting pre-trained models to improve learning speed in OCL is a very important topic, and we hope to address this in future works.

Q: A clear solution on how to resolve the issue of model ignorance when using pre-trained models.
A: First, we must acknowledge that, up until now, using an appropriate pre-trained initialization is the only efficient way we have found to resolve ignorance. However, our research also indicates that the efficacy of pre-training is influenced by various factors, including the domain alignment between the pre-training and target tasks, the volume and quality of the pre-training dataset, specific attributes of the target dataset, and the architecture of the backbone network. Generally, we posit that having insight into the expected data distribution of upcoming tasks allows for the selection of a pre-trained model trained on a similar distribution, which is a prudent and dependable approach.

Q: Why to claim that using $max()$ function is easily affected by a small number of outliers in the parameters.
A: Thanks for your suggestion. By definition, the max() function selects the highest value in the dataset. It naturally does not consider the frequency or distribution of other values, which may cause some unexpected problems in optimization or evaluations.

Q: Why a smoother classifier sometimes hampers the model's ability to perform rapid classification on the current task.
A: We have observed this phenomenon in the sensitivity analysis and ablation study. The reasons behind it are straightforward: compared to a sparse classifier, a smoother one tends to give vague predictions, which often indicate degrading performance. Therefore, it is important to ensure that $\mathcal{L}_{ce}$ remains the dominant term in optimization, while $\mathcal{L}_s$ serves more as a regularization.

Q: How the size of the memory buffer relates to the number of learning tasks?
A: Since each dataset comprises a varying number of classes, it is typically the case that datasets with more classes require the storage of more data, leading to a need for a larger memory buffer. Our experimental setup largely adheres to the parameters established by [1].

[1] Wei Y, Ye J, Huang Z, et al. Online prototype learning for online continual learning. ICCV, 2023.

Q: Is there a concern about over confidence or under confidence with this approach?
A: We appreciate your insightful question and agree that the overconfidence or underconfidence of a model's predictions is a promising aspect for understanding the model's behavior during continual learning, especially in online scenarios. In fact, model myopia, recency bias in the classifier, and even ignorance can be considered from the perspective of the model's prediction confidence. For example, the bias towards the current task can be perceived from the perspective of overconfidence, and the $\mathcal{L}_s$ we proposed can help alleviate this overconfidence. However, how to evaluate the confidence level and how existing techniques addressing issues like overconfidence will perform in OCL remains unclear and warrants further exploration. We hope to leave it our future works.

Respected Reviewer WX6u, We first thank you for your valuable and insightful feedback, and for recognizing our motivation and theoretical analysis. Below, we address your concerns in a point-by-point manner and welcome further discussion if anything remains unclear.

Q: Justification behind the limitation on the request frequency on the memory buffer
A: We address this question from three aspects.

1. Our initial intuition for limiting request frequency to the memory buffer is based on practical considerations. In real-world scenarios, such as autonomous vehicles and sensor network data classification where OCL could be applied, ensuring real-time accessibility of the memory buffer without incurring training delays is typically impractical. This is detailed in Appendix D.1.
2. When take the traditional setting ($Freq=1$ and replay a small part of data in the memory each time step), the training time is typically much longer and significantly reduce model's throughput as illustrated in Figure 9 & 10, let alone replaying the whole memory buffer at $Freq=1$.
3. Additionally, we find that using a proper pre-trained initialization and setting $Freq=1$ enables all baselines to achieve very high performance as shown in the following table. A high-frequency replay on a moderately sized memory buffer naturally resolves most issues like forgetting, myopia and even ignorance because it makes the setting more akin to an offline, non-continual one. This, to some extent, contradicts the intuition of the OCL setting in my opinion. Here, we take the performance $A_{AUC}$ on EuroSat as an example, for each time step we replay $10$ percent of the data in memory buffer:

Q: Comparison with computationally budgeted continual learning
A: We address this question from three aspects.

1. Some missing references: We first thank the reviewer for pointing out the missing related work on budgeted continual learning. We will cite those references in the revised version.

2. Difference between our analysis and theirs: The main difference is that in our setting, data is strictly treated as an online stream, meaning no revisit is allowed except for the data in the memory buffer. The starting points of the two papers are different: they aim to address the CL problem with limited training iterations, whereas we strive to train a model to match the original speed of the input data stream. Notably, in our experiments, the computational budget is much lower than the mentioned paper. In our setting ($Freq=1/50 or Freq=1/100$), the computational budget $\mathcal{C}$ allows revisiting less than $0.1$ percent of all observed data at given step (in average), whereas in their paper, this proportion is $25-50$ percent.

3. Including computationally budgeted continual learning methods in the comparison table: We take the suggestion and add some baseline methods in computationally budgeted continual learning denoted as CBCL with ACE (We find the implementation of CBCL is very close to ER). For results for larger datasets like CLEAR and ImageNet, we will leave them in the future version. It appears that the method performs slightly better than ER in our original Table 1, but it is less competitive compared to our method.

Q: The relation between the defined challenges and the usual stability-plasticity trade-off
The stability-plasticity trade-off is well-known in continual learning. Techniques to prevent forgetting often limit new knowledge acquisition. We thank the reviewer for giving us chances to clarify relationships between ignorance vs. plasticity and myopia vs. stability.

1. We first acknowledge that both ignorance and plasticity emphasize the need to improve the model’s capability to acquire new knowledge. However, our concept of ignorance extends beyond plasticity by incorporating the model's throughput. As illustrated in Figure 2 and Figure 11, although the method in [6], referred to as CCL or Distillation Chain in our work, enhances plasticity for OCL, it significantly reduces the model's throughput. This reduction potentially harms its practicality and performance, as detailed in Section 2.
2. Our examination of model myopia aims to refine the concept beyond its interpretation in the context of stability. Previous literature focusing on model stability attributes the model's decreasing performance to the interference or overlap in the feature representations or gradient contention of new and old classes, advocating for safeguarding existing knowledge to mitigate forgetting. Conversely, we contend that this performance degradation arises more from the model's limited exposure to subsets of classes during training, which causes it to favor features unique to these subsets. This leads to a myopic classification perspective with poor generalization. I think as we mentioned in our theoretical analysis (Section 4), our proposed concept of model's myopia actually offers a fresh perspective.

Overall, our defined new challenges serve as an extension or an independent aspect (the other side of the coin) compared to the traditional stability-plasticity trade-off.

Note: Due to strict space limits, our responses to the other review questions are consolidated in the Author Rebuttal above. We apologize for any confusion caused by not responding individually. Please refer to the Author Rebuttal or let us know if you have any other questions to discuss further.

Q: Justification behind $A_{AUC}$ (Area under the Accuracy Curve) but not Average Accuracy $A_{avg}$
A: At this point, we respectfully disagree with the reviewer's opinion. Compared to $A_{avg}$, $A_{AUC}$ is typically perceived as a more suitable and modern evaluation metric for the OCL scenario, especially in boundary-free settings like ours. Specifically, $A_{AUC}$ was first proposed by [1] and has been widely adopted as a crucial metric in existing OCL literature to substitute the old common used $A_{avg}$, such as [2], [3], [4]. This metric helps address the limitation of $A_{avg}$, which only provides evaluation about the model's performance at specific task transition points, usually occurring only 5-10 times in most OCL setups.

In contrast to [1], we measure accuracy more frequently—at least 20 times—by evaluating it after every $\Delta n$ samples ($\Delta n=500$ for eurosat). Instead of taking an average on these observed accuracy, we compute the area under the accuracy-to-number-of-samples curve (AUC) $A_{AUC}=\frac{1}{n}\sum_{i=1}^t acc(i \cdot \Delta n) \cdot \Delta n$ (actually similar to the average taken on more observations $A_{avg}=\frac{1}{t}\sum_{i=1}^tacc(i\cdot \Delta n)$), providing a more precise evaluation than simply taking average accuracy ($n$ denotes the total number of training data). As illustrated by the following table (we directly copy the performance of RM[5] in [1]), a higher $A_{AUC}$ typically induces higher $A_{avg}$ and good real-time inference performance, but a high $A_{avg}$ does not necessarily represent good real-time inference performance. We highly respect reviewer's suggestion and plan to include a more detailed discussion including the comparison between these two metrics (including both numerical results and illustrations) in our revised version.

[1] H. Koh, D. Kim, J.-W. Ha, and J. Choi. Online continual learning on class incremental blurry task configuration with anytime inference. ICLR, 2022.
[2] Moon J Y, Park K H, Kim J U, et al. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning ICCV 2023.
[3] Koh H, Seo M, Bang J, et al. Online boundary-free continual learning by scheduled data prior. ICLR, 2023.
[4] Seo M, Koh H, Jeung W, et al. Learning Equi-angular Representations for Online Continual Learning. CVPR, 2024.
[5]Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. CVPR, 2021

Q: Readability and typos (especially on figure 2 and table 1)
A: We feel sorry for any confusion or inconvenience for you. We take your suggestions on the readability of figures. We will enlarge the captions of our figures to ensure a better readability. Plus, we will carefully proof-read our text to ensure that no grammar mistakes or typos still exist.

Q: How the proposed approach solves the introduced Myopia could be improved
A: We first appreciate your insightful question and suggestion to create a graph showing the weight sparsity with and without the sparsity-regularization term. We have included this in Figure 1 of the attached PDF. It can be seen that the proposed sparse regularization term $\mathcal{L}_s$ prevents the classifier from becoming excessively sparse during training. Additionally, we want to clarify that myopia is addressed by the overall design of our proposed method including the sparsity-regularization term, NsCE, as illustrated in Figure 5 of our main text. From the visualization, it can be seen that our model quickly learns the current task while minimizing confusion between past categories and those in the current task.

Q: The relation between the defined challenges and the usual stability-plasticity trade-off
A: The stability-plasticity trade-off is one of the most well-known concepts in continual learning. It is recognized that techniques to mitigate catastrophic forgetting often constrain a model’s capability to acquire new knowledge. Some recent studies also perceive these as independent challenges, as the reviewer mentioned in [6]. We thank the reviewer for giving us the opportunity to further clarify the relationship and differences between ignorance vs. plasticity and myopia vs. stability.

1. ignorance vs. plasticity: We first acknowledge that both ignorance and plasticity emphasize the need to improve the model’s capability to acquire new knowledge. However, our concept of ignorance extends beyond plasticity by incorporating the model's throughput, a critical factor in high-speed data stream environments. We observe that the single-pass nature of OCL challenges models to learn effective features within constrained training time and storage capacity, leading to a trade-off between effective learning and model throughput. As illustrated in Figure 2 and Figure 11, although the method in [6], referred to as CCL or Distillation Chain in our work, enhances plasticity for OCL, it significantly reduces the model's throughput. This reduction potentially harms its practicality and performance, as detailed in Section 2.
2. myopia vs. stability: Our examination of model myopia aims to refine the concept beyond its interpretation in the context of stability. Previous literature focusing on model stability attributes the model's decreasing performance to the interference or overlap in the feature representations or gradient contention of new and old classes, advocating for safeguarding existing knowledge to mitigate forgetting. Conversely, we contend that this performance degradation arises more from the model's limited exposure to subsets of classes during training, which causes it to favor features unique to these subsets. This leads to a myopic classification perspective with poor generalization. I think as we mentioned in our theoretical analysis (Section 4), our proposed concept of model's myopia actually offers a fresh perspective to understand the performance degradation in OCL.

Overall, our defined new challenges serve as an extension or an independent aspect (the other side of the coin) compared to the traditional stability-plasticity trade-off.

[6] Wang M, Michel N, Xiao L, et al. Improving Plasticity in Online Continual Learning via Collaborative Learning. CVPR, 2024.

Q: Whether the sparsity would be affected by methods that focus on re-arranged last layer weight updates

A: After the reviewer's reminder, we are also very interested in how sparsity would be affected by methods focusing on re-arranged last layer weight updates. After implementing ER-ACE and ER-AML, we found that the phenomenon of parameters rapidly becoming sparse is indeed somewhat mitigated, though not as significantly as with our proposed regularization term $\mathcal{L}_s$, as illustrated in Figure 1 of the attached PDF. Additionally, incorporating ACE or AML can also boost performance for baselines like ER and SCR. We believe that when ACE and AML nudge the learned representations to be more robust to new future classes, they indirectly decrease the sparsity of the model parameters.

For GSA, the sparsity is not affected. Due to very limited time, we are not entirely sure whether this part is perfectly embedded or if further tuning would help, as the authors only provide hyperparameters for CIFAR-100 and there are some differences between the original paper's general settings and ours. For SS-IL, we did not find its implementation, so it may be left for future works.

Overall, re-arranging the latter layer weight updates is an interesting and important problem in the area of OCL. We genuinely appreciate this insightful suggestion and plan to conduct detailed exploration of the dynamics of latter layer weight parameters during training in future works. This may lead to more intuitive designs.

I really appreciate the time and effort the authors have put into their work and rebuttal. I genuinely found this discussion very interesting and I believe this work to be of high quality.

About the $Freq=1$ scenario

I appreciate the authors honesty. My intuition was indeed that the proposed work performance gain might not be as significant in this setup. That being said, I fully agree that limiting the frequency of replay makes perfect sense and should be a prior focus compared to the $Freq=1$ scenario.

Comparison with BudgetCL

I agree with the authors comments. Thank you for the clarification.

Discussion on the link with stability-plasticity

Thank you for the clarifications.

Extra experiments

Thank you for sharing your findings. I also agree with your interpretation. I found these experiments particularly interesting and I believe them to be valuable to the community. I think such experiments would be worth including in the main draft.

Sparsity metric

Thank you for defining $\frac{1}{s(w)}$ which I believe makes more sense to me and is more easily understable. I apologize for suggesting $1-s(w)$ which of course was not suited.

On the usage of $A_{AUC}$

Thank you for the interesting reference. I still believe that the final average accuracy is valuable and could be reported in appendix, and I thank the authors for including it in the rebuttal. If I am not mistaken, I believe it is not currently included. In any case, I was convinced by the authors arguments and the usage of $A_{AUC}$ now makes perfect sense to me. Again, I appreciate the effort of clarifying and justifying the choices made in this work. After carefully checking the manuscript, I realize such choices were already justified in the appendix.

For all the above reasons, I will increase my score to 8.