Label Delay in Online Continual Learning

We thank the reviewer for their efforts reviewing our paper and that the reviewer finds our method to be simple and that they agree on the setup of delayed labels.

Q1: the challenge of matching input data at time step 𝑡 with labels at time step 𝑡+𝑑 perfectly may prove more difficult than acquiring a data stream with instantaneous annotations.

Throughout our work we argue that modeling the delay is essential in OCL settings, and we found that a straightforward experimental model for demonstrating how such delays impact the model performance is the ($x_t, y_{t-d}$) formulation.

Throughout our work, we argue that modeling delay is crucial in OCL settings. We found that a straightforward experimental model, using the (x_t, y_{t-d}) formulation, effectively demonstrates how delays impact model performance. Decoupling the input and label streams could be an interesting generalization, where data and labels arrive asynchronously. We excluded this to maintain focus on our main message. Including this discussion in future work would enhance the manuscript. However, we highlight the numerous experiments varying delay times (as in Figures 2, 3, 4), demonstrating IWMS's superiority over other methods. We do not expect performance rankings to differ significantly with variable delays.

Q2: The proposed setting is not clearly formalized

We will review our manuscript to ensure the notations and explanations are as clear as possible and will add clarifications where necessary to prevent misunderstandings. Feedback from other reviewers (bdVn, dj2M, and HwKE) indicated that our presentation was clear, but we understand that improvements can always be made. Please let us know any specific comments the reviewer has in mind.

The Annotator does not provide the samples, only the labels. The scenario, to which the reviewer refers to, in which we simply wait for the annotator to reveal the labels and ignore the delays, is described as the Naïve case as this is the simplest and in practice the most popular approach.

Q3: part 1: Unclear notations. The notation and evaluation criteria in the paper are somewhat unclear. We will review our manuscript to ensure the notations and explanations are as clear as possible and will add clarifications where necessary to prevent misunderstandings. Please let us know any specific comments you have in mind.

We appreciate the reviewer's feedback on the evaluation criteria and the suggestion to include more comparisons with existing Online Continual Learning (OCL) methods. However, we would like to clarify both the original OCL formulation ([Cai] in Figure 3c and our manuscript does indeed address the evaluation of forgetting and performance degradation. Specifically, these aspects are thoroughly considered and illustrated in Figure 4 of the main paper, which evaluates the model's performance across all previously encountered data distributions. We apologize if this was not immediately clear and will ensure that the manuscript highlights these evaluations more prominently.

OCL papers often focus less on forgetting and more on online accuracy, i.e., next batch accuracy, because the data stream is chronologically ordered. In this setup, older data (e.g., from 2010) always precedes newer data (e.g., from 2024) and might never reappear. For example, images of Nokia phones are far less common now than those of iPhones. Thus, the goal is to predict future data accurately. Despite this, we still report forgetting and past data accuracy, detailed in the supplementary material.

Q3: part 2: A comparison with these online learning with label delay

We would like to point the reviewer to the supplementary material, which explains the primary reasons for the incompatibility of traditional online learning approaches with modern OCL settings. Briefly, the main issue is the general lack of information retention mechanisms in traditional methods, which are crucial for addressing the complexities of continual learning tasks. Traditional online learning methods often assume immediate access to labels and do not account for the challenges of learning from data streams with delays. In contrast, modern OCL approaches, including our proposed framework, focus on retaining and utilizing past information effectively to mitigate forgetting and adapt to new data distributions. We have highlighted these distinctions in our paper and supplementary material to clarify why direct comparisons with traditional methods may not fully capture the unique challenges and contributions of our work. However, we will ensure that these discussions are more clearly articulated in the manuscript.

Q4: More comparisons with existing OCL Could the reviewer point out which exact method should be included in our comparisons? Please note we used the best performing (under computationally budgeted) OCL as shown recently in [Ghunaim et al.].

Q5: No convincing comparison to [A, B]

We appreciate the reviewer's suggestion to improve our comparison with prior work. Figures 16 and 17 in the Supplementary Material visually explain the core difference with the framework considered by [B]. In our framework, all labels become available after a fixed duration, making it fully supervised. Gomez et al. do not consider unsupervised and semi-supervised methods for cases where newer data distribution states are accessed using unsupervised techniques, which is our core contribution. We'll emphasize this distinction in the main paper.

Regarding [A], we address the incompatibility issues of such online learning work from 2005 in the related work section, L66. The methods proposed by [A] describe how to wait for labels that may come in a different order than the data points, involving sorting labeled instances or updating on the most recent labels. Our new experiments show that simply rehearsing on the newest supervised data, as [A] suggests, leads to performance collapse (i.e., per chance accuracy).

We thank the reviewer for their feedback.

We fully agree with the claim that the significance of these metrics is highly dependent on the exact application. In real world applications often times the final metric is high level, such as "customer satisfaction" or "viewer retention rate" that is difficult if not intractable, therefore these quantitative metrics are just proxies for such higher level objectives. What the research community deems important and relevant can, and indeed does change over time. Our message is simple: "regardless of the choice of the metrics, label delay is a very common problem that needs to be addressed". We truly appreciate the reviewer pointing out the dependency of the relative significance on the specifics of the actual problem.

Regarding the notion of incremental learning, we would like to reiterate on our answer to the reviewer @HwKE, this is a positive signal that we will emphasise in the main manuscript: We argue that our experimental setup does not strictly fit the definition of the domain incremental learning, because it does have a changing distribution of the underlying labels as well, while in other benchmarks the labels distribution is static, such as Permuted MNIST, Rotated MNIST [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks by Goodfellow et al.] and Clear [The CLEAR Benchmark: Continual LEArning on Real-World Imagery by Lin et al.] An outstandingly clear comparison is provided in Figure 1 of [CLAD: A realistic Continual Learning benchmark for Autonomous Driving by Verwimp et al.] that further refines the definitions of modern continual learning benchmarks.

Finally, to ensure that the requirements of the reviewer are met, we would like to point out that we indeed report both metrics in our manuscript:

• Online accuracy is reported in Figure 2, 3, 4 in the main manuscript and in Figure 5, 9, 11, 12, 13 and 18 in the supplementary material
• Overall forgetting performance, mentioned by the reviewer is reported as the final score in Figure 19. The curve just gives a richer representation of the performance across all previous tasks. Please find the final scores below:

CLOC Dataset

CGLM Dataset

FMoW Dataset

Yearbook Dataset

We thank the reviewer for their recognition of our presentation, the problem we study and the solution we proposed, and appreciate the reviewer for the valuable suggestions. Here are our responses to the reviewer's concerns:

Q1: Evaluation metric

Using the Online Accuracy metric, in online continual learning, appears to be the common practice in most works, see [Online Continual Learning with Natural Distribution Shifts by Cai et al.] [Real-time evaluation in online continual learning: A new hope by Ghuneim et al] [Computationally budgeted continual learning: What does matter? by Prabhu et al] [Rapid Adaptation in Online Continual Learning by Hammoud et al.] . While evaluation on past tasks has been reported, they were often reported in supplementary as the main focus was the most recent task with OCL. The reason is among the motivation prior works provide, [Cai et al.], is that the top priority for a time-ordered stream is the ability to predict current task and future task (note that evaluation is done before training at each time step). Performance on past tasks is not relevant in case those tasks (which happens to be from the past) might never be presented again (think of images of old laptops or images of devices (for classification) that are no longer popular these days nor will they be in the future.). Thus the majority of works considering OCL, they investigate it on streams that are time ordered (unlike classical offline CL), where performance on next batch accuracy (online accuracy) is the top priority while still reporting the past task accuracies for completion is supplementary.

To reiterate on our results, the reviewer can find the performance on past tasks metric described and evaluated on all settings of the main experiment in Section A.13, Figure 19. The backward transfer in Figure 19 is computed by the average accuracy of the last model on previous data following the evaluation protocol in [Online Continual Learning with Natural Distribution Shifts by Cai et al.]

Q2: Comparison to existing Continual Learning methods

As mentioned in line xx, our work is built on top of previous works of CL with fixed budget [Real-time evaluation in online continual learning: A new hope by Ghuneim et al] [Computationally budgeted continual learning: What does matter? by Prabhu et al] and delayed evaluation [Rapid Adaptation in Online Continual Learning by Hammoud et al.]. As consistently shown in these works, other existing semi-supervised CL methods like CaSSLE and most existing supervised CL methods like regularization based methods are not as effective as ER [On Tiny Episodic Memories in Continual Learning by Chaudhry et al]. [1] is a contrastive learning based method and [3] is a pseudo-labeling based method. We have tried our best to tune these two types of methods and include more advanced variants of them in section 6, although the modified version might be a little bit different from the two suggested ones. In conclusion, despite best efforts, since the computational complexity is constrained these methods failr terribly. This is already well documented in several other prior art that conducts these experiments under fixed computation [Rapid Adaptation in Online Continual Learning by Hammoud et al.] [GDumb: A Simple Approach that Questions Our Progress in Continual Learning by Prabhu et al.] where more sophisticated methods generally struggle.

Furthermore, prior to this discussion, we tried using the publicly available implementation of [2] and failed to reproduce their results.

Q3: Limited memory size

We agree with the reviewer that we should not limit the buffer size since all data are stored anyway, and this is what we set for the experiments for small datasets, such as FMoW, Wild-Time and CGLM in our experiments section. The current buffer size set is more due to the experiment time and I/O time. We have tried our best to set the buffer size to be $2^{19}$ (line 193), which will be enough to store all the data in a small dataset like Yearbook, FMoW, and CGLM, but not enough for CLOC(39M, Supplementary A1). We further have buffer size analysis in supplementary A14, showing that our methods is robust to different buffer sizes and the performance is not relied on large buffer size.

Q4: Potential limitations and code accessibility

We agree with the reviewer that we should discuss the potential limitations of our approach. We will add this in the final version of the paper. We will also make our code accessible after the paper is accepted.

Q5: Training time

We agree with the reviewer that the training time can be influenced by various factors. Our current computation budget is based on the number of forward-backward passes, which is generally a good proxy for time and it has been widely used in CL literature[8,9]. While the training time metric could be sensitive to multiple other factors, e.g. code optimization, hardware, data I/O speed, and implementation.

Here we add the training time for our method, ER, constrastive learning-based method, pseudo-labeling based method, and TTA methods with the same number of forward-backward pass in Table 1. Most of the experiments are using a single A100 GPU with 12 CPU. The training time is measured in hours. This table shows that the training time is could be entirely different due to various other factors.

I thank the authors for taking the time to address my concerns. I have read their rebuttal carefully.

Evaluation Metric

I get the authors point, although I respectfully disagree. The Online Accuracy is not the common practice in most work in OCL. See [1,2,3,4,5,6,7,8] for examples. Now, I understand that the metric is indeed very dependent on the specific application and that in your case of study you focus on Online Accuracy. While this makes sense, I believe this should be clarified in the manuscript, since I do not think that Online Accuracy is obviously the most important metric.

Regarding Figure 19, I do not think this is the standard definition of Backward Transfer, or at least it differs from the definition of [1]. In any case, if my understanding is correct, IWMS indeed improves Online Accuracy at the cost of some marginal performance drop on previous tasks for specific cases (FMoW dataset). This does not in any case diminish this work's quality, although I believe that this potential drawback of the approach could be more clearly stated in the manuscript, in a limitation section for example.

Comparison to existing OCL methods

While your work is based on previous studies, their findings regard the fully supervised scenario. I still believe that including some existing work on Semi-Supervised OCL in your comparison would improve the manuscript.

About the setup

Thank you for clarifying. I understand the setup more clearly now. However, I am not sure to understand, if new classes appear in the stream, but are unlabeled yet, how do you predict them? To my understanding, you would not predict them before they are labeled.

Training time

Thank you for sharing the training time, I believe including it in the appendix would improve the paper.

Code

Thank you for sharing the code with a live demo.

[1] Mai, Zheda, et al. "Online continual learning in image classification: An empirical survey." Neurocomputing 469 (2022): 28-51.

[2] Guo, Yiduo, Bing Liu, and Dongyan Zhao. "Online continual learning through mutual information maximization." International conference on machine learning. PMLR, 2022.

[3] Koh, Hyunseo, et al. "Online boundary-free continual learning by scheduled data prior." The Eleventh International Conference on Learning Representations. 2023.

[4] Wang, Maorong, et al. "Improving Plasticity in Online Continual Learning via Collaborative Learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Wei, Yujie, et al. "Online prototype learning for online continual learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[6] Gu, Yanan, et al. "Not just selection, but exploration: Online class-incremental continual learning via dual view consistency." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[7] Buzzega, Pietro, et al. "Dark experience for general continual learning: a strong, simple baseline." Advances in neural information processing systems 33 (2020): 15920-15930.

[8] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. "Gdumb: A simple approach that questions our progress in continual learning." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.

We thank the reviewer for engaging in this thoughtful conversation and bringing our attention towards richer representations of performance in OCL settings.

We would like to respond to the raised points in order:

First point

The Online Accuracy is not the common practice in most work in OCL. See [1,2,3,4,5,6,7,8] for examples.

Comment on [1]

In the referenced paper [1], under the "Evaluation Metrics" Average Accuracy is defined as follows:

$\mathrm{Average Accuracy}(A_i)=\frac{1}{i}\sum_{j=1}^ia_{i,j}$

which is closely related to Online Accuracy, but we appreciate the fact that in such scenario performance on past iterations are re-evaluated in every step. In our setting evaluating and reporting this metric would be infeasible, as the number of steps on the CLOC, CGLM and FMoW dataset is 3-to-4 orders of magnitude larger than the examples provided in the survey (the maximum number of steps in [1] is 20, for comparison in our experiments the maximum number of "tasks" is 296 119).

Regarding Figure 19, I do not think this is the standard definition of Backward Transfer, or at least it differs from the definition of [1].

We truly appreciate the reviewer pointing out the difference in the definition. We recognise that the backward and forward transfer can, and should be reported (as defined in [1]) at given points during the training. To this end we have re-run the Naïve and IWMS experiments with $d=50$ on the two datasets where IWMS was performing the best and the worst to provide a full comparison against the baseline. We simplified the table representation by splitting the validation data into 100 equal sized ranges along the time axis, such that the ranges would correspond to the training data range:

I would like to thank the authors for taking the time to address all my points and for their detailed response.

I agree that Average Accuracy is not the only metric considered in the paper I referenced, but it is still the most commonly used one for comparison. I guess we could argue on which is the most important one depending on the problem at hand, and again I understand that Online Accuracy might make more sense in your context.

Overall conclusion

The authors have addressed my concerns and I will increase my score to 6.

In any case, if my understanding is correct, IWMS indeed improves Online Accuracy at the cost of some marginal performance drop on previous tasks for specific cases (FMoW dataset).

It is correct, thank you for highlighting this. We will address the limitations on backward transfer on FMoW.

Comment on [2, 5, 6, 7]

We would like to point out that Guo et al. details the evaluation metric in 6.1 as follows:

We first learn from the data stream of all tasks for each dataset, and then test the final model using the test data of all tasks. We report the average accuracy of all tasks from 15 random runs for each dataset

We would like to argue that this metric is taking an excessively strong measure to remove noise from the metric. In our experiments, we experienced that rerunning the same training with different seed results in negligible (less than 0.01%) differences in the results. Running the experiments 15 times to evaluate the metric of [2] is infeasible for us.

This similarly hold for [5] as well, since they report the Average Accuracy and Average Forgetting across 15 runs.

Furthermore, although the main manuscript of [6] does not provide the detail about re-running the Average Accuracy, the corresponding code is set by default to 15 re-runs with different seeds. (follow this URL for reference: https://github.com/YananGu/DVC/blob/6f12984d10a4a1c4609f221b939f93d94fc8258e/general_main.py#L29 )

In [7], the number of random initialization is dropped to 10, otherwise they report the Average Accuracy as well.

Comment on [3]

We would like to point out that [3] introduces their own metric: Knowledge Loss/Gain Ratio [3] claiming that the metrics used by [2, 5, 6] are relying on the notion of task boundaries therefore they define a new objective that is "appropriate for periodic data distribution". In our paper we cannot make such assumptions about periodicity.

Comment on [4]

The accuracy metric proposed by [4], Learning Accuracy (LA) using Model Plasticity is formally defined for the $j$-th task as:

$$l_j = a_j^j$$

where $a_j^i$ is the accuracy evaluated on the test set of task $j$ after training the network from task 1 to task $i$. We would like to argue that this metric is similar to the Online Accuracy metric apart from the notion that here the test samples are drawn from a different distribution, whereas the Online Accuracy is evaluated on the $j$-th batch of data before it is used for training. If we assume that both the test and the training batch is drawn from the same distribution at time-step $j$, the two metrics are arguably the same. (Please note that the training is only done on the batch after the evaluation in the case of Online Accuracy.)

Comment on [8]

The paper "GDumb: A simple approach that questions our progress in continual learning" highlights the weakness of the then-standard metrics in Continual Learning such as Average Accuracy and Accuracy at end. While the paper aimed to steer the CL research community towards higher standards, these metrics stayed popular.

In fact, [Online Continual Learning with Natural Distribution Shifts by Zai et al.] uses GDumb's argument to propose the Online Accuracy metric that recently increasing number of work adopted.

Resolution

Now, I understand that the metric is indeed very dependent on the specific application and that in your case of study you focus on Online Accuracy. While this makes sense, I believe this should be clarified in the manuscript, since I do not think that Online Accuracy is obviously the most important metric.

We recognize and fully agree with the reviewer's criticism that our choice of metric needs more justification, as we present it as the "go-to" metric in our narrative. We will definitely address these points in the updated manuscripts more carefully, if the reviewer agrees.

Second point

While your work is based on previous studies, their findings regard the fully supervised scenario. I still believe that including some existing work on Semi-Supervised OCL in your comparison would improve the manuscript.

Due to time limitations we could not fully reimplement [Contrastive learning for online semi-supervised general continual learning. by Michel et al.] to report scores in the rebuttal period, but we are happy to include them in the final manuscript.

For reference, while we think the comparison would indeed interesting, we remain skeptical about contrastive methods outperforming IWMS, or even the Naïve baseline under our experiments' computational constraints, as we discuss the results of 6 different contrastive methods in detail in the Supplementary Material A.8.

We thank the reviewer's recognition for our extensive experiments and the simplicity of our proposed method and the value suggestions. The following are our responses to the reviewer's concerns:

Q1: Need further explanation regarding the implementation and each module within the IWMS method and technical innovation.

Our implementation of IWMs is closely related to the experimental setup in Section 5. It is built upon the best performing method, ER, and we further improve it by sampling the buffer samples that are more similar to the unlabeled data. All the implementation details and "technical innovation" are detailed in full in Section 1, 3 and 4, in Figure 1 and in Algorithm 1 and 2. Line 149 - provides "technical insights regarding the implementation and each module within the IWMS method". Line 154 and 163 - "delves deeper into the memory buffer". Section A.12, Line 756 - Analyses the difference between online learning continual learning and online continual learning.

As we point it out in the main paper in Section 4, L137, the novelty lies in how to learn the unlabeled distribution. Our experiments in Section 6 demonstrate that existing literature, which focus on operating unlabeled features to learn the most recent distribution, are often neither effective nor efficient. Our approach avoids unnecessary computation on the most recent unlabeled distribution, even if it closely matches the evaluation distribution. Instead, we sample previously labeled data similar to the current unlabeled data. This allows us to construct a pseudo-distribution close to the most recent distribution and conduct purely supervised learning on the pseudo-distribution.

Q2: Need further explanation regarding the memory buffer.

This is all detailed in Section 4, Line 154, but we will reiterate it here for completion. At time step $t$, we keep a memory buffer to store both the raw data and the features of the labeled data from time step $t'$($t'$< $t$) computed in $t'$. We sample the data according to the similarity between the features from the current unlabeled data $x^t$ and the features in the buffer. We then take the buffer data whose features are most similar to x^t to construct a new labeled batch. This batch is our buffer batch.

Q3: Need to analyze and discuss the difference between online learning, continual learning, and online continual learning, and further clarify the topic of this work.

We already had a Section 2 and A.12 in the paper discussing this in full depth. In fact, the title of the section is "Online Learning vs Online Continual Learning" and "Considering catastrophic forgetting".

We provide further explanation below: Online Learning and Online Continual Learning both involve learning from data arriving sequentially, but Online Learning typically deals with single-task streams, often assumed to be from an i.i.d. distribution. In contrast, Online Continual Learning is more concerned with non-stationary streams that undergo frequent changes in distribution. Continual Learning, on the other hand, is a broader concept that encompasses any type of learning process where the data is revealed sequentially and the distribution of the data may change over time. Recent discussion in continual learning has focused on the scenario where the computation is fixed for fair comparison.

Our work mainly focuses on the online continual learning scenario, where the distribution of the data changes over time and the data is revealed batch-wise. As motivated in line 26, we further consider the label delay problem, where the labels of the data are not available immediately, and most recent data distribution is unlabeled. To this end, we propose solutions from various literatures and propose our simple yet effective method, IWMs, to address the label delay problem in online continual learning.

We appreciate the reviewers' positive comments on our method, experiments, and paper presentation. We address the above concerns here.

Q1: The discussion of related work on the label delay problem is not comprehensive.

We thank the reviewers for pointing out the missing references. We will include the suggested references in the final version of the paper, covering both the label delay and asynchronous labels scenarios.

Q2: The paper considers only the simple case of a fixed delay time step d.

We acknowledge that this is an excellent suggestion and plan to add further ablations on varying delay time steps during the rebuttal period. However, we would like to point out to the reviewer the numerous experiments we have conducted where we vary $d$ across all experiments (as in Figure 2,3,4), which already demonstrate the superiority of IWMS over other methods. We do not generally expect the performance ranking to differ significantly with variable delay times.

Q3: Comparison to the online learning approaches

We have a thorough discussion on this point in the supplementary A12. Furthermore, We appreciate the reviewer’s suggestion to compare our work with existing research on online learning methods that handle delayed labels. We would like to point the reviewer to the supplementary material A12 (page 23, L756), which explains the primary reasons for the incompatibility of traditional online learning approaches with modern OCL settings. Briefly, the main issue is the general lack of information retention mechanisms in traditional methods, which are crucial for addressing the complexities of real world continual learning tasks, such as training a feature extractor that both learns new concepts faster (forward transfer) without losing the capability to perform well on already seen problems (backward transfer). To highlight that without rehearsing on memory samples the methods suffer significant performance degradation, we implemented the OL algorithm that is analogous to the mentioned papers in the special case in which all the labels (or feedback) arrives in order with a fixed constant $d$ delay. We ran new experiments (with identical experimental environment described in the main experimental section, Section 6) on the two largest datasets, CLOC (39M) and CGLM (580K), with computational budget $\mathcal{C}=2,8$ respectively, for $d=10,50$ and $d=10, 50, 100$ respectively. The results show extreme underperformance:

[Online Accuracy of Online-Learning (no memory rehearsal) on CLOC]

[Online Accuracy of Online-Learning (no memory rehearsal) on CGLM]

The results clearly indicate the necessity of memory rehearsal: models on CLOC saturate at <6.5% for delay=10 and <4% for delay=50. In the case of CGLM dataset the performance collapses in all three delay scenarios <1%.