Time-Sensitive Replay for Continual Learning

We thank the reviewer for their time and thorough analysis of our paper. We are happy that the reviewer finds our experiments satisfactory and our review of replay models suitable. We address the reviewer’s questions as follows:

In the third paragraph of the introduction, the authors introduce the term "evolving" in the sentence "This enables the method to adapt to evolving data streams without prior knowledge of future tasks." What does evolving mean? Does it refer to any assumptions about the distribution of tasks over time?

By “evolving” we refer to the changing distributions of examples over a data stream, meaning that we expect the frequency of examples belonging to different classes to change. We have clarified this in our revisions.

According to the definition of task given in the preliminaries, could task be any dataset? Is there no assumption between consecutive tasks, or about the distribution between tasks, or that tasks are related?

Because of the nature of our experiments and our method, we assume that a dataset contains examples of a constant shape with the same modality. A task is instead defined as a distribution of classes. We clarify this in our revision.

The reviewer believes that the paragraph on confidence in Section 3 could be better explained. For example, the symbol n appears in the first line and is explained in line 4 which makes it difficult to understand.

We appreciate that this may cause some confusion and have corrected this in our revision.

The reviewer thinks there is a typo in the equation of a in section 3.

We thank the reviewer for pointing out this typo. It has been corrected in our revision.

Is n in the second paragraph of section 4.1 different from the n in section 3?

No, n refers to the number of validation examples tested to obtain a sequence of logits of length n in both instances. We clarify this in our revision.

At the end of the second paragraph of section 4.1 it is explained that "the oldest logit is discarded". Why the oldest? Is there time dependence in the distribution?

The oldest logit is discarded because it has the least relevance to monitoring currently occurring prediction drifts. We are more interested in recent changes than changes that have already occurred. We address this in our revision.

Additional changes have been highlighted in the revised paper.

We thank the reviewer for their time and thorough analysis of our paper. We are happy that the reviewer finds our paper straightforward and our experiments effective. We address the reviewer’s questions as follows:

Experiments are performed in synthetic settings using benchmarks that are common but do not have a natural notion of time. Given the proposed setting, why haven't experiments been performed on suitable real-world datasets that have a notion of time?

We wanted to remain consistent with theoretical CL work, many of which compare with image datasets constructed to be introduced over time. It would be interesting future work to extend our method for datasets that have time information and where time is important, but doing so would likely require a recurrent architecture that in our opinion falls beyond the scope of this work.

Continual learning has a rich literature and there are many works on the topic. What is the justification for selecting the methods that have been used in the paper? Do they have SOTA performance? It is very common in CL literature to provide an extensive comparison.

We selected models with state-of-the-art performance for both replay and regularisation. However, we appreciate and agree with the comment that TSR and TF-TSR should be compared with more recent methods, and have provided additional experiments in the revised paper. We provide three other continual learning methods: Experience Replay (ER), Continual Learning by Modelling Intra-Class Variation (MOCA) and Proxy-based Contrastive Replay (PCR).

The code is not provided which makes judgment about reproducibility challenging.

A link to an anonymous repository containing our code is provided on Page 2 in the final paragraph of the Introduction and in Section E.1 on Page 15.

Additional changes have been highlighted in the revised paper.

We thank the reviewer for their time and thorough analysis of our paper. We are happy that the reviewer finds our idea interesting and that a task-free implementation is useful. We address the reviewer’s questions as follows:

First, whether such a validation dataset is available for each task is questionable. Without this, the forgetting performance cannot be evaluated.

We agree that a pre-established validation buffer severely impacts the practicality of the TSR method. Similarly to TSR’s reliance on a knowledge of task information, in most real-world cases, it is unrealistic to expect a priori samples of a data stream. Both of these limitations are addressed in the TF-TSR method. Page 5 of our paper contains a paragraph on the TF-TSR Validation and Replay Buffer changes. Here, we acknowledge that a validation set cannot be expected prior to training, so validation examples must be stored in memory as new data arrives. In this instance, there is no additional information or data required by TF-TSR prior to training. We appreciate that this aspect should be highlighted more in the paper, and so have made revisions on Page 5. We provide further experiments varying the size of the validation buffer in the Appendix on Page 18.

Second, even if we have such validation datasets, evaluating the performance on all previous tasks using the validation data at each time step will introduce a lot of computational complexity, especially when the number of tasks is large. The detection complexity can be much higher than the replay complexity. Even though an analysis of the theoretical complexity is provided in the appendix, it is unclear if the proposed algorithm can achieve better computational complexity than other replay-based methods in theory.

Evaluations are required at each batch to monitor for prediction drift over previous tasks. On Pages 13 and 14 of the Appendix, we discuss the theoretical complexity of this process. We appreciate that it is unclear how the replay must be reduced to provide benefit despite consistent validation tests, and so have made revisions in the paper. Furthermore, we have provided additional discussion in this section about the complexity of forward-pass validation tests. We describe the process of sampling from the validation buffer to reduce this complexity further. Moreover, on Page 9, we empirically compare the training time of TSR and TF-TSR with existing baselines and find that despite the expense required to monitor performance over previous tasks, our methods nonetheless achieve a lesser training time.

First, in the comparison, even if the proposed method has a shorter training duration, the final accuracy can be, for example, 2% lower than CEC on MNIST. This somehow indicates that the condition to determine when to replay is not so helpful. In other words, the proposed algorithm does not tell us correctly when to replay, and more data replay clearly improves the performance as in CEC.

Even if the mean accuracy may sometimes be lower for our experiments than other full replay models, our results always fall within the error of baseline evaluations and there is never a statistically significant difference in our results. We have made some revisions in our experiments section to emphasise this.

Second, only two baselines (among which only one is replay-based) are considered in the paper, which are also relatively outdated. Considering that there are many advanced replayed-based CL methods out there, more baseline methods should be investigated here. Specifically, the question here is that if we can select the replay samples wisely, do we still need to care about when to replay? Maybe the answer is still yes, but this investigation will further justify the contribution of this paper.

We appreciate and agree with the comment that TSR and TF-TSR should be compared with more recent methods. We provide three other continual learning methods: Experience Replay (ER), Continual Learning by Modelling Intra-Class Variation (MOCA) and Proxy-based Contrastive Replay (PCR). These additional results demonstrate that TF-TSR achieves a classification accuracy that is not significantly different to current state-of-the-art methods with lower time requirements. An opportunity for future work may be to investigate whether sampling algorithms may enhance TSR.

The writing can be improved and there are multiple typos.

We agree with the reviewer that there are a couple of oversights in our writing. We have corrected these oversights in the revised paper.

Additional changes have been highlighted in the revised paper.

We thank the reviewer for their time and thorough analysis of our paper. We are happy that the reviewer finds the premise and motivation of the TSR method strong. We particularly appreciate the reviewer’s suggestion to compare with further continual learning baselines. Additional experiments are provided to address this. We address the reviewer’s questions as follows:

Can you explain if and how you took into account the size of the validation memory in the evaluation?

We agree that a pre-established validation buffer severely impacts the practicality of the TSR method. Similarly to TSR’s reliance on a knowledge of task information, in most real-world cases, it is unrealistic to expect a priori samples of a data stream. Both of these limitations are addressed in the TF-TSR method. Page 5 of our paper contains a paragraph on the TF-TSR Validation and Replay Buffer changes. Here, we acknowledge that a validation set cannot be expected prior to training, so validation examples must be stored in memory as new data arrives. In this instance, there is no additional information or data required by TF-TSR prior to training. We appreciate that this aspect should be highlighted more in the paper, and so we have made adjustments in this section. We provide further experiments varying the size of the replay buffer in the Appendix on Page 18.

Why did you not compare with other very simple and efficient baselines (e.g., simple experience replay)?

We thank the reviewer for pointing out this oversight. Our perspective was that comparing with more recent and superior replay methods was sufficient. We have provided additional experiments to remedy this. In addition to our current baselines, we provide three other continual learning methods: Experience Replay (ER), Continual Learning by Modelling Intra-Class Variation (MOCA) and Proxy-based Contrastive Replay (PCR). These additional results demonstrate that TF-TSR achieves a classification accuracy that is not significantly different to current state-of-the-art methods with lower time requirements.

A considerable limitation of the paper is that it ignores cases where the data distribution changes continuously over time. In such cases, task boundaries cannot be defined or detected. In my opinion, most real-world applications of continual learning do not have task boundaries, since the distribution is constantly changing.

We agree that it is important to consider changing distributions in a data stream for continual learning settings. We intended that distribution changes are represented as tasks in our task-sequential setting, where examples may abruptly shift throughout the data stream. We also employed synthetic datasets to adjust the distributions and how distributions change over each benchmark image dataset. These synthetic datasets are described on Page 6 and used on Pages 8 and 9, where we observe that TF-TSR can successfully detect tasks even under these more challenging settings. We appreciate that this should be made clear in the paper, and so have made revisions to the definition of a task on Page 3 and have added a further description of synthetic datasets in Page 18 in the Appendix.

Additional changes have been highlighted in the revised paper.