Bridging Inter-task Gap of Continual Self-supervised Learning with External Data

Thank you for your valuable comments. We address your questions or concerns below.

Q1: Novelty and technical contributions

A1: Thank you for your comments. However, we would like to clarify that our work is not a simple combination of contrastive learning and external data, but can bring many insights to the CCSSL field. Our novelty and contribution are mainly:

1. We point out the problem of ignored inter-task comparisons in CCSSL and demonstrate through many experiments (Fig 1 Right, Table 7 in Appendix A2.1) that this problem can exacerbate the model's confusion of inter-task classes. As reviewers oaUg and hAsz said, this problem is “significant but often overlooked”, “The finding that existing regularization-based CCSSL methods overlook inter-task discrimination is novel”, we believe that this problem is inspiring and novel. Although this problem has often been overlooked in previous works, it should be considered in any future work in CCSSL.

2. We demonstrate the soundness and effectiveness of using external data in CCSSL, which is novel in this field. As reviewers oaUg and hAsz said, “the proposed method leveraging external data is novel”, “BGE offers a creative solution”. For a long time, there has always been a gap between replay-based and regularization-based approaches (i.e., replay-based approaches generally perform better than regularization-based approaches, but have limited applicability due to privacy, safety, and other concerns). Considering the characteristics of contrastive learning (refer to Appendix A.2.4), we propose that using external data can also play a role in improving the performance of existing regularization-based approaches while avoiding the aforementioned potential concerns.

Q2: The performance improvements in realistic applications

A2: Admittedly, there is a relevance between the performance improvement of BGE and the quality of the external datasets. However, in realistic applications, we do not have to use wild data as external data. Instead, we can choose high-quality general datasets such as ImageNet, which are already capable of handling most of the realistic tasks.

In our paper's experiments (Table 1, 2, 5, 6), we artificially added a lot of OOD data to the external data, intending to test the ability of our method in more challenging scenarios, but such scenarios are not common in real-world situations, as most domains have established relatively well-developed public datasets nowadays. Therefore, our method can make sense in most real-world scenarios. In addition, even in extreme cases (where the external data we collect contains many OOD data), our OPO sampling algorithm can select external data that is more suitable to be added to the training, and the results in the Table 3 in our paper demonstrate the performance improvement of the sampling algorithm when the external data contains a variety of OOD data.

Q3: More detailed discussion of external data quality

A3: To clear the quality requirements for external data, we conduct experiments using various datasets as the external dataset. In Table 6, we show the results using Internet data (CC3M), generated data (GenImage), or fine-grained data (CUB-200) as the external dataset. Although these datasets have different characteristics, they are all still effective in improving the baseline method. One of them, CUB-200, is a dataset containing only various classes of birds, which can still be utilized by the BGE even though it is of very low quality for our task (due to lack of diversity). To simulate using external data in the wild, we use CC3M as the external dataset, which has images harvested from the Internet with very simple filtering (thus representing a wider variety of styles). Nevertheless, BGE still enhances the baseline method. We believe that even if we truly use external data in the wild with only a little filtering, the results would probably be in line with using CC3M.

In addition, the sampling algorithm we designed can also filter out OOD data to further improve the quality of external datasets. Therefore, concerns about applicability from the perspective of external data quality are less significant.

Thank you for your valuable comments. We address your questions or concerns below.

Q1: Analysis of how BarlowTwins “contrast” during learning

A1: We treat BarlowTwins here as a generalized contrastive method. Although it is not directly contrast the anchor with negatives, [1] has approximately regularized its loss to keep positive pairs aligned (alignment) and negative pairs away (divergence). Specifically, after computing the cross-correlation matrix of two views' features, BarlowTwins constrains the diagonal elements to be close to 1 (first loss term) and off-diagonal elements to be close to 0 (second loss term). [1] indicates that the first term can be formulated as an upper bound on alignment, and the second term as an upper bound on keeping class centers apart (approximate divergence). Thus, its loss optimization has an implicit contrast effect. Therefore, BarlowTwins can also be explained by contrastive learning theory, justifying the use of "contrastive" word in paper. We will clarify this in future editions to avoid misunderstanding. In addition, we have evaluated some other contrastive methods, including SimCLR (Table 4, PDF Table R3Q5 (a)) , BYOL (Table 8) and VICReg (PDF Table R3Q5 (b)).

[1] Towards the generalization of contrastive self-supervised learning. Huang W, et al. ICLR 2023.

Q2: More explanation about privacy concerns

A2: Please see the common response.

Q3: Is this a weighted sampling?

A3: No, it’s a uniform sampling. We conduct a weighted sampling experiment based on "PFR CIFAR100 4 tasks, with CIFAR10 as external data", which ensures that the data in each epoch is well-balanced from each task. However, the result is 62.69, lower than 64.37 for uniform sampling.

We believe the reason may be: 1. $D_e^{t-1}$ has a limited size, thus it cannot provide sufficiently diverse external data as seeing more tasks. 2. Since the current task has not been learned before, the learning of in-task data is crucial to improve the performance of the current task.

Q4: Is diversity encouraged here because you sample the current task data uniformly one by one when selecting the closest external data?

A4: Yes, we sample one closest external data by one current task data for diversity purposes.

External data do not necessarily belong to the same class as in-task data, and their classes may also be similar to future classes. Since the feature distribution learned by contrastive learning is relatively uniform, the distribution of external data obtained by letting each in-task data select the nearest external data is also relatively uniform, enhancing diversity. The understanding of "$D_e^t$ estimates $D_t$ well" is correct, but $D_t$’s feature distribution is relatively uniform. If the goal of sampling $D_e^t$ is merely to proxy $D_t$, we should select data with the most distinctive features of each class in $D_t$ (usually selected through class prototypes in supervised learning). In contrast, the goal of diversity helps in selecting external data that may feature future tasks, rather than exclusively best matching the old class features .

Q5: More contrastive learning method paradigm experiments

A5: We use BarlowTwins for most of our experiments because most of the prior works commonly include it, thus using it is convenient for comparisons. We also show results using another self-supervised learning method, BYOL, in Appendix A.2.2.

To further exhibit the generalizability of BGE, we extend BGE to SimCLR (ImageNet100 benchmarks, ImageNet-900 as external dataset) and VICReg (CIFAR100 benchmarks, CIFAR10 as external dataset) based on CaSSLe and FT, the experimental results are shown in the PDF Table R3Q5(a) and R3Q5(b).

Q6: Minor question responses

(Eq. 1, L#59, Eq. 2) Presentation questions: Thank you for your suggestions, we'll revise them carefully.

(L#138-143) Presentation of inter-task discrimination: Inter-task confusion is not specific to CCSSL; it's partly caused by catastrophic forgetting, which is common in all continual learning scenarios. My point in here is that due to the lack of labels, CCSSL needs to learn by inter-sample comparisons, and the absence of inter-task comparisons exacerbates inter-task confusion.

(L#52, L#54) Presentation of our method’s advantages: In paragraph of L#52, we compare BGE with prior continual learning methods using external data. So the “more generalizable and robust to OOD data” and “not require extensive external data” are our advantages over them. Some prior methods use external data in a supervised manner, needing to generate pseudo-labels, which degrade in quality with OOD data, affecting performance. Other methods require extensive data to stabilize feature space when training, compared with them, our required external data is less.

(L#283) OPO sampling algorithm analysis: This paragraph refers to the Table 3. It's true that in some cases it's not obvious.

Sorry for the misunderstanding of above minor questions. We will revise them in the future.

(L#496) Hyperparameters search: The learning rate in CaSSLe and PFR’ code is 0.3. We follow them and search hyperparameters around them.

Discuss the concurrent work: [1] discovers the lack of inter-task comparisons in existing CCSSL methods concurrently with us, but [1] solves it by simply saving old task data for comparison with new task data. We argue that saving old task data may be impractical due to privacy concerns and inaccessibility, so we propose using external data to compensate for inter-task comparisons. Our work not only resolves inter-task confusion but also provides a new performance improvement scheme for CCSSL without saving old task data based on the properties of contrastive learning (the properties refer to Appendix A.2.4).

[1] Integrating present and past in unsupervised continual learning. Zhang et al. CoLLAs 2024.

Thank you for your valuable comments. We address your questions or concerns below.

Q1: The trade-off between performance improvement and time consumption

A1: The additional computational consumption of BGE comes mainly from the additional amount of data, therefore, we statistic the performance and corresponding computational consumption ratios based on PFR 4 tasks as the external data budget $K$ increases.

Budget	0	2000	5000	10000	20000	30000
Performance	60.92	61.96	62.79	64.37	65.43	65.55
Cost	1	1.12	1.3	1.6	2.2	2.8

The cost represents the ratio of the number of iterations to the baseline. As the budget increases, the improvement of BGE to the baseline method becomes more significant until the amount of external data reaches approximately 20,000, at which point it is difficult to improve performance by further increasing the budget. Considering the trade-off between performance improvement and time consumption, we set a budget of 10K in our method.

At the same time, to validate the effectiveness of BGE under the constraint of computational resources, we control the computational consumption of BGE to be consistent with the baseline method to conduct the following experiments. All experiments are conducted under the “PFR+BGE, CIFAR100 4 tasks, CIFAR10 as external dataset” setting.

1. Train less epoch. In this case, the model can only be trained for fewer epochs when more external data is used, the results of BGE with different external data budgets are shown below:

Budget	0	2000	5000	10000	20000	30000
Epoch	500	431	357	277	192	147
Performance	60.92	61.47	61.99	62.87	62.59	63.13

The second row of the table indicates the number of epochs that can be trained at different budgets. When the budget is large (>10000), even if the number of epochs that can be trained is low, the performance is still better than the baseline.

2. Incorporating external data by mixup. In this case, instead of adding external data directly to the training, we mix each in-task data with a randomly selected external data when training, making the number of data iterations the same as the baseline. The only additional consumption introduced here is the mixup operation, which is negligible.

Method	PFR	PFR+BGE$_{mixup}$	PFR+BGE
Performance	60.92	62.57	64.37

Q2: How BGE scales with the size of the external datasets?

A2: As shown in the first table of A1, as the size of the external datasets increases, BGE provides more improvements to the baseline method. Until the budget reaches about 20,000, further increases in scale cannot seem to keep improving performance. The budget is slightly larger than the scale of our in-task data at this time. Therefore, we observe that the scale of external data should roughly match the scale of in-task data in our method.

Q3: How does BGE handle the potential privacy concerns that may arise from using external data, especially if the data contains sensitive information?

A3: We can limit the selection of external data to publicly available datasets on the Internet, and refer to the license and other statements of each dataset to select data that are allowed to be publicly used, and they usually do not have privacy risks.

If in some special cases, using publicly available datasets as external data cannot harvest the desired results, and therefore we have to collect the data by ourselves, we can introduce some techniques from other domains (e.g. privacy filtering, differential privacy, machine unlearning) that are complementary to our method to address privacy concerns. Since the number of candidate external data is almost infinite, even after filtering, some useful external data can still be retained.

Another perspective of privacy is data accessibility, where we may not have access to old task data. For example, some existing models on the Internet may only release its checkpoint but not the training data for some privacy reasons. While if we choose to use publicly available external datasets, there are no accessibility concerns.

Thank you for your valuable comments. We address your questions or concerns below.

Q1: The goal and setups of CCSSL

A1: In self-supervised learning, we expect the model could learn discriminative representation from unlabelled seen data. In Continual Self-Supervised Learning (CSSL), the data to be seen is set as a non-stationary data stream with task intervals, and we expect that the model can accumulate the discriminative representation of all seen data as tasks come. Continual Contrastive Self-Supervised Learning (CCSSL) focuses on contrastive self-supervised methods in continual learning. In CCSSL, the concept of “a task” represents "a stationary data distribution", and by learning this task, the model can discriminately represent the data for this distribution. In concrete experimental setups, we divide all the classes of the dataset equally into each task, and the classes in each task are not intersected.

CCSSL is crucial because it allows the model to retain the discriminative representation for old tasks while enhancing the representation for new tasks in continual training without training from scratch, enhancing the training efficiency in practical applications. Meanwhile, self-supervised learning offers more generalized representations than supervised learning, so SSL also has the advantage of less forgetting in continual learning research.

Q2: Additional baseline: compare with a model trained solely on external data

A2: We appreciate your comments about useful baseline settings. We conduct experiments with training the model using solely external data, the results are shown in the PDF Table R1Q2. Although total external data (CIFAR10 or a subset of CC3M (~431K images)) are jointly used to train the models, their performances are worse than our method.

Q3: Additional baseline: change external data every epoch during Joint+ED experiment

A3: We also update the external data at each epoch in Joint+ED, achieving a result of 68.02, which is still close to Joint without ED (68.09). This indicates that BGE’s performance is not dependent on seeing all samples from ED.

Q4: Why we can’t simply retain the previous tasks’ datasets

A4: Please see the common response.

Q5: The experimental anticipated scenario

A5: Our evaluation aims to measure the performance of different methods in maintaining stable discriminative representations in continual self-supervised learning, rather than pursuing the best performance on each downstream task. Therefore, we use linear probing as a straightforward tool for the classification task without updating the backbone. As the reviewer suggested we also evaluate the KNN classification accuracy in the PDF Table R1Q5, where incorporating BGE on top of PFR also improves the KNN accuracy. Other evaluation methods we will explore in future work.

Q6: Related information about external data budget $K$

A6: We report the budget $K$ value in our paper's L228 and L256. We set $K$ 10K in the main experiments. We also conduct experiments as $K$ changes, under the setting of CIFAR100 4 tasks, using CIFAR10 as the external dataset (in the PDF Table R1Q6).

As $K$ increases, the improvement of our method becomes more significant. However, it also introduces additional computational costs. Considering the trade-off between performance and efficiency, we choose the $K$ value 10K.

Q7: Are the groupings of classes the same or randomized between experiments?

A7: To ensure fairness, groupings of classes were the same for all experiments.

Q8: The cost, pruning, and sweep interval of external data selection sweep algorithm

A8: After extracting all data features, 13,000 in-task data sweep the external dataset at a rate of approximately 48 seconds per 100,000 data. Compared with training time (typically many hours), this cost can be ignored.

It is certainly possible to use a subset when the external dataset is too huge. In our paper’s Table 6, the experiment on CC3M( total ~3M) just used a randomly chosen subset (~431K). To further validate, we set the external data to ImageNet-1K (~1.3M) or a randomly selected subset of it (~100K) and the results are shown in the PDF Table R1Q8(a). These results demonstrate that using a subset of a large external dataset yields comparable results to using the full dataset.

We also explored the impact of updating external data at different intervals, setting the update interval to 100 epochs and 200 epochs, with results shown in the PDF Table R1Q8(b). The results indicate that more frequent updates do not improve BGE's performance. Considering that the external data must compensate for the inter-task comparison, we update external data at the end of each task's training.

Q9: Discuss the limitation

A9: Please see the common response.

Q10: Statistical significance

A10: We report the statistical significance of primary experiments in Appendix A.2.6. And experiments in Table 1 and 2 are all performed with the same seed, the same ordering of tasks, to eliminate the effect of hidden variables.

Q11: t-SNE Figure

A11: To ensure fairness, we set all parameters of each t-SNE plot strictly consistent. We show the t-SNE plots to visualize the problem of inter-task confusion and the effect of our solution in an intuitive and illustrative manner, and to achieve complementarity with the quantitative results in the subsequent tables.

Q12: Writing comments

A12: We appreciate your valuable suggestions about figures, tables, typography, and citations, and we will revise them carefully.

I thank the authors for the response.

For example, when we continue to train a model based on other people's open-source checkpoint, its training data may not be released together. At this point, it is not possible to continue training the model using replay-based methods, whereas BGE can still be used.

Could you explain how BGE can be used in this case? I thought BGE required selecting external data based on their similarity with in-task training data.

In terms of data quality, BGE does not require all external data to be of the same domain as the in-task data, as shown in our ImageNet100 experiments using the DomainNet dataset as the external dataset.

There are 6 domains in DomainNet, and one of them is "real" which is the same domain as IN-100. Do the authors know (or can they check) what fraction of the images which were selected from this external dataset came from the "real" domain (and hence were in-distribution) vs other domains (and hence were OOD).

For tasks with scarce data (e.g., medical images), the real external datasets are limited, but thanks to the development of image generation, methods for generating scarce data have been widely explored.

I see trends like this often, and I find such methods highly concerning for a couple of reasons. (1) Models trained on generated images have issues, as has been reported multiple times. Though the studies look at "model-inbreeding" from the perspective of training generative AI on the outputs of generative AI, rather than training representation learning on the outputs of generative AI, it is still cause for concern if the representation learning is to learn the biases of the generative model. (2) Using a generative model to create samples for representation learning can work via transfer learning from the knowledge in the generative model to the representation model. To do this, the generative model needs to understand the domain that is being generated and has its own representations inside the model. So instead of distilling the information via image space, you can adapt the generative model and avoid having to train a new model [1, 2]. But also, how was this generative model trained if there's no large datasets available for the domain of interest?

I remain concerned that there is a mismatch between the scenario being explored in the paper and any real world scenario. Can the authors provide concrete examples of some tasks and data domains where they see their method being deployed? One where a dataset comes in continually and must be destroyed soon after training on it; one where there is no large publicly available in-domain dataset already available to train the model on. And exactly what external data would you propose to use?

[1] Mittal et al. "Diffusion Based Representation Learning", ICML 2023.

[2] Hudson et al. "SODA: Bottleneck Diffusion Models for Representation Learning", 2023.