Augmenting Negative Representation for Continual Self-Supervised Learning

Q1: Figure 2 and the gradient of the proposed AugNeg loss

We apologize for any confusion caused by the unclear Figure 2. We have revised the figure to better align with the explanation in Equation 2. Additionally, we responded the reviewer's question on the gradient in Common Response 3.

Q2: Additional explanation on Equation 5

we responded the reviewer's question on Equation 5 in Common Response 4.

Weakness 1: The scenario of continual self-supervised learning and advantages of CSSL in resource saving

Thank you for your insightful question. To provide a detailed response to the reviewer's inquiry, we will reiterate the setting of continual self-supervised learning (CSSL) outlined in Section 3.1 and Figure 1(b). As depicted in Figure 1(b), the CSSL scenario considers a situation where the training dataset is incrementally added. Specifically, it allows sequential access to each task dataset ($D\_t$) during the training of the $t$-th task. However, it explicitly acknowledges the impossibility of accessing previous task datasets ($D\_{1, ..., t-1}$). In this context, the challenge arises from the unavailability of data sampling from the entire dataset ($D\_{1, ..., t}$), leading to the Low Diversity of Negative Samples problem, a key motivation in CSSL as discussed in Section 3.2 and illustrated in Figure 1(b).

Training with the CSSL scenario has two benefits: 1) reducing the demand for memory resource and 2) minimizing GPU cost for training. In the same CSSL scenario mentioned earlier, consistently training on the entirety of data (Joint) can be considered a conventional training approach. Consequently, when the size of each task dataset is constant (e.g., $\|D\_t\| = N$), retraining the Joint model at each $t$ task incurs linear increases in both memory and GPU resources (e.g., $N\*t$). Considering the billing structure of major cloud services (e.g., AWS) based on resource usage over a unit period, the total cost for memory and GPU resources until obtaining the $t$-th Joint model is approximately $N\*(t+1)\*t/2$. In contrast, for CSSL, the total memory cost is $N\*t$ to obtain the $t$-th model since it uses only the new $t$-th task dataset for each task $t$. Regarding GPU resources, with an additional cost $\alpha$ introduced by the CSSL algorithm, the approximate cost becomes $N + (N + \alpha)\*(t-1)$.

In summary, when considering the cost of memory and GPU resources in each learning scenario, CSSL algorithms offer a significant cost reduction compared to the Joint. Furthermore, the cost efficiency of CSSL algorithms becomes more pronounced as the size of each task's training dataset increases (i.e., as N increases). Taking this cost efficiency into account, we believe that achieving performance levels of the proposed AugNeg compared to the Joint, from approximately 93% (AugNeg(MoCo) on CIFAR-100, Class-IL 5T) to 96% (AugNeg(BYOL) on DomainNet, Domain-IL 6T), is a substantial and meaningful outcome.

Weakness 2: Additional experiments in ImageNet-1k

We agree that presenting additional experimental results on ImageNet-1k would contribute significantly to validating the efficiency of our algorithm. We are currently conducting them as supplementary experiments. However, given our current limited GPU resources, obtaining the complete set of experimental results may take considerable time. If we manage to complete any partial results before the end of the author response period, we commit to sharing those results.

Weakness 3: Additional related works on augmentation in SSL

Thank you for sharing valuable papers. Upon comparing our paper with the ones you provided, we believe our distinctive contribution lies in proposing to consider additional negative samples obtained from both the current and previous models in Continual Self-Supervised Learning (CSSL) scenarios. From this perspective, we acknowledge the importance of comparing and discussing our method with those designed for successful self-supervised learning in the conventional single-task setting. In a forthcoming revision, we will update the related work section to incorporate discussions on augmentation methods including the papers the reviewer shared.

Weakness 0: Changes from the previous submission version

Firstly, we regret that the extent of our paper's updates might not meet the reviewer's satisfaction. We appreciate all the feedback received so far and have made diligent efforts to enhance the paper based on the provided comments. The updates in this submission include:

1. Title and Algorithm Names: revised the paper title and the names of the proposed algorithm.

2. Introduction Enhancement: Improved the final paragraph in the Introduction.

3. Method Section:

   • Attenuated the discussion on an issue with conventional loss function and introduced a discussion on the problem of the low diversity of negatives.
   • Newly introduced the idea of extending AugNeg to non-contrastive learning in Section 3.4. Analytical details on existing gradients and other specifics have been relocated to the Supplementary Material.

4. Experimental Section Changes:

   • Re-performed AugNeg experiments on CIFAR-100, with results reported in Figure 3 and Table 1.
   • Tried to address the reproduce issue of CaSSLe in Table 3, essentially re-conducting the main experiments. As a result, achieved slightly improved results consistently compared to the previous version.
   • Added more experimental scenarios for downstream task.
   • Included experiments in the Supplementary Materials covering the size of the Proj(MLP) layer, results based on the queue size in AugNeg(MoCo), and experiments with an increased mini-batch size.

Also, we will further enhance our paper by incorporating the comments received in this review through additional revisions.

Q1: Experimental results using other hyperparameters

We already reported experimental results on various dimensions of MLP, various queue sizes, and large mini-batch size in the Supplementary Materials.

Q2: Experimental result on Domain-IL

As mentioned before, we have re-conducted most of the experiments while writing the current paper, so we redrew the figures from scratch. Upon reviewing the previous version of the paper, it seems there was a typo in the table. We apologize for this typo. We have double-checked all results, and there are no typos in the currently drawn tables and figures.

Q3: Experimental result of MoCo with a extended queue size

Firstly, in this paper, the MoCo experiments used a default queue size of 65,536. We believe the reviewer's intention is to inquire about the results when using a queue of size 65,536 $\times$ 2 in MoCo + CaSSLe (please clarify if otherwise). The results for this experiment are already presented as one of the ablation studies in Case 4 of Table 5 in the manuscript. From this experiment, we experimentally validated that using additional negatives is distinct from simply increasing the queue size, as MoCo + CaSSLe with a doubled queue size (60,09) shows little difference compared to the original CaSSLe results (60.11).

Weakness 1: Improvements presented in Table 1 is not significent

While the numeric improvement of AugNeg over CaSSLe may appear modest, we believe it gains a competitive edge when considering performance of Joint. For example, AugNeg(MoCo) shows a relatively modest increase of 3.6% over MoCo + CaSSLe. However, considering that MoCo + CaSSLe already achieves performance close to the upper bound (Joint) at around 89.8%, the fact that AugNeg(MoCo) reaches the Joint's performance of 93.2% with a similar training cost and low standard deviation is deemed a meaningful improvement. Moreover, we assert that our AugNeg variants demonstrate superior performance in other experiments, including ImageNet-100, DomainNet, and downstream tasks. We would appreciate the reviewer considering these points.

Weakness 2: Reproduce issue of CaSSLe

We made diligent efforts to reproduce the CaSSLe experiments in the manuscript, but unfortunately, we were unable to achieve results identical to those reported in the CaSSLe paper. With the Github issue #12 concerning BYOL in the ImageNet-100 experiment, there is issue #10 discussing reproducibility concerns in the ImageNet-100 experiment. Note both indicate a common trend of obtaining results 5-6% lower than those reported in the original paper, aligning with the tendencies observed in our experimental results. Also, these issues remain unresolved, and the authors have not provided a comprehensive solution.

We hope the reviewer has taken into account the current challenging situation regarding reproducibility. Additionally, we conducted all experiments of our paper in the unified environment and have made the experimental code publicly available. We would appreciate it if these considerations could be taken into account in the final evaluation.

Weakness 3: Suggestion for Figure 4

Thank you for your valuable feedback. We have updated Figure 4 in the manuscript and made corresponding modifications according to the reviewer's comments. Additionally, in the Supplementary Materials, we included the same analysis on AugNeg(BYOL), so please take a look. The discrepancy in Task 0 performance between MoCo + CaSSLe and AugNeg(MoCo) in the original graph was due to a typo. As evident in Figure 3, for all experiments in our paper, we initiated training with the same Task 0 model, conducting both CaSSLe and AugNeg, and reported the results accordingly. We apologize for the typo and have corrected it.

Weakness 4: No clear message

See Common Response 3 and 4. We will update our paper following these responses.

Dear Reviewer Tkp5,

Thank you once again for providing insightful comments. We have made every effort to respond to all your comments within the limited discussion period and have completed the second revision, incorporating all reviewer’s comment into the manuscript.

We want to reiterate that while our proposed AugNeg is a simple idea, its core contribution lies in identifying issues from a different perspective in the field of CSSL research (specifically, pointing out the problem of both 1) the conventional loss form and 2) the low diversity of negative samples). We have introduced a novel loss function to overcome these challenges via model-based augmentation of negative representations. Additionally, through diverse experiments using not only CIFAR-100 but also ImageNet-100, DomainNet and downstream tasks, we experimentally demonstrated that AugNeg achieves consistent performance improvement with little difference in training cost compared to CaSSLe. Furthermore, we believe that the analysis suggested by the reviewer clearly highlights the advantages of AugNeg, confirming its ability to achieve enhanced both plasticity and stability, as aligned with the Method Section.

As the discussion period is coming to a close, we would greatly appreciate it if the reviewer could review our response (and ideally the revised manuscript) and provide feedback. Thank you again.

Weakness 2: Captions and Figures

We have revised and updated the captions for all Figures and Tables in the manuscript.

Weakness 3 and Q1) The more intuitive explanation for AugNeg Loss

Thank you for the question. We have summarized our response regarding the role of additional negatives of the proposed AugNeg Loss in Common Response 3, so please check that for details.

Additionally, we would like to clarify that $\mathcal{N_{t-1}}$ is not $\mathcal{N_{t}}$ employed in $\mathcal{L}_1$ during the previous task (t-1) training. As stated in the manuscript, these refer to negatives obtained from $t$ and $t-1$ models during training task $t$. Their implementation varies based on the contrastive learning algorithm; for SimCLR, all samples in the current mini-batch, excluding the identical sample, are considered as negatives. While for MoCo, the entire queue containing negatives used in previous iterations as a positive serves as negatives for the current iteration. At each training iteration for each task, even with the same input mini-batch, the $t$ and $t-1$ models have distinct output features for negative samples (as the $t$ model continues to update, while the $t-1$ model remains frozen). The core idea behind the AugNeg Loss proposed in Equation (2) is to concurrently consider these diverse features in CSSL.

Q2) Projection layer used in CaSSLe

Indeed, we utilized it. However, in Section 3.3, $z_{i, t} = Proj(h_{\theta_t}(x_i))$ represents the standard Projection layer commonly employed in SSL. We defined the Predictor for CSSL as $\tilde{z}\_{i, t} = Pred(z\_{i,t})$ and employed them to express the AugNeg Loss.

Q3) Question on extra regularization

Please refer to Common Response 4

Weakness 1: Applicability of AugNeg to other methods

We appreciate the valuable suggestion. The rationale behind conducting experiments focused on MoCo V2, SimCLR, BarLow, and BYOL is rooted in the findings of the CaSSLe paper, where these algorithms demonstrated superior performance across the majority of datasets, with MoCo V2 and BYOL particularly excelling. In alignment with the reviewer's suggestion, we conducted additional experiments by selecting one contrastive and one non-contrastive learning-based algorithm each to explore the applicability of AugNeg to various continual representation learning scenarios.

Given that results for prominent contrastive learning-based algorithms (SimCLR, MoCo V2) were already presented in the manuscript, we introduced Supervised Contrastive Learning(SupCon) [1] as an additional algorithm to investigate the effectiveness of AugNeg in scenarios where labels are available. To accomplish this, we modified the implementation of SimCLR in Equation (2) of the manuscript, adapting it to utilize supervised labels, specifically incorporating additional negatives in SupCon + CaSSLe.

For the Non-contrastive learning-based algorithm, we selected the recent VICReg [2] algorithm to explore the potential application of AugNeg to various SSL algorithms. To implement AugNeg, we added additional regularization proposed in Equation (5) of the manuscript and conducted experiments accordingly.

We conducted experiments with the CIFAR-100 and ImageNet-100 datasets, employing the Class-IL (5T, 10T) scenario as our experimental setting, and the results are as follows.

Based on the results of the above experiments, we observe that AugNeg can be applied to additional contrastive and non-contrastive learning algorithms, leading to consistent performance improvement compared to CaSSLe. In particular, we observed successful application of our proposed AugNeg to VICReg, one of the latest SSL algorithms, achieving consistent performance improvements of 2-3% in both CIFAR-100 and ImageNet-100 experiments. We believe that these experimental findings suggest the potential for the proposed AugNeg to be widely applicable to future various contrastive and non-contrastive learning algorithms.

[1] Khosla, Prannay, et al. "Supervised contrastive learning." Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020.

[2] Bardes, Adrien, Jean Ponce, and Yann LeCun. "VICREG: VARIANCE-INVARIANCE-COVARIANCE REGULARIZATION FOR SELF-SUPERVISED LEARNING." 10th International Conference on Learning Representations, ICLR 2022. 2022.

-- revision of the response --

We discovered a error in the experiment with AugNeg(VICReg) on ImageNet-100. After rectifying the error and re-running the experiment, we have input the revised results into the table and adjusted the response accordingly. We apologize for any inconvenience caused by this.

Dear Reviewer E4Aj,

Thank you once again for providing insightful comments. We have made every effort to respond to all your comments within the limited discussion period and have completed the second revision, incorporating all reviewer’s comment into the manuscript.

We want to reiterate that while our proposed AugNeg is a simple idea, its core contribution lies in identifying issues from a different perspective in the field of CSSL research (specifically, pointing out the problem of both 1) the conventional loss form and 2) the low diversity of negative samples). We have introduced a novel loss function to overcome these challenges via model-based augmentation of negative representations. Furthermore, we assert that our contributions are most evident in the successful application of AugNeg to six methods, including additional experiments conducted in response to Reviewer E4Aj. This demonstrates the versatility of our proposed idea in CSSL, involving both contrastive and non-contrastive algorithms. In particular, we believe that the successful integration of AugNeg with MoCo and BYOL, which exhibit the best performance in the CSSL setting, is remarkable. This integration (AugNeg with MoCo and BYOL) leads to the new state-of-the-art performance in various experiments. Additionally, we believe our approach holds valuable potential for application and extension to continual representation learning in different domains, such as multi-modal representation learning.

As part of additional experiments, we are currently conducting supplementary experiments on AugNeg(VICReg) using ImageNet-100 and DomainNet in three CSSL scenarios. We plan to incorporate these results in the final camera ready version. Furthermore, we are in the process of extending our application experiments to other SSL algorithms, aiming to include the results in the final camera ready version also.

As the discussion period is coming to a close, we would greatly appreciate it if the reviewer could review our response (and ideally the revised manuscript) and provide feedback. Thank you again.