Addressing Sample Inefficiency in Multi-View Representation Learning

We thank the reviewer for their detailed feedback and highlighting the core contributions of our work. We are also grateful to the reviewer for appreciating that “writing is clear,” and our insights suggest a “strategy that reduces the data needed to learn good representations” that “may benefit the community.” We would like to clarify certain aspects of our work, especially our focus on the theoretical foundations of correlation-based non-contrastive self-supervised learning (NC-SSL) methods and the specific contributions of our research.

1. Emphasis on Theoretical Contributions:

We would like to clarify that we agree with the reviewer that multi-view or data augmentation for non-contrastive SSL are well-established tools in the literature (see Sec. 5, where we cover related work, including multi-view SSL) but do not claim this to be our core contribution. Instead, we aim to offer a principled and theoretically grounded understanding of the features learned by pretraining with correlation-based non-contrastive SSL frameworks, specifically Barlow Twins and VICReg. In particular, we show that pretraining with these loss functions recovers features that span the top eigenfunctions of the augmentation-defined data kernel. Building on these insights, we present practical recommendations to make SSL training more compute and sample efficient. Therefore, we offer a deeper understanding of these algorithms and why specific useful heuristics (e.g., using multiple augmentations) are helpful in good feature learning.

2. Longer Pretraining:

We thank the reviewer for sharing solo-learn [1] as a reference for several SSL algorithms pretrained for 1000 epochs. However, we would like to highlight that while solo-learn pretrains for 1000 epochs on CIFAR-10, they consider only Resnet-18 architectures that are computationally much cheaper than the Resnet-50 used in our experiments, making it difficult to run 1000-epoch pretraining on our compute budget. Nevertheless, we worked hard over the last few days to incorporate the reviewer’s feedback. We have updated Fig. 4 & 5 with 100-epoch pretraining on STL-10 and 400-epoch pretraining on CIFAR-10 (see Appendix A). Note that our core claims still hold, i.e., using multiple augmentations leads to good downstream task performance earlier in training.

3. Addressing Misinterpretations of 8 Augmentations Results:

We would like to clarify the misunderstanding regarding the performance of 8 augmentations.

• First, Fig. 4 demonstrates the utility of multiple augmentations in improving convergence, i.e., learning better features earlier in training using the full dataset. We have added results for 8 augmentations while pretraining with BarlowTwins loss on CIFAR-10 to demonstrate that increasing the number of augmentations leads to better downstream performance earlier in training (see Appendix A), in line with our claims.
• On the other hand, Fig. 5 inspects the limits of SSL feature learning with fewer unique unlabeled samples. To this end, we vary the dataset size such that the total number of samples (after augmentations) in each epoch is the same, i.e., we compare training on 100% dataset with 2 augmentations, 50% dataset with 4 augmentations, and 25% dataset with 8 augmentations. The reviewer is correct in pointing out that training with 25% of the dataset with 8 augmentations can slightly hurt downstream performance compared to training with 50% dataset with 4 augmentations. We note this is not a discrepancy between our theory and empirical results. Instead, this indicates that while multiple augmentations can recover competitive performance with fewer samples, there is a limit to their utility in the extremely low data regime. To further highlight this tradeoff between downstream task performance and the number of unique unlabeled samples, we vary the number of augmentations and dataset size to develop an improved Pareto frontier, as shown in Fig. 6.

4. On additional large-scale experiments and baselines:

a. Theoretical Corroboration: Our core contributions include theoretical results that explain why heuristics like multiple augmentations and high-dimensional projectors help with self-supervised pretraining. Our experiments demonstrate that these insights are practically valuable and translate to improvement in performance, convergence, and sample efficiency. In this context, CIFAR-10 and STL-10, while smaller in scale, are nonetheless relevant and provide a robust platform for demonstrating the efficacy of our method. Our results align well with theoretical expectations, indicating that our findings are not merely incidental but are rooted in solid theoretical understanding, and we expect this to translate to other settings, including experiments with larger datasets (such as Imagenet).

b. Adding Imagenet Experiments: While we acknowledge that ImageNet is a benchmark in representation learning, we must consider the practical constraints of computational resources. Given our current computing budgets, it is difficult to report results (with proper hyperparameter tuning) on ImageNet. However, we are trying hard to incorporate these results within the rebuttal deadline, and if our compute budget permits, we will follow-up the reviewer’s concern with results on ImageNet.

c. Comparisons with Other SSL Methods: Regarding the comparisons with other Self-Supervised Learning (SSL) methods like BYOL, SimSiam, and DINO, while we agree that a comprehensive comparison is valuable, we believe these comparisons are outside the scope of this work. In particular, we seek to establish theoretical foundations for correlation-based non-contrastive SSL. We aim to extend our theoretical framework to other families of non-contrastive SSL, with an eye toward practical recommendations to improve these methods.

In conclusion, our paper aims to strengthen the theoretical underpinnings of correlation-based SSL methods, providing novel and fundamental insights into the field. We appreciate the opportunity to clarify these aspects and believe our work contributes significantly to a deeper understanding of SSL. We hope this clarification addresses the concerns raised and demonstrates the value and rigor of our research.

We acknowledge the reviewer’s engagement with our paper. While some points raised and references cited by the reviewer are interesting and contribute to our overall understanding of non-contrastive SSL, we would like to highlight several aspects of the review that necessitate further discussion.

1. Note on ICLR review guidelines:

Most of the reviewer’s criticism of our theoretical contribution relies on [2] Zhai et al paper available on arXiv in June 2023. However, this contradicts the ICLR guidelines: “consider papers contemporaneous if they are published (available in online proceedings) within the last four months” (https://iclr.cc/Conferences/2024/ReviewerGuide#FAQ). The same goes for [4] Cabannes et al. 2023, published at ICML 2023 on July 03, 2023 $[proceedings](https://proceedings.mlr.press/v202/).$

2. Feedback on theoretical typos:

We thank the reviewer for pointing out some of the typos in the theory sections of our paper. Indeed, as the reviewer pointed out, we have a typo in the definition of $T_M$, which should follow the definition in the text (see the line below). We will update the writing in the theory section of our manuscript, per the reviewer’s suggestions, but we would like to point out that these do not change the core takeaway of our work. Moreover, we feel that the reviewer’s feedback either focuses on minor points that detract from a holistic evaluation of our work or lacks specific constructive feedback on improving our work. We would therefore request the reviewer to kindly offer more specific points or concerns they may have regarding the theoretical arguments presented in our work.

3. Misinterpretation of the scope of our work:

The reviewer cites several references ([1], [3], [4]) as superior or more definitive than our findings. However, these references predominantly offer theoretical and potential insights without concrete experimental validation. Furthermore, [1] & [3] specifically deal with contrastive learning frameworks (as the reviewer also notes). In the related work section, we mention how our work builds on insights from theoretical advances in contrastive SSL and cite [1]. While we agree with the reviewer that these works present useful theoretical insights, we strongly disagree that these works are “superior” to our work. On the contrary, we believe that our work presents complementary viewpoints, thereby improving the overall quality of scientific know-how in this research direction. For example, Section 6.2 in [4] mentions, “However, when the total amount $m \times n$ of pre-processed data is held fixed, it is generally better to process many inputs with two views $m = 2$, rather than a few inputs with many augmentations.” Our results (Fig. 5 & 6) show that this statement is not generally true in practical settings and that a tradeoff exists where multiple augmentations can sometimes yield better performance with fewer unlabeled samples. Therefore, it is indeed possible to use multiple augmentations to improve the sample efficiency of SSL algorithms, contrary to the recommendation provided by [4]. Unfortunately, the reviewer has currently overlooked these empirical contributions of our work.

We would like to reiterate that our work is a timely research (evidenced by concurrent submissions addressing similar questions) that marries theoretical insights with practical recommendations, thereby bridging the gap between SSL theory and practice. Specifically, we focus on non-contrastive SSL algorithms like BarlowTwins and VICReg. Based on the reviewer’s comments, we agree that it would be interesting to adopt a similar approach as our work and extend existing theoretical literature to present practical recommendations that apply to more general SSL algorithms. Therefore, our work, in addition to the concurrent works (mentioned by the reviewer), contributes to a better understanding of SSL algorithms and paves the way for more compute and sample-efficient SSL frameworks. However, that is outside the scope of our current paper.

In conclusion, we urge the reviewer to reassess our paper, considering its scope, content, and contributions rather than comparing/contrasting it against external publications, especially those not aligned with ICLR guidelines. We believe our work represents a significant step forward in SSL research by presenting a holistic perspective from theoretical insights to practical recommendations, thereby drawing the attention of SSL practitioners to recent theoretical developments.

Dear reviewer ArGT,

This is the supervisor of the student who wrote this paper replying to you now.

First, I want to clarify that I had approved the student's response, and I consider it to be appropriate. I say this as someone who has served as an AC for several ML conference for a few years now. Nothing in the initial response was insulting to you, nor did it say anything unfair. You perhaps disagreed with it, which is fine, and you can and should state so and provide a counter-rebuttal. But claiming that the rebuttal was so bad that the student should, and I quote, "...consult their advisors or senior collaborators about how to do research, how to write a paper, and how to interact with peer reviewers" is completely unfair and inappropriately aggressive. Notably, it was pointed out to you that you were criticising this paper for doing some of the same work as two contemporaneous papers (you stated it was a "known result"). And, as the initial rebuttal noted, that is a violation of ICLR guidelines on your part. You should simply have apologised for violating the guidelines or just clarified that you shared these papers simply for information's sake, not because you are claiming that this paper is not novel and/or should have cited them. That would be fine. Instead, you attacked the graduate student, their work, and their professionalism. This is shameful to see, and quite frankly, smacks of bullying.

Second, it is 100% fair to say that your critiques in point 2 were not about the theory, per se, but about the writing (indeed, you titled the point "On the mathematical writing of this work"). Perhaps "typo" is not quite the right word choice... But, the point of the rebuttal holds: your critiques are critiques on the writing style, not on the theory itself, and arguably, critiques of that nature fall into the category of minor points, as stated in the rebuttal.

Third, I want to end by noting the irony that unlike the initial rebuttal to your review, your reaction was deeply unprofessional. Speaking as an AC, and as a supervisor, I would encourage you to tone down your hostility and act with a greater deal of decorum in future interactions on conference reviews. Please don't needlessly bully students on OpenReview. Behaviour like this degrades our community and weakens our collective ability to effectively train the next generation of scientists.

Sincerely,

Senior author for paper 6318

I agree that this is not productive, and there is no need for further back and forth. We have your comments, which were helpful in some places, so thank you for those components. And as noted, everything is on the record for all to see, which I am also happy about. I stand by the claim that the student's initial rebuttal was perfectly professional, and did not insult you, but merely pointed out the fact that you had violated ICLR guidelines (which you did).

I also maintain that the student did not deserve to be spoken down to in the way you did. I think it is a real shame that you are unwilling to recognise and/or reflect on the unnecessary hostility in your reply.

We will also be commenting to the AC, of course.

Sincerely,

Senior author for paper 6318

We appreciate the time and effort the reviewer has dedicated to evaluating our paper. We are glad that the reviewer found our presentation and contribution good (with a score of 3/4), and acknowledge that our work presents “several enhancements to non-contrastive Self-Supervised Learning (SSL) method” and “offer practical recommendations […] which enhances the paper's utility for potential users”. Furthermore, the reviewer agrees that our core insights, specifically that not needing large projection dimensions for non-contrastive SSL, “constitutes a significant improvement” and that the “paper conducts a valuable comparison of the utility of multiview technique.”

Below, we would like to address the concerns raised by the reviewer.

1. On Experimental Results and Convergence:

We thank the reviewer for pointing out this apparent lack of clarity in the result figures. We would like to reiterate that our main claim is that using more augmentations will lead to faster convergence, i.e., learning better features earlier in pretraining. Our current results demonstrate that using multiple augmentations leads to better linear readout performance earlier in training, both in terms of full passes over the dataset (epochs) and wall clock time (see Fig 4). We incorporated the feedback and ran new experiments where we pretrain each model to 100 epochs (see updated Fig. 4 & 5) to align with standard practice in SSL pretraining of ResNet50 networks. However, the reviewer’s concerns are valid in settings with sufficient compute budget to pretrain models till convergence and select the best-performing checkpoint. To address this concern, we have run additional experiments wherein we pretrain ResNet50 models with BarlowTwins on CIFAR-10 up to 400 epochs and have added these new results (see Appendix A). In line with our claims, using multiple augmentations leads to similar (or even slightly better) performance much earlier in training.

2. Innovation and Utility of Multiview Technique:

We agree with the reviewer that using multiple views is a well-established heuristic in contrastive SSL frameworks like SwAV. However, our core contribution is two fold:

• We provide theoretical principles that explain why a multiview approach is useful in practice while extending this approach to non-contrastive learning setups like BarlowTwins and VICReg.
• Furthermore, our results demonstrate that the multiview approach not only yields good performance but is also useful for learning good features from fewer unlabeled samples. As highlighted by Reviewer QPCx our insights present “a strategy that reduces the data needed to learn good representations may benefit the community.”

We would also like to note that SwAV presents a useful empirical approach to use multiple augmentations and would be a special case of our theoretical framework, wherein using crops of smaller size leads to a particular choice of mapping $M$ (and correspondingly operator $T_M$). Nevertheless, we have also added some new experiments (see Appendix A) where we adopt a SwAV-style approach, i.e., 2 augmentations of full resolution and additional augmentations of smaller resolution, and demonstrate minor performance gains (at lesser compute cost than full-view augmentations) over the vanilla 2-view approach. While the decreased performance gain (as opposed to using full views for all augmentations) could stem from an implicit bias over the scale of details in learned features from the particular choice of $M$, these results still align with our theoretical conclusions regarding the utility of multiple augmentations. Taken together, we would like to disagree with the reviewer’s claim about the inferiority of our method compared to SwAV and instead present SwAV as a special (compute-efficient) case of our framework.

3. Memory Usage and Computation Efficiency:

The reviewer's point on memory usage and computation efficiency is well-taken. Our omission of an in-depth discussion on this aspect was due to space constraints and our primary focus on the theoretical aspects of SSL. However, we agree that this is an important factor and will add a section discussing computational resources and memory usage in the context of our proposed methods. Would such an addition address the reviewer's concerns in this area?

4. Connection Between Graphs and Main Idea:

We thank the reviewer for pointing out this apparent lack of clarity. The main idea of our paper is to make SSL more compute and sample efficient, i.e., to learn good features with a lower compute footprint from fewer unique unlabelled samples. Our results demonstrate that contrary to dominant heuristics for NC-SSL pretraining, it is possible to learn good features with smaller projector dimensionality (Fig. 3), thereby reducing the compute footprint of our models. Furthermore, our theoretical results suggest the potential of using multiple augmentations to learn good features with fewer passes over the dataset, i.e., training epochs (Fig. 4) as well as with fewer unique unlabeled samples (Fig. 5). Therefore, we believe these results are strongly aligned with the main idea in the paper but are happy to add further clarification in relevant sections. We would greatly appreciate specific feedback from the reviewer on this point.

In summary, we would like to thank the reviewer for supporting our paper with high scores on contribution and presentation (3/4) and hope that the above clarifications will improve the reviewer’s confidence in the soundness of our work. We are happy to clarify any other concerns that would potentially improve the reviewer’s score of our paper.

We thank the reviewer for their thorough review and insightful comments. We are glad the reviewer found that our paper “follows a good structure,” presenting compelling theoretical insights and “providing a deeper understanding of something that had previously been made to work by 'engineering tricks'.” We also appreciate your recognition of the paper's core contributions, where we “provide concrete improvements to two well-known non-contrastive SSL techniques.”

1. On training with full dataset with 4/8 augmentations :

We thank the reviewer for this valuable suggestion. Accordingly, we add new experiments with 8 augmentations with Barlow-Twins on the full CIFAR-10 dataset (Appendix A).

From the above table, it’s clear that increasing the number of augmentations improves downstream performance in line with our claim. However, using more augmentations also increases the per-epoch time, thereby increasing the overall experiment time. In other words, a tradeoff exists between the performance and the compute time. This tradeoff is further demonstrated in Fig. 6 of our manuscript, where we present a Pareto frontier of performance vs compute time by varying the number of augmentations and the fraction of the dataset. Therefore, given our computing budget, we have restricted our experiments to 8 augmentations.

In summary, we appreciate the constructive feedback and agree that the aspects highlighted for improvement are crucial for a more comprehensive understanding of non-contrastive SSL techniques. We are committed to exploring these areas and believe they will significantly enhance the depth and applicability of our research.