Understanding the Benefits of SimCLR Pre-Training in Two-Layer Convolutional Neural Networks

Thanks for your detailed review. Our answers to your question are as follows.

---

Weaknesses:

W1&Q1:

• W1: Impact of the contributions. The paper studied a very specific setting: simple binary classification task and a two-layers ConvNet with $RELU^q$ output activation. In practice, this setting is not used often. For instance, SimCLR paper uses a ResNet-50 and was used on multiple multiclass classification benchmarks. There is a big gap between the theoretical setting used in this paper and the practical setting, so it is difficult to understand the potential impact of this paper contributions. It may be difficult to do, but it would be interesting to relax some of the assumptions to be able to get theoretical for a family of models which include ResNet-50.
• Q1: The paper should discuss the scope of the contributions, as the theoretical setting seems quite different from the practical setting. In particular, is it possible to remove or relax some assumptions to include a larger family of models?

AW1&AQ1: We admit that it requires some specific assumptions to provide an accurate theoretical analysis. We would also like to explain that we choose to study two-layer CNNs because it allows a direct comparison with the results of direct supervised learning on two-layer CNNs in [1]. However, even for this simple setting and two-layer CNN architecture, the influence of contrastive learning on supervised fine-tuning has not been analyzed in theory before. We believe that our current analysis provides a clear first step towards the theoretical understanding of the advantages of contrastive learning.

In our theoretical analysis, we are following a signal learning framework. We believe that the insight behind this signal learning framework can help us understand many practical learning tasks as well.

The idea of the signal learning framework is that, when training complicated neural networks on a finite training dataset, there may exist many (probably infinitely many) different parameter configurations that can all lead to $100\\%$ training accuracy. However, neural networks with different parameter configurations may perform drastically differently on test data. We can generally treat the “information in the data input that are related to the label” (characterized by the vector $\boldsymbol{\mu}$ in our theoretical study) as the “signal” in the data, and treat the “information in the data input that are not related to the label” (characterized by the Gaussian noise vector $\boldsymbol{\xi}$ in our theoretical study) as “noises”. Then clearly, The signal is the label-dependent and generalizable part of the information, and noise is the label-independent and ungeneralizable part of the information, and whether the obtained neural network after training can achieve good prediction depends on whether it utilizes the “signal part” or the “noise part” to fit the training data, and this is exactly how we build our theory.

Our analysis on SimCLR demonstrates that, although SimCLR does not utilize any labeled data, the knowledge that “data augmentation should not change the label” suffices to help the neural network to capture more signals in the pre-training stage. We believe this characterization is true for more practical scenarios as well. The idea it conveyed that SimCLR pre-training encourages signal learning in the supervised learning of CNNs also holds for more general cases. Although it is difficult to verify on real data because “signals” and “noises” and the level they are captured by a more complicated network is hard to quantify, we have provided additional real-data experiments to justify our claim (to a certain extent) in the revision. The added experiments can be found in Appendix A in the revised paper.

For extension to more complicated architecture, currently there is a lack of tools and techniques to enable such a performance comparison with directly supervised learning in more modern models. Even some of the direct supervised learning has not been well studied in theory in these modern models. It is one of the important future directions to extend the current analysis of the influence of contrastive learning on supervised learning on CNNs to more complicated models.

---

Q4: Why does the paper focus $ReLU^q$ activation function for the theoretical analysis? Is it possible to include other activation functions?

AQ4: In this paper, we analyze the advantage of SimCLR pre-training by comparing the performance of SimCLR pre-training followed by fine-tuning with direct supervised learning in [1]. Therefore, we adopt the same two-layer CNN model as that in [1] to allow a direct comparison.

It is possible to conduct similar sets of analysis for CNNs with other activation functions. However, since CNNs with other activation functions may have different training dynamics, the corresponding conditions and assumptions may also vary.

---

Q5: What are the assumptions about $\mu$? Can $\mu$ be the zero/null vector?

AQ5: $\boldsymbol{\mu}$ here refers to the signal patch of the data in this problem. Apart from the requirement on the signal-to-noise ratio (SNR) in Condition 4.1, we do not pose any extra assumptions on the signal $\boldsymbol{\mu}$. Since SNR is defined as $\mathrm{SNR}=\||\boldsymbol{\mu}\||_2/(\sigma_p \sqrt{d})$ and we require $n_0\cdot\mathrm{SNR}^2=\tilde\Omega(1)$ in Condition 4.1 to achieve a small test loss, it implies that the signal $\boldsymbol{\mu}$ can not be a zero vector.

---

Reference:

[1] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. NeurIPS 35:25237–25250, 2022.

[2] Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In ICML, pages 9965–10040. PMLR, 2022.

[3] Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer relu convolutional neural networks. In ICML, pages 17615–17659. PMLR, 2023.

---

W2: Lack of justification/motivation. The paper has many assumptions that are not well motivated, or claims that are not well justified.

AW2: The following are our explanations of the points mentioned in W2.

• L162: SimCLR is one of the most popular contrastive learning methods for vision tasks. The data model adopted in this paper (Definition 3.1) is also inspired by the image data: the image inputs consist of different patches. Within these patches, only some of the patches are related to the class label of the image, while other patches are unrelated to the labels. Thus, the data model is designed to have a signal patch $y_i\cdot\boldsymbol{\mu}$ and a noise patch $\boldsymbol{\xi}$ that is irrelevant to the label [1]. Therefore, we claim that this data model with such design is suitable to study SimCLR.

• L153:­­ Please note that we consider convolution filters and data patches that are both flattened into vectors. The image patches, which are 2-dim in nature, are reshaped into a vector, and the convolution filters are also reshaped into vectors of the same size, and the inner product between the vectorized filters and image patches are exactly equivalent to the desired computation when both the filters and the image patches are in 2-dim shapes. Therefore, our analysis exactly covers 2-dim images, or even 3-d tensors formed by 2-dim images with RGB channels.

• L215: $ReLU^q$ activation function with q > 2 is chosen here since the proof relies on the smoothness assumption.

• The data augmentation assumptions &Q3: The data augmentation follows the following rules. For the data $\tilde{\boldsymbol{x}}$, an augmented data point $\tilde{\boldsymbol{x}}^{\text{pre-training}}$ is generated from the distribution $\mathbb{P}(\boldsymbol{x}|y= y_i)$. Given this distribution, one patch of $\tilde{\boldsymbol{x}}^{\text{pre-training}}$ is assigned as $y_i\cdot\boldsymbol{\mu}$ and the other patch of $\tilde{\boldsymbol{x}}^{\text{pre-training}}$ is assigned as $\tilde{\boldsymbol{\xi}}_i \sim \mathcal{N}(\mathbf{0}, \sigma_p^2\cdot(\boldsymbol{I} -\boldsymbol{\mu}\boldsymbol{\mu}^\top\cdot\| |\boldsymbol{\mu}\||_2^{-2}))$.

• $\mu$: The vector $\mu$ refers to the signal and we do not have any extra assumption posed to it apart from the certain condition on signal-to-noise ratio ($\mathrm{SNR}$) mentioned in Condition 4.1.

---

W3: Clarity. The presentation and clarity of the paper could be enhanced. The paper includes a considerable number of notations, which can make it challenging to keep track of them all. Some notations appear to be undefined, while others are introduced later in the text, making it difficult to fully understand them until the subsequent paragraphs are read.

AW3: Thanks for pointing out this. We have revised the paper to make these notations clearer.

• $\Omega$: $\Omega$ is introduced in the notation section. Due to the relatively fixed structure of the introduction section, the notation part could only be put at the end of the introduction section, around L106. We have added a footnote for $\Omega$ in L84 to explain it.

• $\boldsymbol{W}$ refers to the matrix of filters, $\boldsymbol{w}_r$ refers to the r-th row of the matrix $W$, namely the r-th filter.

• $[F(W,x)]_r$ refers to the r-th dimension of the output. Here,we adopt the way of first proposing the equation, and then explaining its mechanism and all the terms within it one by one.

• $[n_0]$ refers to the set $\\{1,2,\dots,n_0\\}$. We have already added this explanation of it in the Notation section.

• L200: We have revised this to specify the dataset used in the loss function computation $L_{S_{unlabeled}}(\boldsymbol{W})$.

• L158: This data model definition has already been applied to a series of works [1,2,3]. We believe there isn’t any problem with it. We are happy to answer any questions if you still feel there are some specific problems with it.

---

W4: Things to improve the paper that did not impact the score: A lot of equations do not have number so it is difficult to reference them in the review.

AW4: Actually, we are following the style where equations in the paper are numbered only if they are referenced in the paper. We believe it is common to follow such a style for numbering equations.

---

Questions:

Q2: L279: "we remark that most of these assumptions are non-essential." It seems to be counter-intuitive to make some assumptions that are non-essential. How do the theoretical results change if these non-essential assumptions are removed?

AQ2: The cited sentence here means most of these assumptions are easy to satisfy for most cases, and thus we only need to focus on the first two key conditions in Condition 4.1. We have revised this sentence to “we remark that most of these assumptions are easy to satisfy” in the paper, making it more precise.

---

---

Thank you for your response. We would like to further address your concern regarding the gap between the two-layer CNNs considered in this paper and practical applications as follows.

• In [1], the authors study the direct supervised learning of two-layer CNNs. The goal of our paper is to establish comparable theoretical guarantees for SimCLR pre-training followed by supervised fine-tuning, allowing us to demonstrate how SimCLR enhances learning performance in comparison to [1]. Therefore, we believe that studying two-layer CNNs is a natural and appropriate choice that serves our purpose well.

• Please note that even for direct supervised learning, results similar to those in [1] have not been established for deeper models, and similar studies [2,3,4] mostly focus on two-layer models. Compared to previous works, our research examines the more complicated procedure of “SimCLR pre-training -> remove projection head -> attach classification layer -> supervised fine-tuning”, which is more challenging to study. Therefore, while we agree that there is a big gap between our theory and the practice, we must emphasize that this gap exists in most existing theoretical studies. We sincerely hope that our technical contribution in developing a framework for the theoretical analysis of SimCLR is recognized, as we believe our novel analysis tools may help advance theoretical studies towards more practical scenarios.

Although our paper focuses on two-layer CNNs, we believe the insights obtained from our theoretical study are applicable to more general settings. As demonstrated in the additional experiments on ViTs in Appendix A in the latest revision of the paper, the phenomenon that SimCLR pre-training enhances signal learning during supervised fine-tuning is also observed. We believe our theoretical analysis represents an important step towards theoretically understanding the benefits of SimCLR pre-training.

---

Reference:

[1] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. NeurIPS 35:25237–25250, 2022.

[2] Spencer Frei, Niladri S. Chatterji, and Peter L. Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. COLT, 2022.

[3] Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization. NeurIPS, 2023.

[4] Yiwen Kou, Zixiang Chen, Yuan Cao, and Quanquan Gu. How does semi-supervised learning with pseudo-labelers work? a case study. ICLR, 2023.

---

Thank you for your detailed comments and suggestions. We address your concerns as follows.

---

Weaknesses:

W1: The paper aims to understand how SSL methods like SimCLR benefit downstream fine-tuning. However, the literature review provided is insufficient to comprehensively cover this topic, which has been widely studied both theoretically and empirically. A more thorough literature review on SSL's impact on fine-tuning performance is necessary.

AW1: Thanks for your comment. We have revised the paper to have a more thorough literature review about SSL's influence on fine-tuning performance. Please feel free to point out if you find any important papers missing here.

---

W2: The problem setting of 2-layer CNN presented appears overly simplified, potentially deviating from the practical deep learning problems that commonly employ SSL methods such as SimCLR. While deriving theoretical guarantees for modern architectures like transformers is challenging, a more realistic framing of the problem would better justify its relevance and importance.

AW2: To the best of our knowledge, the influence of contrastive learning on supervised fine-tuning has not been well studied in theory even for the simple two-layer CNN model. As a theoretical paper, it inevitably requires specific simplification of models and assumptions to have an accurate and clear analysis of the advantage of SimCLR pre-training on the fine-tuning of two-layer CNNs. This model allows a direct comparison with the results of direct supervised learning on two-layer CNNs in [1]. We believe that our current analysis is a clear first step towards the theoretical understanding of the advantages of contrastive learning.

---

W3: The experiments in the appendix seem questionable in terms of their validity and relevance to the stated problem. Typically, SimCLR pretraining leverages large-scale unlabeled datasets. However, the experiments employ only 250 and 200 unlabeled data points for the synthetic data experiment and real data experiment, respectively. These small dataset sizes are atypical for SimCLR pretraining and may not represent the true behaviors of SimCLR pretraining combined with downstream fine-tuning tasks.

AW3: In the revised version of the paper, we have added experiments with large-scale unlabeled datasets for both synthetic-data and real-world data (MNIST) experiments in Appendix A. The synthetic-data experiment with large-scale unlabeled data ($n_0=100000$) is presented in Figure 5. The real-world data experiment with large-scale unlabeled data ($n_0=13800$) is presented in Figure 6. In these large-scale unlabeled datasets experiments, we could still observe similar results as before. Compared with direct supervised learning, SimCLR pre-training followed by supervised fine-tuning may require far less labeled data to achieve a high test accuracy.

---

Questions:

Q1: How would the theorem transfer to more modern backbone architectures such as ViT? Can you provide more experiment ablation on more architectures?

AQ1: Our current analysis focuses on relatively simple two-layer CNNs, because SimCLR has not been well understood in theory even in training such simple models. Therefore, it is natural to first consider our current setting, which is a relatively cleaner and more standard setting for theoretical studies that has been considered in a series of previous works [2,3,4].

For more complicated architecture, currently there is a lack of tools and techniques to enable such a performance comparison with directly supervised learning in more modern models. Analysis of the influence of contrastive learning on supervised learning of more complicated models is an important future direction.

In terms of experiments on more complicated architectures, we would like to point out that architectures like ViTs are orthogonal to our study. We believe the significance of theoretical analyses on CNNs should not be denied because the theory does not cover ViT.

Moreover, we would also like to clarify that, unlike two-layer models where we can use the inner products between network weights and “signal” or “noise” vectors to characterize the level of “signal learning” or “noise memorization”, for more complicated models there may not be any convincing metric on the the level of “signal learning” or “noise memorization” by the model. Therefore, even if we add experiments on ViTs or other models, the observations would simply be like “the model makes worse predictions on more noisy data, and makes better predictions on less noisy data”. We believe such results may not be sufficiently meaningful.

---

---

Q2: Can you provide more ablation on the SNR rate and the number of the unlabeld data / labeled data?

AQ2: Thanks for your suggestions. We have added several groups of experiments with different signal-to-noise ratio ($\mathrm{SNR}$), the size of unlabeled data $n_0$ and size of labeled data $n$ in the Appendix A. The key takeaways from these experiments are summarized as follows.

• The unlabeled data size $n_0$ in Figure 7: While all other conditions remain the same, larger size of unlabeled pre-training data $n_0$ achieves a better performance in the fine-tuning.

• Signal-to-noise ratio ($\mathrm{SNR}$) in Figure 8: For experiments with smaller $\mathrm{SNR}$, it requires more (labeled or unlabeled) data to achieve a good test performance.

• Labeled data size $n$ in Figure 9: In the SimCLR pre-training followed by supervised fine-tuning, given a satisfactory number of unlabeled data $n_0$, the condition on the size of labeled data $n$ to achieve a high test accuracy is mild.

---

Q3: What happens if the representation does not follow isotropic Gaussian distribution? Typically SimCLR type of loss results in non-isotropic representations, in that case, does the theorem still hold just by simply replacing the variance term in the SNR equation with the maximum variance across different embedding dimensions?

AQ3: It is possible to generalize our data model to the data distribution with different scales of Gaussian noise across different dimensions. In this setting, the signal-to-noise rate (SNR) will be defined as $\mathrm{SNR}=\||\boldsymbol{\mu}\||/\sqrt{\max_{k\in[d]}\mathbb{E}[\||\xi^{(k)}\||^2]\cdot d}$, where $\xi^{(k)}$ is the k-th dim of $\boldsymbol{\xi}$. The difference between this and our current definition of SNR is that the norm of noise is replaced with the maximum of the norm of noise entry.

In addition, we would like to point out that, even in our current setting, the representations learned by SimCLR are non-isotropic representations (see the results of Theorem 5.3). Even if the data are isotropic, the first hidden layer outputs of the CNN trained by SimCLR are not isotropic.

---

Reference:

[1] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. NeurIPS 35:25237–25250, 2022.

[2] Spencer Frei, Niladri S. Chatterji, and Peter L. Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. COLT, 2022.

[3] Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization. NeurIPS, 2023.

[4] Yiwen Kou, Zixiang Chen, Yuan Cao, and Quanquan Gu. How does semi-supervised learning with pseudo-labelers work? a case study. ICLR, 2023.

---

W2: I generally feel a lack of discussion about how the theory presented in this paper can be related to the real-world use of SimCLR. I think this is because the paper omits enough context about how can we really ensure that the simplifying assumptions made here are mild enough, so that it can be eventually connected to the real-world.

AW2: First of all, we would like to mention that the accurate theoretical analysis of the training process should be studied in some specific assumptions. Our assumptions are not stronger than the existing theoretical works such as [4]. We adopt the same data model to allow a direct comparison of the performance of SimCLR pre-training followed by supervised learning with that of directly supervised learning in [4].

Regarding the real-world use, in our theoretical analysis, we are following a signal learning framework. We believe that the insight behind this signal learning framework can help us understand many practical learning tasks as well.

The idea of the signal learning framework is that, when training complicated neural networks on a finite training dataset, there may exist many (probably infinitely many) different parameter configurations that can all lead to $100\\%$ training accuracy. However, neural networks with different parameter configurations may perform drastically differently on test data. We can generally treat the “information in the data input that are related to the label” (characterized by the vector $\boldsymbol{\mu}$ in our theoretical study) as the “signal” in the data, and treat the “information in the data input that are not related to the label” (characterized by the Gaussian noise vector $\boldsymbol{\xi}$ in our theoretical study) as “noises”. Then clearly, The signal is the label-dependent and generalizable part of the information, and noise is the label-independent and ungeneralizable part of the information, and whether the obtained neural network after training can achieve good prediction depends on whether it utilizes the “signal part” or the “noise part” to fit the training data, and this is exactly how we build our theory.

Our analysis on SimCLR demonstrates that, although SimCLR does not utilize any labeled data, the knowledge that “data augmentation should not change the label” suffices to help the neural network to capture more signals in the pre-training stage. We believe this characterization is true for more practical scenarios as well. The idea it conveyed that SimCLR pre-training encourages signal learning in the supervised learning of CNNs also holds for more general cases. Although it is difficult to verify on real data because “signals” and “noises” and the level they are captured by a more complicated network is hard to quantify, we have provided additional real-data experiments to justify our claim (to a certain extent) in the revision. The added experiments can be found in Appendix A in the revised paper.

---

W3: The main body of the paper is quite theory-biased, and allocates its significant portion with the proof sketch. This may narrow down the potential audience, and I personally think the authors might consider to give more highlight on their empirical results as well as the theoretical ones.

AW3: We would like to point out that this paper is exactly a theory paper. It aims to theoretically prove the advantage of contrastive learning on fine-tuning by providing a detailed overall theoretical analysis of the SimCLR pre-training followed by supervised fine-tuning.

Empirical experiments on both synthetic and real-world datasets are conducted to verify our theoretical results. We have included the key takeaway of the experiments in Remark 4.4 in the main body of the paper. Due to the page limit, we include more experiment details in Appendix A. In the revised version of the paper, we have added more experiments for both synthetic-data and real-world dataset in Appendix A.

---

Questions:

Q1: Is there any potential that the analysis presented in this paper can be generalized to more complex architecture?

AQ1: First of all, we would like to point out that, to the best of our knowledge, the influence of contrastive learning on supervised fine-tuning has not been analyzed in theory before even for this simple two-layer CNN architecture. We believe that our current analysis provides a clear and meaningful first step towards a better theoretical understanding of the advantages of contrastive learning.

We would also like to explain that we choose to study two-layer CNNs because it allows a direct comparison with the results of direct supervised learning on two-layer CNNs in [4]. For more complicated architectures, even direct supervised learning has not been well studied in theory. We believe studying the performance of SimCLR (as well as direct supervised learning) on more complicated neural network architectures is an important future work direction.

---

Q2: The overall setup assumes that the labeled data at fine-tuning stage is sampled from the same distribution where the pre-training data is sampled, considering a pure unsupervised learning scenario. But one important aspect of SimCLR like training is in its transferrability; as such, could the analysis can be generalized in any way if we relax the assumption of the labeled data to another distribution? For example, I am curious if the analysis could be extended when the labeled data is more noisy that the pre-training data.

AQ2: As you have mentioned, the setting where SimCLR pre-training and supervised fine-tuning use data from different distributions can be understood as combining our current setting with transfer learning. We do believe that considering the influence of pre-training on different downstream tasks is a good future research direction. However, we would like to point out that even the ‘pure’ contrastive learning setting studied in our paper has not been well understood in theory, and therefore, it is more natural to consider our current setting, which is arguably cleaner for theoretical studies.

---

Q3: In my understanding, Theorem 4.3 implies that one still require the unlabeled sample size for SimCLR, $n_0$, at the similar rate of those for supervised learning to optimize the loss. But could this result be due to that the data considered is too easy to discriminate, even without labels, and perhaps the complexity of SimCLR can be much worse under a more challenging data assumption?

AQ3: We believe this is a misunderstanding. The condition on $n_0$ in our paper is $n_0\cdot \mathrm{SNR}^2 = \tilde\Omega(1)$ (in Condition 4.1), it is not the same as the condition $n\cdot\mathrm{SNR}^{q}=\tilde\Omega(1)$ (for $q>2$) (Condition 4.2 in [4]) for the number of labeled data $n$ in direct supervised learning in [4].

We would also like to emphasize that the data model is the standard data adopted by a series of theoretical works [4,5,6]. We adopt the same data model to allow a direct comparison of the performance of SimCLR pre-training followed by supervised learning with that of directly supervised learning in [4].

---

Reference：

[1] Yamini Bansal, Gal Kaplun, and Boaz Barak. For self-supervised learning, Rationality implies generalization, provably, ICLR 2021.

[2] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss, NeurIPS 2021.

[3] Zhiquan Tan, Yifan Zhang, Jingqin Yang, and Yang Yuan. Contrastive Learning Is Spectral Clustering On Similarity Graph, ICLR 2024.

[4] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. NeurIPS 35:25237–25250, 2022.

[5] Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In ICML, pages 9965–10040. PMLR,2022.

[6] Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer relu convolutional neural networks. In ICML, pages 17615–17659. PMLR, 2023.

Thanks for your detailed comments and suggestions. We provide our responses to your questions as follows.

---

Weaknesses:

W1: To my knowledge, there have been several works that study contrastive learning in theoretical aspects, although I could not find a discussion about them from the paper: e.g., [Bansal et al., 2021; HaoChen et al., 2021; Tan et al., 2024]. I think the paper should incorporate any discussions about such a line of research. For example, I am curious whether the paper’s observation about SimCLR as a matrix power method can be related to the spectral view of contrastive learning [HaoChen et al., 2021].

AW1: Thanks for pointing out this. We have revised the paper and added discussions about them. You can find the discussion in Section 2 (related work) in the revised paper. The following is the comparison between our paper and these papers.

[1] presented a new upper bound of the generalization gap of classifiers by first performing self-supervised training to learn representations, followed by fitting a simple classifier such as linear classifier to the labels. Although both [1] and our paper focus on the theoretical analysis of self-supervised learning, but the assumptions and the problem settings are different. [1] assumed the training satisfies certain natural noise-robustness and rationality, while our paper poses assumptions to the training setting and the signal-to-noise ratio (SNR). In the fine-tuning stage, [1] adopted a simple classifier such as linear classifier, while more complex two-layer CNN architecture is adopted in our paper.

[2] considered a spectral contrastive loss and performed spectral clustering on the population augmentation graph. We believe the setting in [2] is significantly different from our setting, and the spectral view discussed in [2] is not directly comparable to our study.

[3] extended the idea proposed by [2] to general loss functions by showing the equivalence between InfoNCE loss and spectral clustering. Furthermore, [3] also extended this to more general settings, including multi-modal scenarios. But as mentioned above, the idea of spectral clustering on population augmentation graphs in [2,3] is still different from our idea of signal learning analysis for contrastive learning.

In addition, we would like to point out that while [2,3] only focus on the analysis of contrastive learning (corresponding to the pre-training stage in our paper), our paper studies how contrastive learning influences the performance of the following fine-tuning stage.

Thank you for the response. I appreciate that the authors have incorporated a discussion about related works.

• The authors mentioned that Bansal et al. (2021) have also focused on the theoretical analysis of self-supervised learning. But if so, I think there should be a more in-depth discussions about the comparison of the conditions and how mild the current setup is and why it is significant. I don't think the difference between the use of linear vs. 2-layer CNN is a major point.

• Also, if there is work such as Bansal et al. (2021), I don't think the authors' claim that the work is the first analysis on SimCLR holds anymore, while the claim was used multiple times in the justifications throughout the response letters. I would also suggest the authors to further check if there is other theory papers that analyze contrastive learning beyond the list the reviewers' have suggested; I believe they will be just a partial subset of the full literature and I think these work should be adequately discussed in the manuscript.

• Although I agree that having simplifying conditions is important in theory works, but I am not fully convinced that the current assumptions are mild enough to produce practical significance in SimCLR analysis; even while the assumptions may be useful for supervised setups.

Overall, I found the response addresses some of my concerns; but I still feel lack of practical significance due to the unrealistic assumption of analysis, e.g., compared to Bansal et al. (2021). In this respect, I am keeping my original score of 5 this time.

---

Thanks for your detailed response and the follow-up questions. Here we provide our answers to these questions.

Q1: The authors mentioned that Bansal et al. (2021) have also focused on the theoretical analysis of self-supervised learning. But if so, I think there should be a more in-depth discussions about the comparison of the conditions and how mild the current setup is and why it is significant. I don't think the difference between the use of linear vs. 2-layer CNN is a major point.

A1: We would like to reemphasize that “[1] studies fine-tuning a simple (linear) model, while we consider fine-tuning a complicated two-layer CNN” is indeed a major and important difference between [1] and our work. In particular, the results in [1] are only meaningful when fine-tuning gives an “underfitting” model (training loss is strictly bounded away from zero). On the other hand, our work focuses on a setting where fine-tuning can give an “overfitting” model (training loss can be arbitrarily close to zero).

Please note that, in the literature of generalization theory, “underfitting” models and “overfitting” models require completely different techniques to study. The type of result in [1] is the more classic approach focusing on bounding the “generalization gap” by characterizing the “complexity” of the classifier. However, in deep learning, people frequently find that such classic complexity-based generalization bounds cannot provide meaningful results for complicated neural networks - the upper bound of test accuracy given by these techniques are usually larger than $100\\%$, making the result “vacuous" (this is also why [1] has to focus on fine-tuning simple linear classifiers). This motivates a lot of works to develop new methods for generalization bounds of neural networks, and [2], which studies “benign overfitting” of two-layer CNNs, is a representative work along this new line of research, where the key to obtaining tighter generalization bounds is to accurately analyze the model training process. Our work, aiming to establish comparable results as [2] for SimCLR, also follows this more advanced line of research.

For example, to see the difference between [1] and our work, please note that the generalization bound in [1] has the form $\sqrt{ C / n }$, where $C$ is the “complexity” of the simple classifier produced in the “simple fit” stage. When applied to the two-layer CNN considered in our work, this bound would be $\sqrt{ d / n }$. However, according to our assumptions, we have $d \gg n$, and this generalization bound $\sqrt{ d / n }$ is much larger than one, making the conclusion meaningless. In comparison, our work can give a tight generalization bound under the very mild assumption that $n = \tilde{\Omega}(1)$ (meaning that $n$ only needs to be poly-logarithmically large). This demonstrates the drastic difference between [1] and our work. However, we would like to clarify that such a drastic difference does not mean that [1] gives bad bounds, it is simply due to the fact that [1] and our work focuses on completely different settings, and trying to apply the result in [1] to our setting cannot give a fair comparison.

Below, we list some other differences between our work and [1]:

• [1] only studies the generalization gap. On the other hand, in our work, we do not only establish generalization bounds, but also demonstrate that “SimCLR pre-training + supervised fine-tuning” converges and the training loss function can be optimized arbitrarily close to zero.

• [1] does not need to specify any particular model architecture, but instead, relies on assumptions that two terms appearing in the generalization bound, the “robustness gap” term and the “rationality gap”, are small, which are assumptions that are not easy to theoretically verify. In comparison, our work is of a different flavor – we focus on a specific CNN architecture, and our bounds are more concrete without terms like the “robustness gap” or “rationality gap”.

Overall, we still believe that [1] and our work consider significantly different settings and our results are not directly comparable. We would also like to argue that in machine learning theory literature, it is very common that two theoretical works on the same topic are not directly comparable because they make completely different assumptions.

---

---

Q2: Also, if there is work such as Bansal et al. (2021), I don't think the authors' claim that the work is the first analysis on SimCLR holds anymore, while the claim was used multiple times in the justifications throughout the response letters. I would also suggest the authors to further check if there are other theory papers that analyze contrastive learning beyond the list the reviewers' have suggested; I believe they will be just a partial subset of the full literature and I think these works should be adequately discussed in the manuscript.

A2: Thanks for your suggestion. What we meant is only that along the line of works that study the “signal learning” of neural networks (such as [2]), our work is the first to analyze SimCLR. We believe that we never make such claims in our paper, and we will make sure to tune down our claim and provide clearer context in our future discussions.

We appreciate your reminder for us to give a more thorough literature review. We have also identified additional related works on theoretical analysis of self-supervised learning:

• [6] introduced a theoretical framework for contrastive learning under a latent classes assumption and obtained the generalization bound for contrastive learning. The established generalization bound is also a complexity-based bound similar to [1].

• [7] considered a certain class of reconstruction-based pretext tasks where reconstruction is achieved by linear regression, and analyzed sample complexity of such methods.

The deadline for revising the manuscript during the discussion period has passed. But we will make sure to include the references above in our camera-ready version of the paper,

---

---

Q3: Although I agree that having simplifying conditions is important in theory works, but I am not fully convinced that the current assumptions are mild enough to produce practical significance in SimCLR analysis; even while the assumptions may be useful for supervised setups.

A3: We would like to further address your concern regarding the gap between the two-layer CNNs considered in this paper and practical applications as follows.

• In [2], the authors study the direct supervised learning of two-layer CNNs. The goal of our paper is to establish comparable theoretical guarantees for SimCLR pre-training followed by supervised fine-tuning, allowing us to demonstrate how SimCLR enhances learning performance in comparison to [2]. Therefore, we believe that considering the same settings and assumptions as in [2] is a natural and appropriate choice that serves our purpose well.

• Please note that even for direct supervised learning, recent related works [3,4,5] usually have similar assumptions as our paper. Compared to previous works, our research examines the more complicated procedure of “SimCLR pre-training -> remove projection head -> attach classification layer -> supervised fine-tuning”, which is more challenging to study. Therefore, while we agree that there is a big gap between our theory and the practice, we must emphasize that this gap exists in most existing theoretical studies. We sincerely hope that our technical contribution in developing a framework for the theoretical analysis of SimCLR is recognized, as we believe our novel analysis tools may help advance theoretical studies towards more practical scenarios.

Although our paper makes a set of assumptions to enable precise theoretical analysis, we believe the insights obtained from our theoretical study are applicable to more general settings. As demonstrated in the additional experiments on ViTs in Appendix A in the latest revision of the paper, the phenomenon that SimCLR pre-training enhances signal learning during supervised fine-tuning is also observed. We believe our theoretical analysis represents an important step towards theoretically understanding the benefits of SimCLR pre-training.

---

Reference:

[1] Yamini Bansal, Gal Kaplun, and Boaz Barak. For self-supervised learning, Rationality implies generalization, provably, ICLR 2021.

[2] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. NeurIPS 35:25237–25250, 2022.

[3] Spencer Frei, Niladri S. Chatterji, and Peter L. Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. COLT, 2022.

[4] Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization. NeurIPS, 2023.

[5] Yiwen Kou, Zixiang Chen, Yuan Cao, and Quanquan Gu. How does semi-supervised learning with pseudo-labelers work? a case study. ICLR, 2023.

[6] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. ICML 2019.

[7] Jason D. Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. Predicting what you already know helps: Provable self-supervised learning, NeurIPS, 2021.

Thanks for your constructive review. We address your comments as follows.

---

Weaknesses:

W1: Unlike other works focused on similar data distributions and network architectures, such as Cao et al., 2022, and Kou et al., 2023, the authors provide only a sufficient condition on the amount of unlabeled data, whereas previous studies offer tight necessary and sufficient conditions. I agree that providing only a sufficient amount of pretraining data effectively demonstrates SimCLR’s advantage over supervised learning. However, I believe that characterizing tight necessary and sufficient conditions would be valuable, as it may facilitate comparisons with other pretraining methods in future work.

AW1: Thanks for raising the question about matching lower bounds. Please note that the condition on $n$ for our upper bound is already very mild. As shown in Condition 4.1 for Theorem 4.2, the condition for the number of the labeled data is $n\geq\Omega(\log(1/\delta))$. Since it is obvious that we need at least one labeled data to perform supervised learning, this condition $n\geq\Omega(\log(1/\delta))$ for successful learning already matches the natural requirement $n\geq 1$ up to a logarithmic factor. In this sense, we believe no specific analysis on lower bounds are necessary in our study.

In addition, we would like to point out that it is also quite common in the existing papers to compare only one side of the bound (only the upper bound or the lower bound) to obtain the conclusion, such as in [1,2]. [1] compared the lower bound of Adam and with the upper bound of SGD, and demonstrated their convergence to different global solutions with significantly different generalization errors. [2] theoretically analyzed the upper bound to show how knowledge distillation improves the performance of neural networks in the learning and compared the result with lower bounds for learning without knowledge distillation.

Therefore, by establishing an upper bound for SimCLR pre-training followed by supervised fine-tuning and comparing it with the lower bound of direct supervised learning, our paper follows the same nature as recent works [1,2]. Therefore, it may not be necessary to give both the upper and lower bounds in order to obtain the above conclusions.

---

W2: The data augmentation techniques introduced in the problem setting is too unrealistic. While I understand that defining data augmentation in an abstract manner within a signal-noise model is necessary, changing only the noise patch to new i.i.d. noise seems overly simplistic. This approach leads to augmented data from different pretraining samples with the same (unseen) label following the same distribution, which differs from practical scenarios. I believe a more practical definition for data augmentation could be used—such as applying a linear transformation to the noise patch.

AW2: We agree that we consider an ideal case of data augmentation. As a theoretical paper, it is inevitable that we need to consider a certain idealized setting, and considering such an ideal version of data augmentation helps us to establish precise guarantees of the SimCLR algorithm. Moreover, to the best of our knowledge, the influence of contrastive learning on supervised fine-tuning has not been analyzed before even for this ideal setting. Therefore, we believe our paper can serve as a first step towards more realistic theoretical studies.

---

W3&Q1: limited practical insights: Could you provide some practical insights into the theoretical findings? Specifically, can the main proof technique—interpreting SimCLR as a matrix power method and characterizing top eigenvectors—be translated to real scenarios to offer intuition on why SimCLR is effective?

AW3&AQ1: Our theoretical analysis provides a signal learning framework. We believe that the insight behind this signal learning framework can help us understand many practical learning tasks as well.

The idea of the signal learning framework is that, when training complicated neural networks on a finite training dataset, there may exist many (probably infinitely many) different parameter configurations that can all lead to $100\\%$ training accuracy. However, neural networks with different parameter configurations may perform drastically differently on test data. We can generally treat the “information in the data input that are related to the label” (characterized by the vector $\boldsymbol{\mu}$ in our theoretical study) as the “signal” in the data, and treat the “information in the data input that are not related to the label” (characterized by the Gaussian noise vector $\boldsymbol{\xi}$ in our theoretical study) as “noises”. Then clearly, The signal is the label-dependent and generalizable part of the information, and noise is the label-independent and ungeneralizable part of the information, and whether the obtained neural network after training can achieve good prediction depends on whether it utilizes the “signal part” or the “noise part” to fit the training data, and this is exactly how we build our theory.

Our analysis of SimCLR demonstrates that, although SimCLR does not utilize any labeled data, the knowledge that “data augmentation should not change the label” suffices to help the neural network to capture more signals in the pre-training stage. We believe this characterization is true for more practical scenarios as well. Although it is difficult to verify on real data because “signals” and “noises” and the level they are captured by a more complicated network is hard to quantify, we have provided additional real-data experiments to justify our claim (to a certain extent) in the revision. The added experiments can be found in Appendix A in the revised paper.

---

Q2: Can the same proof technique be extended to ReLU networks and the 0-1 test loss considered in Kou et al.2023?

AQ2: In this paper, the key focus is the analysis of the SimCLR pre-training stage. The results of the pre-training stage serve as the initialization of the fine-tuning stage, and therefore we could analyze how the pre-training stage influences the performance of the fine-tuning stage.

We believe that it is a clear and concise way to illustrate the advantage of SimCLR pre-training on the fine-tuning stage in our setting by comparing it with direct supervised learning on two-layer CNN in [3].

Analyzing how the pre-training influences the fine-tuning on ReLU networks is also an interesting future direction. Similar sets of analysis could be applied to the fine-tuning on ReLU networks, but due to the different training dynamics, the specific techniques and tools may be different.

---

Reference:

[1] Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization. ICLR, 2023.

[2] Zeyuan Allen-Zhu, Yuanzhi Li. Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. ICLR 2023.

[3] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. NeurIPS 35:25237–25250, 2022.

Thank you for your response, which addresses most of my concerns. I raised my score to 6.

However, I would like to clarify my first question. Specifically, can your proof technique using the matrix power method be interpreted as providing insight into the underlying mechanism that explains why SimCLR effectively learns meaningful signals in real-world scenarios? For example, recent works on feature learning, such as [1], have used their proof techniques to offer high-level intuition about the mechanisms behind methods like Cutout and CutMix.

[1] Junsoo Oh and Chulhee Yun. Provable Benefit of Cutout and CutMix for Feature Learning. NeurIPS, 2024.