Understanding Self-supervised Learning as an Approximation of Supervised Learning

Dear Reviewer tUzQ,

We sincerely appreciate the valuable time and effort you have dedicated to reviewing our manuscript. We address each of your questions and concerns individually below. Please let us know if there are any comments or concerns we have not adequately addressed.

---

[W1 & Q2] it seems to me that there is significant overlap with [a]

In Section 2, we address [a]. [a] and our paper differ in terms of approach, objective, and theoretical setting:

In terms of approach, [a] builds on an established contrastive loss (Equation (1) in [a]) that attracts or repels other 'samples'. In contrast, our paper starts from scratch with a formulated supervised objective (Equation (5) in our paper) that attracts or repels 'pseudo-labels' (prototype representations).

In terms of objective, [a] explores how attracting and repelling other samples relates to alignment and uniformity. In contrast, our paper demonstrates how attracting and repelling pseudo-labels can manifest as attracting and repelling other samples. Thus, while [a] focuses on the properties of contrastive loss, we focus on the foundation of contrastive loss as an approximation of supervised learning.

Additionally, in terms of theoretical setting, [a] primarily examines the limiting behavior in an asymptotic setting (Theorem 1 in [a]) where the number of negative samples approaches infinity. On the other hand, our approach addresses a more general setting (Theorem 1 and 2 in our paper), offering broader applicability.

---

[W2 & Q1] What is the relationship between these two upper bounds and the supervised learning paradigm?

The loss in self-supervised learning is shown to serve as an upper bound for a supervised learning objective. Specifically, starting from the problem we formulated and assuming common practices in self-supervised learning, interestingly, a generalized form of the InfoNCE loss—commonly used in self-supervised learning—emerges as an upper bound. This provides a perspective that views self-supervised learning as an approximation of supervised learning. It offers insights into the type of optimization problem that self-supervised learning is solving. The concepts of prototype representation bias and balanced contrastive loss that arise in this process can be valuable for understanding and enhancing self-supervised learning.

---

[W3] the proposed new SSL format does not seem to have significant advantages

The accuracy of the standard NT-Xent loss (a special case of the generalized NT-Xent loss with $\lambda=1$) is 65.98%, while the accuracy of the balanced contrastive loss is 67.40%, showing a gap of 1.42%. Considering that the chance-level accuracy for ImageNet, which consists of 1,000 classes, is merely 0.1%, achieving this level of improvement solely through proper balancing is significant.

The purpose of the experiments in this section is to check the validity of the theory. To this end, we focus on examining whether a more balanced contrastive loss, as predicted by the theory, effectively enhances performance. To ensure thoroughness, we conduct a comprehensive set of experiments across a parameter grid.

---

[W4] there are many related works (like [b-c]) on the understanding of SSL

Given the extensive body of work related to self-supervised learning, we had to selectively discuss relevant studies. As mentioned in Section 2, our work falls into the category of contrastive learning. However, [b] addresses non-contrastive learning. Therefore, non-contrastive learning is discussed in Section 5.1, focusing on major algorithms with references. [c] proposes a practical method that leverages video datasets to learn invariance when the dataset is not object-centric. However, we aimed to keep the setting streamlined to avoid complicating the theoretical analysis. In the revised manuscript, we have included the papers in appropriate sections.

Dear Reviewer Cr2E,

We deeply appreciate your time and effort to review our manuscript. We address each of your questions and concerns individually below. Please let us know if there are any comments or concerns we have not adequately addressed.

---

[W1] this choice may vary substantially across datasets and problem domains

We agree that the choice of data augmentation should vary depending on the application. The fundamental spirit of self-supervised learning lies in leveraging domain knowledge about the application when labels are not available. Specifically, it involves utilizing knowledge about what kinds of transformation invariance the desired representation should possess for a given application. Our work focuses on grounding contrastive losses under the assumption that such data augmentation is already provided. In reality, controlling prototype representations using only domain knowledge is challenging. Exploring which data augmentation techniques should be tailored to each application will be an interesting future direction.

---

[W2] it provides limited insights into how these parameters impact performance across diverse datasets

Our work focuses on theoretical understanding, and the experiments were designed to align with this purpose. The purpose of the experiments is to check the validity of our theory by demonstrating the potential performance improvements predicted by the theory through the derived balanced contrastive loss. Therefore, we selected canonical datasets such as ImageNet and CIFAR-10 and focused on providing complete results across a grid of parameters for alpha and lambda.

---

[W3] there are alternative frameworks in self-supervised learning

Those algorithms are not theoretical frameworks, so their ideas are expressed intuitively, but they can be discussed as follows: Comparing self-supervised learning to bootstrapping stems from the process of constructing pseudo-labels using only representations without external input. This aspect can be connected to our framework, where surrogate prototype representations are generated using representations and treated as pseudo-labels. The predictor used here can be viewed as an additional module designed to predict these pseudo-labels. This algorithm falls under non-contrastive learning, which is discussed in Section 5.1.

Additionally, in clustering algorithms, the cluster assignments of transformed images are made consistent. This can be interpreted within our framework as guiding the representations of transformed images to converge toward a single prototype representation, thereby assigning them to the same cluster. Following the suggestion, we added a discussion section to the paper to address this topic. Thank you very much for your suggestion.

---

[W4] The paper’s findings, especially around balancing attraction and repulsion forces, suggest potential for optimization.

For example, one could consider an algorithm that adjusts alpha and lambda dynamically rather than treating them as fixed hyperparameters. Our framework enhances the understanding of the roles of alpha and lambda: alpha reflects hedging the risk with multiple negative samples, while lambda adjusts the relative magnitudes of the attracting and repelling forces. By leveraging this intuition, dynamically adjusting alpha and lambda based on the overall distribution of representations could potentially improve the learning process.

---

[Q1] How would the assumptions, such as balanced data and the specific choice of cosine similarity, affect the generalizability of this framework to domains

In proving our theorems, we rely on certain assumptions that are common practices in self-supervised learning, such as balanced datasets and cosine similarity. As a result, generalization becomes less straightforward in scenarios where these assumptions are violated. However, by understanding the role these assumptions play at different stages of the proof, we may pave the way for the development of more generalized algorithms in the future.

---

[Q2] To what extent can prototype representation bias be quantitatively minimized through practical data augmentation strategies?

According to our framework, the following idea can be considered: data augmentation methods that merely apply color distortions or Gaussian blur may struggle to adequately cover images with the same label in the augmented image space (Figure 2 in our paper). Data augmentation leveraging generative AI may offer an alternative.

---

[Q3] Could the theoretical framework be adapted or extended to include asymmetrical architectures, given their prominence in modern self-supervised learning algorithms?

We discuss the asymmetric architecture in Section 5.1.

Dear Reviewer JRCA,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript. We address each of your questions and concerns individually below. Please let us know if there are any comments or concerns we have not adequately addressed.

---

[W1] the connection has been addressed in previous literature [1], for example attraction/repelling and normalization has been brought up in [2].

Since papers on contrastive learning inherently include attracting positive pairs and repelling negative pairs, they may seemingly appear similar at first glance. We clarify the differences below: [1] is basically a paper on unsupervised learning, which is different from self-supervised learning. Both are in a situation where the label is unknown, but unlike unsupervised learning, self-supervised learning has an additional aspect of generating pseudo-labels from data. Therefore, in our paper, we formulate the problem using pseudo-labels (prototype representations) generated from data. In addition, the contrastive loss discussed in [1] is a variation of the triplet loss (Definition 2.3 in [1]) that operates on three samples without hard negative mining. The loss we address is a loss between a sample and prototype representations that incorporates hard negative mining. We demonstrate how this relates to an InfoNCE-type loss. Therefore, while [1] is about “classical contrastive loss in the context of unsupervised learning,” our paper is about “InfoNCE-type loss in the context of self-supervised learning.”

[2] explores how alignment and uniformity relate to contrastive loss (sample attraction/repulsion). In contrast, we demonstrate how our formulated supervised objective (pseudo-label attraction/repulsion) translates into contrastive loss. Thus, while [a] focuses on the properties of contrastive loss, we focus on the foundation of contrastive loss as an approximation of supervised learning. Through this process, common practices (like normalization) are unified within a single theoretical framework. Additionally, [a] focuses on an asymptotic setting (Theorem 1 in [a]) where the number of negative samples approaches infinity, whereas we address a more general setting (Theorems 1 and 2 in our paper), ensuring broader applicability.

---

[W2] the framework seem to conflict with the use of projection head

Extracting features before the projector boosts performance, but it does not mean that self-supervised learning algorithms fail to work when features are extracted after the projector. Therefore, the improvement in performance from features before the projector is not in conflict with the proposed framework but rather can be interpreted within it.

We interpret this as follows: During the process where contrastive loss directly manipulates features after the projector, the features before the projector are indirectly pushed closer or farther apart, encouraging the learning of more generalized representations. This process leads to the acquisition of noise-reduced and more robust high-level features. It can also be seen as a form of regularization leveraging the information bottleneck effect.

---

[W3] aggressive augmentation negatively impacts supervised learning [3]

The supervised setting in [3] involves training with a cross-entropy loss (Subsection B.8.1 in [3]), which is conceptually different from our supervised setting (Equation (2) in our paper) as the loss. Thus, it is challenging to directly apply their results to our interpretation.

However, we can discuss the following: Theoretically, if we know that the target representation must be invariant to certain transformations, we can enforce transformation invariance by aligning the representations of transformed data. We assume knowledge of such transformations to develop the theory.

In practice, however, identifying these transformations is challenging, leading to some reliance on domain knowledge. This domain knowledge is not perfect, and since data augmentation inherently modifies the data, it has the potential to introduce negative effects.

For instance, from a human perspective, color distortion does not alter the semantic meaning of a dog in a photo, so the representation should ideally be invariant to color distortion. However, from the model's perspective, some level of color information might contribute to identifying the object as a dog.

In a supervised setting, where access to the label is already available, excessive data augmentation might be unnecessary. Conversely, in self-supervised learning, where robust pseudo-labels must be constructed solely from transformed data, more aggressive data augmentation is often required.

Dear Reviewer JRCA,

Thank you for your thoughtful feedback and for taking the time to reconsider your evaluation during the discussion period. We deeply appreciate your engagement.

We would like to address your concerns regarding the significance of the consequences derived from our proposed connection to supervised learning.

1. Theoretical contribution Self-supervised learning implies the idea of constructing pseudo-labels from samples. However, when we see self-supervised learning losses composed solely of samples, it is not immediately apparent how they relate to pseudo-labels. In this work, what we mathematically showed is that pseudo-label attraction/repulsion (Equation (7)), i.e.,

$$-s\\left(f\_{\\theta}(t(x)), \\hat{\\mu}\_{y}\\right) + \\lambda \\max\_{y' \\neq y} s\\left(f\_{\\theta}(t(x)), \\hat{\\mu}\_{y'}\\right),$$

where $\\hat{\\mu}\_{y} := \\mathbb{E}\_{T}f\_{\\theta}(T(x))$ and $\\hat{\\mu}\_{y'} := \\mathbb{E}\_{T', X' \\vert y'}f\_{\\theta}(T'(X'))$, can be optimized as sample attraction/repulsion (Equation (13)), i.e.,

$$-\\log\\frac{\\exp(\\alpha s\\left(f\_{\\theta}(t(x)), f\_{\\theta}(t'(x))\\right))}{\\left( \\sum\_{x' \\in \\hat{\mathcal{X}}} \\exp(\\alpha s(f\_{\\theta}(t(x)), f\_{\\theta}(t'(x')))) \\right)^{\\lambda / \\nu}},$$

which generalizes the widely-used InfoNCE-type losses. This connection contributes to the firm foundation of self-supervised learning by addressing a crucial gap in the literature. To clarify this, we have updated the contents of Section 3.2, temporarily highlighted in "blue". We believe that technology built on a shaky foundation can be difficult to trust and may encounter limitations in long-term development.

2. Practical contribution: While it is true that our submission builds upon some established practices, our contribution lies in unifying those practices into a cohesive framework. In addition, to be considered a good theory, we show not only internal consistency but also a reasonable degree of predictive power. In this regard, we provide evidence that leveraging the prototype representation bias and the balanced contrastive loss emerging from our theoretical development can lead to performance improvements. This can guide practitioners toward lines of research on how to reduce the bias in prototype representations or how to effectively balance contrastive losses.

In summary, we believe that our work will benefit the self-supervised learning community by serving as a basis and providing guidance for research.

We hope this clarifies the broader importance of our work. We value your constructive feedback, which encourages us to continue refining and communicating the implications of our findings effectively. Thank you once again for your time and consideration.

Sincerely,
Authors.

Dear Reviewer NNEK,

We sincerely appreciate the time you have taken to review our manuscript. We address your concern below. Please let us know if there are any comments or concerns we have not adequately addressed.

---

[W1] Using the prototype representation bias, as defined by the authors, to represent the gap between SSL and SL is overly simplistic. In reality, no practical augmentation can achieve a very low prototype representation bias unless label information is available.

The purpose of this paper is to provide a theoretical understanding rather than focusing on practical applications. In this context, the concept of prototype representation bias is introduced to better understand the connection between supervised and self-supervised learning. In reality, there are bound to be some limitations in controlling prototype representation bias through data augmentation. This is an inherent limitation of self-supervised learning itself, which must utilize data augmentation derived from domain knowledge because there are no labels. However, assessing the practicality of a newly proposed concept is not straightforward. Ideas inspired by the concept of prototype representation bias may lead to new algorithms in the future.

Additionally, in this framework, all other parts except the part where $\\mathbb{E}\_{T, X | y} f_{\\theta}(T(X))$ is approximated by $\\mathbb{E}\_{T}f_{\\theta}(T(x))$ (with available images) are proven rigorously (Note that in the repelling component, even this approximation is not used). In the attracting component, the approximation relies on 1) the intuition that it is natural and 2) the tendency shown in the experiment. With the supervised objective defined this way, the math works seamlessly, and the InfoNCE loss, widely used in self-supervised learning, naturally emerges. Bridging the gap between supervised and self-supervised learning in a principled way is a necessary piece that is missing in the literature. In machine learning, we believe that it is a long-standing tradition to value a theory when it is reasonably solid and its limitations are properly acknowledged.