Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning

Note: We thank the reviewer for their very thoughtful feedback and comments. Following the reviewers’ suggestions, we conducted multiple experiments to strengthen our work. Since there is no dedicated common space for experiments, we present them at the beginning of the responses to reviewers 8fbK (part 1) and J3ay (part 2).

Summary of experiments (part 2)

Experiment (5) We reproduced the experiment in Fig. 5 with the ViT architecture on CIFAR10, CIFAR100, mini-ImageNet and Tiny-ImageNet.

CIFAR10

CIFAR100

Mini-imagenet

Tiny-Imagenet

Experiment (6) We reproduced the experiment in Fig. 2 (top) with a ResNet-50 trained with the CL loss instead of the DCL loss:

CIFAR10

Mini-imagenet (C=100)

Experiment (7) We trained pairs of CL or DCL and NSCL-trained models starting from the same initialization, trained using the same batches at each iteration. In order to evaluate the similarity/alignment between the pairs of models we used the Centered Kernel Alignment (CKA) (https://proceedings.mlr.press/v97/kornblith19a/kornblith19a.pdf) and Representation Similarity Analysis (RSA) (https://www.frontiersin.org/journals/systems-neuroscience/articles/10.3389/neuro.06.004.2008/full) metrics (0 = no alignment, 1 = perfect alignment). As can be seen, these paired models achieved very high CKA and RSA values, demonstrating that the learned representations are strongly aligned across the two training objectives.

Experiment (8) We verified augmentation collapse by training DCL ViT models and measuring cosine similarity (0 = no collapse, 1 = perfect collapse). We measured the averaged cosine similarity between the embeddings of augmentations of the same sample and the cosine similarity between augmentations of different samples (over the full dataset and 5 augmentation for each pair of samples). Same‑sample augmentations averaged ~0.90, while different‑sample augmentations averaged < 0.15, confirming highly aligned embeddings of augmentations of the same sample.

Response to J3ay

Reviewer: The closseness of the two loss values (for DCL and NSCL) does not directly imply the same behavior and structure of the learned representation. I am grateful that this is also commented on by the authors.

You are absolutely right: a small difference in objective values does not by itself guarantee identical embedding geometry. The loss‑gap bound is only the first step. To better understand representation‑level similarity we: (i) Introduce directional CDNV (Sec. 3.2): a geometry‑aware variance measure that both DCL and NSCL implicitly drive to low values during training and (ii) link directional CDNV to downstream accuracy (Prop. 1, Sec. 3.2).

To further substantiate the alignment between CL/DCL‑ and NSCL‑trained models, we ran dedicated experiments quantifying representation similarity. We trained matched CL/DCL and NSCL models from the same initialization with identical batches at each iteration and measured alignment using Centered Kernel Alignment (CKA) and Representational Similarity Analysis (RSA)—both standard embedding‑similarity metrics (ranging between 0–1). As shown in experiment (7) (in the response to J3ay), the paired models achieve very high CKA and RSA values, indicating that their learned representations are strongly aligned.

Reviewer: (minor) Sometimes, the generally-used variables (e.g., K on L220) are introduced earlier in the text and it can be a bit confusing.

Thanks for pointing this out. We will revisit these widely used variables later in the text as a reminder.

Reviewer: Experiments on a larger dataset (e.g., ImageNet) are missing.

Thank you for the suggestion. To scale-up our experiments, we conducted plenty of experiments with the ViT architecture, as well as the ImageNet-1k (1000 classes) and Tiny-ImageNet (200 classes) datasets. The experiments reproduce Fig. 2, 4 and 5 (see experiments (1)–(5) in the response to J3ay).

Reviewer: It is not explained what "upper bound" in Fig 2 (top) stands for?

The "upper bound" in Figure 2 refers to $_{\text{NSCL}} + \log(1 + e^2/(C − 1))$, as noted in the caption. We will clarify this further in the next version of the paper.

Reviewer: Claims on L298-299 are not discussed. What can be the reason for the gap between the losses to increase?

We can express the loss difference as

where

• $A = \sum_{j \neq i, y_j = y_i} e^{\text{sim}(z_i, z_j)}$ is the sum over same‑class negatives, and
• $B = \sum_{y_j \neq y_i} e^{\text{sim}(z_i, z_j)}$ is the sum over all cross‑class negatives.

Early in training, similarities between all samples (same or different class) are roughly equal, so $A/B$ is small—dominated by the fact that there are far fewer same‑class terms than cross‑class ones. As a result, the loss gap is negligible.

As training progresses, same‑class samples become more similar while cross‑class similarities decrease. This increases $A$ relative to $B$, so the gap $\log(1 + A/B)$ grows slightly. The increase remains small because the same‑class terms are still a small fraction of the total. This explains the gradual upward trend observed in Fig. 3.

The gap is still small because the same‑class negatives make up only a tiny fraction of all negatives, and the label‑agnostic bound keeps the difference tightly controlled.

Reviewer: How do you define the "same point" in the augmentation collapse on L66? Is it some small neighborhood?

By "same point," we mean that for two augmentations $x'$ and $x''$ of the same sample, the embeddings satisfy $f(x')=f(x'')$; i.e., they are exactly equal. This definition arises from our analysis of global minimizers under the unconstrained features model, which is an idealized abstraction of the learning setting.

In practice, exact equality is not achievable. Instead, augmentations of the same sample tend to produce very similar embeddings. Empirically, we observe high cosine similarity between such pairs: in experiment (8) (in the response to J3ay) the average cosine similarity between augmentations of the same sample was ~0.9, compared to <0.15 for augmentations of different samples (averaged over 5 augmentations for each pair of samples).

Note: We thank the reviewer for their very thoughtful feedback and comments. Following the reviewers’ suggestions, we conducted multiple experiments to strengthen our work. Since there is no dedicated common space for experiments, we present them at the beginning of the responses to reviewers 8fbK (part 1) and J3ay (part 2).

Response to 8nSm

Reviewer: There is a mismatch between the main theorem and its proof. Theorem 1 states that the bound is for DCL, but according to its proof, the bound should be for CL. Also, the claims in Sections 3 and 4 are for DCL, whereas in the abstract, the gap is claimed to be between standard CL and NSCL losses.

1. The proof actually covers both objectives; we stated Thm. 1 in terms of DCL only to keep notation light. The proof in fact covers both objectives, as shown by the chain of inequalities (that appear in the proof):

which yields

Thus the same bound holds for both CL and DCL.

2. We focused on DCL in the main text since it is an improved variant of the original CL objective (as reported in the original DCL paper). The abstract mentions standard CL because that term is more familiar and one can think of CL as a family of algorithms rather than necessarily a very specific implementation; we will revise it to read "(decoupled) contrastive loss" and add a sentence clarifying that Thm. 1 applies to both CL and DCL. We will also update the theorem statement and proof header to make this dual coverage more explicit.

3. Empirically, we conducted an experiment to reproduce Fig. 2 (top) for CL-training, showing the same behaviour as before (see experiment (6) in the response to J3ay). In addition, we evaluated CL, DCL and NSCL and the bound for DCL-trained models and observed negligible differences in the values of CL and DCL (see experiments (1) and (2) in the response to 8fbK).

Reviewer: Several related theoretical works are not mentioned, e.g., https://openreview.net/forum?id=XDJwuEYHhme, https://openreview.net/forum?id=c5K7gcd2Rw, https://openreview.net/forum?id=cN9GkuBo4w. The authors should discuss the connections and differences compared with these papers.

We appreciate the reviewer highlighting these additional works. In Sec. 2 (lines 80–91) we already review 13 related SSL theory papers, but we are happy to add the suggested works and clarify their differences from ours.

Both papers derive bounds on supervised terms (classification error in the first, supervised loss in the second), but just like the works mentioned in lines 80-91, their analyses rely on very strong assumptions that we avoid:

(1) Thms. 1 and 6 in the first paper rely on Definition 1, the $(\varepsilon,\delta)$-augmentation condition. It effectively assumes that, for almost every pair of images in the same semantic class, there exists a sequence of the model’s augmentations that brings their augmented versions arbitrarily close. For a class such as “cat images”, that would mean crops, colour‑jitter, blurs, etc’, can morph one cat image into another, a claim that is clearly idealized and rarely satisfied in practice.

(2) The bounds in the second paper (Thms. 2.1, 2.3, 2.4) depend on terms such as:

and

which together essentially require embeddings of augmented samples from the same class to be close. While the second term can be somewhat controlled through optimization, the first term is far more restrictive: (i) it implicitly depends on class labels (which are unavailable to SSL algorithms) and (ii) it presupposes intra-class consistency across augmentations, a property that may not hold in practice. In effect, the analysis builds in the very property it seeks to prove (low intra-class variation in embeddings) in order to derive a bound on the supervised loss.

Reviewer: The concept of directional CDNV needs intuitive explanations. Some examples could help.

Thank you for the suggestion. In order to illustrate the CDNV and directional CDNV (and to understand their difference), we created two plots for each one of those. The first plot includes two classes in 2‑D. We illustrate how CDNV measures the total spread of each class relative to the distance between their means—visualised as the radius of two equal‑radius "balls" around the class centres. For the directional CDNV, we keep the same denominator (inter‑centre distance) but now measure the spread only along the line that joins the two centres. In the figure this is visualized as the variance of the 1‑D projections of the samples onto that connecting line.

This visual makes it clear why directional CDNV can be much smaller than CDNV—e.g., if the clouds are elongated orthogonally to the centre‑to‑centre axis, yet still dominates few‑shot error (Prop. 1). We will include these plots with an explanation in the final version.

Reviewer: The theoretical claims show that DCL approximates NSCL when the number of classes is large, and that NSCL leads to perfectly clustered representations and accordingly good few-shot classification performance. However, the theoretical discussions fail to address why DCL leads to good representations. Specifically, do the theoretical claims mean that when the number of classes $C \to \infty$, the DCL representations are perfectly clustered? This doesn't seem to be right under my knowledge.

To clarify, we do not argue that DCL leads to perfectly clustered representations as $C \to \infty$. When $C \to \infty$ then we are able to perfectly minimize the NSCL loss (which leads to perfect clustering) only if the embedding dimension $d \to \infty$. Thus, in any realistic setting where $d$ is finite we will always have a noticeable gap between the two losses. Instead, we show that the DCL and NSCL losses are close which does not directly lead to the same level of clustering in the representations. In fact, similar to the proof of Thm. 2, it is possible to show that the global minimizer of DCL (which is only possible when the embedding dimension $d$ exceeds the number of samples $N$), the embeddings of training samples form an equiangular tight frame (ETF) of $N$ vectors. In that configuration, the directional‑CDNV is exactly zero (and the CDNV to be proportional to the number of samples per class), which is a weaker notion of clustering than the one given by minimizing the NSCL loss.

Reviewer: The experimental validation of diminishing loss gap is weak (Fig. 2). It only verifies $C=10$ and $C=100$. More results on Tinyimagenet (C=200) or Imagenet1k (C=1000) could further strengthen this claim.

Following the reviews, we conducted additional experiments on both ImageNet-1k and TinyImageNet, as well as further experiments with the ViT architecture (see experiments (1)-(5) in the responses to 8fbK and J3ay).

Reviewer: Figure 3 shows that the gap slightly increases over training. Is there any explanation?

We can express the loss difference as

where

• $A = \sum_{j \neq i, y_j = y_i} e^{\text{sim}(z_i, z_j)}$ is the sum over same‑class negatives, and
• $B = \sum_{y_j \neq y_i} e^{\text{sim}(z_i, z_j)}$ is the sum over all cross‑class negatives. Early in training, similarities between all samples (same or different class) are roughly equal, so $A/B$ is small—dominated by the fact that there are far fewer same‑class terms than cross‑class ones. As a result, the loss gap is negligible.

As training progresses, same‑class samples become more similar while cross‑class similarities decrease. This increases $A$ relative to $B$, so the gap $\log(1 + A/B)$ grows slightly. The increase remains small because the same‑class terms are still a small fraction of the total. This explains the gradual upward trend observed in Fig. 3.

Reviewer: In Figure 5, the CDNV of DCL increases at the end of training. Is there any explanation?

Thanks for bringing this up. Throughout the experiments we have consistently observed a phenomenon we refer to as "over‑training" in CL/DCL. At the first stage of training, the directional‑CDNV decays rapidly while the CDNV remains fairly stable (and may even decrease slightly). As training progresses, because the model continues to push samples apart (while still enforcing consistency within augmented views), the CDNV eventually starts to grow again, while the directional‑CDNV continues to decay (or saturates).

Similar to the proof of Thm. 2, it is possible to show that if the embedding dimension $d$ exceeds the number of samples $N$ (an unrealistic but illustrative case), the global minimizer of CL/DCL organizes the embeddings of training samples as an equiangular tight frame (ETF) of $N$ vectors. In that configuration, the directional‑CDNV is exactly zero but the CDNV is proportional to the number of training samples per class.

We will add discussion of this behavior and its origins in the next version of the paper.

Note: We thank the reviewer for their very thoughtful feedback and comments. Following the reviewers’ suggestions, we conducted multiple experiments to strengthen our work. Since there is no dedicated common space for experiments, we present them at the beginning of the responses to reviewers 8fbK (part 1) and J3ay (part 2).

Response to WmN5

Reviewer: The experiment conducted by the author is quite small in scale, as CIFAR-10, CIFAR-100, and mini-ImageNet each contain only 50,000 images. The results could be easily influenced by overfitting, especially in the presence of noisy samples. I am curious whether this phenomenon still holds when the number of samples per class increases or when the data becomes noisier. Nowadays, pretraining typically requires massive and often noisy datasets. Further experiments should be conducted to verify the author’s claims.

Thanks for raising this point. We believe that the reported results are not due to overfitting:

1. We conducted a wide range of experiments across multiple datasets, and as shown in Tab. 1 and Fig.  4, DCL‑trained models transfer well to downstream tasks with simple adaptation methods (e.g., LP, NCCC).

2. Beyond CIFAR and mini‑ImageNet, we also used SVHN, which contains 600 k images, and observed the same trends (Fig. 1).

3. Following the reviewer’s suggestion to scale up, we added experiments on Tiny‑ImageNet (200 classes) with ViT backbone (see experiment (2) in the response to 8fbK) and ImageNet‑1k (1000 classes) with ResNet50 backbone (see experiment (1) in the response to 8fbK). These datasets not only have a significantly larger number of classes but also naturally contain more label noise and sample‑level ambiguity than CIFAR‑like datasets.

We would also be happy to explore specific experiments with noisier data if the reviewer has a concrete suggestion.

Reviewer: The central claim that standard contrastive learning approximates a supervised variant under a growing number of classes is conceptually intuitive and has been suggested in prior works. While Theorem 1 formalizes this intuition with a clean bound, the assumption that the number of classes tends to infinity makes the result less surprising and arguably less practical. The paper ultimately confirms that contrastive objectives yield clustering behavior similar to supervised objectives, but this is already an implicit assumption in much of the literature. Thus, the theoretical contribution, though correct, feels incremental rather than insightful.

We appreciate the reviewer’s perspective and the opportunity to clarify the novelty and practical relevance of our results.

1. Incremental contribution: We agree that it is intuitive to view self-supervised learning (SSL) and supervised learning (SL) as closely related, and we acknowledge that the literature often assumes clustering properties. However, these are very strong and largely unsubstantiated assumptions. Our work provides concrete theoretical guarantees that explicitly establish the relationship between SSL and SL and explain not only why clustering arises, but also what kind of clustering is expected.

Thm. 1 provides a label‑agnostic bound that directly relates CL and SL. It informs us what kind of supervised loss is most related to CL (e.g., NSCL), it holds for all possible labelings simultaneously and does not rely on data‑distributional, augmentation, or architectural assumptions, is substantially more general than previous heuristic arguments (see lines 80–91) that depend on strong, often unrealistic conditions. In response to reviewer feedback, we further substantiated this result by showing in (experiment (7) in the response to J3ay) that the learned representations from DCL and NSCL are highly aligned.

Prop. 1 and its associated experiments make an equally important contribution: they (i) characterize the specific clustering properties that hold in practice for CL‑trained models, and (ii) demonstrate why these properties benefit downstream tasks. Since clustering is often assumed rather than analyzed in the literature, these results fill a crucial gap in our understanding of SSL.

2. Beyond the $C \to \infty$ regime: While the gap in Thm. 1 tightens as the number of classes increases, the theorem does not require $C \to \infty$. As shown in Figs. 2-3, the bound is already tight and correlates strongly with the observed loss gap for moderate class counts (e.g., 10–100 classes in CIFAR‑10/100 and mini‑ImageNet).

In summary, our contributions (i) formalize a widely cited but previously speculated intuition, (ii) validate it in realistic settings, and (iii) link it directly to downstream performance. We therefore believe the work goes well beyond incremental progress, offering genuinely new theoretical and empirical insights into why SSL methods succeed.

Reviewer: This paper does not provide practical guidance for designing CL systems. For example, it does not address important questions such as how augmentation affects learning, the robustness of the theoretical results under incomplete augmentation, or whether CL can outperform supervised learning in all cases. Although the authors introduce an upper bound on the few-shot error, it relies on quantities that are difficult to explicitly control during learning (e.g., class-wise variance) and has limited practical impact on accuracy. Moreover, minimizing loss does not directly translate into subsequent accuracy improvements, weakening the practical relevance of the results.

We thank the reviewer for raising important questions about the practical implications of our work. While our primary focus is not on proposing new practical training techniques, our work provides a foundational framework for explaining why existing self-supervised methods, such as CL and DCL, are theoretically sound, what aspects make them so successful, and what kinds of representations they capture. While these insights may not directly tell practitioners how to choose training hyperparameters, they offer a more informed way to analyze and ultimately improve CL systems in practice.

• Relationship with supervised learning: There is a growing intuition that self-supervised learning and supervised learning are closely related, but it is only through the bound in Theorem 1 and the discussion in lines 155–161 that we identify the mechanism behind this relationship. Our bound quantifies the gap between SSL and supervised learning and clarifies which supervised method (e.g., NSCL) is most closely related to CL/DCL training. In fact, our new empirical results (experiment (7) in the response to J3ay) verify that by showing a very strong alignment between the learned embeddings of CL/DCL and NSCL.

• Practical impact of the few-shot error bound: The reviewer is correct that our bound depends on quantities not directly controlled during training and that it is not a direct estimator of accuracy. However, we argue this is a feature, not a limitation, because its purpose is to explain rather than serve as a new training objective.

A useful analogy is the bias–variance tradeoff. The formula decomposing error into bias, variance, and irreducible error is one of the most important concepts for practitioners. Although one cannot directly control bias or variance during a single training run, the decomposition provides a powerful framework for diagnosing failures (e.g., high variance implies overfitting). This understanding then guides practical decisions such as increasing regularization or gathering more data.

Similarly, our bound in Prop. 1 is an analytical tool for understanding contrastive learning. Its primary value lies not in the final number it produces but in how it decomposes error and introduces new vocabulary for analyzing performance. Specifically, it distinguishes between two different quantities: within-class variance (CDNV) and the variance along the directions separating class centers (directional CDNV). This analytical lens leads to a core finding presented in Fig. 5: successful CL training should significantly reduce directional CDNV while maintaining a relatively stable CDNV. This is a non-obvious insight that explains how label-agnostic methods learn linearly separable features.

Dear reviewer WmN5,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

This is critical to arrive at an informed decision.

Best, AC

Note: We thank the reviewer for their very thoughtful feedback and comments. Following the reviewers’ suggestions, we conducted multiple experiments to strengthen our work. Since there is no dedicated common space for experiments, we present them at the beginning of the responses to reviewers 8fbK (part 1) and J3ay (part 2).

Summary of experiments (part 1)

Experiment (1) We reproduced Fig. 2 (top) with SimCLR-v2 with the ResNet-50 architecture for ImageNet-1k (1000 classes).

Experiment (2) We reproduced the experiments in Fig. 2 (top) for SimCLR-v2 with the ViT_base architecture. The experiments were conducted for TinyImageNet (200 classes), CIFAR10, CIFAR100, and mini-ImageNet.

CIFAR10

CIFAR100

Mini-imagenet

Tiny-imagenet (C=200)

Experiment (3) We reproduced the experiment in Fig. 4 (bottom) with a pre-trained ResNet-50 that was trained on ImageNet-1k and evaluated the model on both ImageNet-1k and TinyImageNet.

mini-imagenet

Full-imagenet

Experiment (4) We reproduced the experiment in Fig. 4 (bottom) with the ViT_base architecture for CIFAR10, CIFAR100, mini-ImageNet.

CIFAR10

CIFAR100

Mini-imagenet

Response to 8fbK

Reviewer: The experiments show that the gap between DCL and NSCL decreases as the number of classes increases, which aligns well with the theoretical analysis. However, the performance difference between the two methods remains noticeable. More discussion on this observation could help clarify how much of the theoretical alignment translates into practical outcomes.

The loss‑gap bound in Thm. 1 is label‑agnostic: it shows how DCL incentivizes the embeddings to support any labeling (see lines 155–161), but it does not imply that the learned embedding geometry will match a specific labeling, as NSCL does. In fact, under the unconstrained features model and assuming $d > N$, any global minimizer of DCL forms an $N$‑way equiangular tight frame (ETF). This configuration drives directional‑CDNV to 0 even when the full CDNV remains non‑zero.

As observed in Figs. 5, DCL effectively drives the directional‑CDNV to low values, while Thm. 2 for NSCL implies that both directional and regular CDNVs converge to low values. As predicted by Prop. 1 (and verified in Fig. 4 (bottom)), this difference explains why DCL achieves reasonable downstream performance, but NSCL—by driving both CDNV measures lower—ultimately performs even better.

In conclusion, we do not expect DCL and NSCL to learn the exact same representations. However, our theory characterizes their relationship, clarifies the types of representations produced by DCL training, and explains why these representations remain effective for downstream tasks using linear probes.

Reviewer: In addition to DCL and NSCL, it would be helpful to include comparisons with standard CL methods. Based on the paper’s analysis, the gap between CL and NSCL is expected to be even larger, which could further support the theoretical claims.

Following the reviews, we conducted an experiment to reproduce Fig. 2 (top) for CL-training (see experiment (6) in the response to J3ay). In addition, we evaluated CL, DCL and NSCL and the bound for DCL/CL-trained models and observed negligible differences in the values of CL and DCL (see experiments (1), (2) and (6) in the responses to 8fbK and J3ay).

Theoretically, CL and DCL differ only by removing the anchor sample from the denominator, and the proof of Thm. 1 applies to both. Specifically:

This implies

Reviewer: Can the authors quantify the similarity of the learned representations—not just the loss values—between CL/DCL and NSCL-trained models? For example, using clustering metrics and others.

We conducted new experiments to quantify the representation similarity between CL/DCL- and NSCL-trained ViT models (see experiment (7) in the response to J3ay).

Reviewer: While the paper presents clear analysis and supporting experiments, further evaluation on a wider range of models and larger-scale datasets would be helpful. This could clarify how well the findings generalize beyond the current experimental setting.

Thank you for the suggestion. Following the reviews, we conducted extensive experiments with the ViT architecture, as well as the ImageNet-1k (1000 classes) and Tiny-ImageNet (200 classes) datasets (see experiments (1)-(5) in the responses to 8fbK and J3ay).

Dear reviewer 8fbK,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

Best, AC