On the Similarities of Embeddings in Contrastive Learning

We thank the reviewer for the thoughtful and constructive review, especially the effort to understand and verify our theoretical results. We are glad the reviewer saw our analysis as a meaningful extension of existing CL theory and appreciated its new perspective on the role of negative pair similarity. Below, we provide detailed responses to your constructive comments.

---

Q1 & C1-1. The difference between cross-view and within-view negatives in the single-modal scenario?

Thank you for the insightful question. The figure below illustrates the structural difference between cross-view and within-view negatives in the single-modal case. Each graph has six nodes for three instances, with edges denoting negative pairs: $(u_i, v_j)$ for cross-view, and $(u_i, u_j)$ for within-view, where $i\neq j$.

Fig.4 Illustration of cross-view and within-view negatives.

One can confirm that cross-view graphs are fully connected bipartite, whereas within-view graphs consist of disconnected subgraphs. This topological difference highlights their non-equivalence, even in the unimodal setting.

We will clarify this distinction in the revision.

---

Q2 & C2. Is excessive separation still a problem under the random mini-batch scenarios?

Yes, as confirmed by additional experiments. In Table 1 of our response to Reviewer BhMK, excessive separation still emerges under random mini-batching and correlates with performance degradation when it occurs more frequently.

---

Q3 & C3. The negative effect of imperfect alignment?

Imperfect alignment degrades representation quality. Using the sigmoid loss (in Example 5.4) with varying bias $b$, we observe in the figure below that lower positive similarities correlate with reduced top-1 accuracy in linear probing. This demonstrates the performance sensitivity to alignment quality.

Fig.5 Negative effect of imperfect alignment.

---

Q4. VRN performance under additive contrastive loss and full ImageNet?

We evaluated VRN loss in the linear probing setting on CIFAR-10. Using Sigmoid loss alone ($t=1$, $b=-1$) yields 81.10% accuracy, while combining it with VRN loss ($\lambda=30$) improves performance to 87.81%, under the same training setup as in our paper. This demonstrates a significant gain from incorporating VRN, under additive contrastive loss.

We applied VRN to SimCLR on full ImageNet, using the same setup as in our ImageNet-100 experiments. After 100-epoch training with $t=0.2$ and $\lambda=40$, our method (SimCLR+VRN) achieved 52.48% top-1 accuracy, outperforming the SimCLR baseline (51.79%). This confirms the scalability of VRN to large-scale datasets.

---

C1-2. The contribution, that within-view negative pairs can mitigate the excessive separation of negative pairs, is supported by neither specific theorems nor experiments.

The mitigation effect of within-view negatives is theoretically supported by Theorems 5.1 and 5.3 (see final paragraph of Sec. 5.1). Theorem 5.3, based solely on cross-view negatives $(c_1, c_2) = (1, 0)$, leads to excessive separation due to the independently additive form of the loss. In contrast, the loss in Theorem 5.1 includes both cross-view and within-view negatives $(c_1, c_2) = (1, 1)$, which alleviates this issue.

To empirically validate this, we evaluated the sigmoid loss (as in Example 5.4) with and without within-view negatives. As shown in Table 2 of our response to Reviewer BhMK, adding within-view negatives reduces the proportion of excessively separated negative pairs, confirming the theoretical insight.

---

C4. Accuracy drops after incorporating VRN?

The initial drop was due to using a fixed VRN weight ($\lambda=30$) at submission time. After tuning $\lambda$, we observed consistent performance gains across all methods and datasets. Updated results are provided in Fig.1 of our response to Reviewer BhMK.

---

S1. Are $(x,y)$ always positive pairs? Later parts seem to allow negative pairs as well.

There was a notational oversight in Sec. 3. The notation $(x,y)$ should represent either positive or negative pairs depending on the sampling. From Sec. 4, we clarify this with $(x,y)\sim p_{pos}$ or $p_{neg}$. We’ll revise Sec. 3 to reflect this properly.

---

S2. The notation of $Var$.

Thank you for pointing this out. The standard form is $Var[X]=\mathbb{E}[(X-\mathbb{E}[X])(X-\mathbb{E}[X])^\top]$, while we wrote $\mathbb{E}[(X-\mathbb{E}[X])^\top(X-\mathbb{E}[X])]$. We’ll revise the appendix to follow the standard or express it as an expectation.

We thank the reviewer for acknowledging that our mathematical analysis provides important insights for contrastive learning in both unimodal and multimodal settings. We are also encouraged by the positive evaluation of our experimental claims and the effectiveness of our proposed VRN approach on classification tasks. Following the reviewer’s constructive suggestions to strengthen our paper, we provide additional experiments as follows.

---

E1. More in-depth analysis of embeddings and the structure of the embedding space after the proposed VRN approach.

To analyze the effect of the proposed VRN loss on the embedding space, we conduct experiments on both synthetic and real-world datasets.

[Experiment on Synthetic Data]

We use a synthetic dataset of 4 samples, each augmented twice (8 embeddings in 3D). Embeddings are optimized directly using SGD (lr=0.5, 100 steps, mini-batch size=2). We compare two setups: (1) SimCLR with temperature $t=0.5$, and (2) SimCLR + VRN loss with $\lambda=3$.

Fig.3. Visualization of learned embeddings.

As shown in the figure above, both methods successfully align positive pairs. However, with SimCLR+VRN, the negative pairs are more evenly distributed in cosine similarity, centering around the theoretical optimum of $-1/3$. Specifically:

• SimCLR: mean = -0.3201, std = 0.2051
• SimCLR+VRN: mean = -0.3327, std = 0.0207

Given the absence of semantic structure in the synthetic samples, a uniform separation among negative embeddings is desirable. The VRN loss seems to facilitate such balanced distribution.

[Experiment on Real Data]

We further assess how VRN affects negative pair distribution in realistic scenarios. Using ResNet encoders trained on CIFAR100 with either SimCLR or SimCLR+VRN, we sample 5,000 negative pairs from augmented training images and compute their cosine similarities.

We repeat this across different batch sizes and report the variance of negative-pair similarities:

Table 3 The effect of VRN loss on the variance of negative-pair similarity, where embeddings are generated from models pretrained with different batch sizes.

We observe that the variance of negative-pair similarity consistently decreases when VRN is used, across all batch sizes. We will include these experimental results and their discussion in the revised version of the manuscript.

---

S1. The notation d in lines 242, 283 and 313 seem to not have been introduced in the paper.

Thank you for pointing this out. The notation $d$ refers to the dimension of embedding vectors, as mentioned in line 96 in Sec. 3 (Problem Setup). We will clarify this in the relevant lines.

We appreciate the reviewer’s positive feedback, including that our paper is well-written, provides a clear explanation of results, and may be useful for future CL research. Below, we address each of the reviewer’s comments in detail.

---

W1. When VRN loss term downgrades the performance in Fig.2?

Fig. 2 in our paper has been updated (see below) with improved hyperparameter tuning:

Fig.1 Effectiveness of VRN.

In earlier results, VRN degraded performance on CIFAR-100 (e.g., DCL, DHEL), likely due to a fixed weight $\lambda=30$. After tuning over a wider range of {0.1, 0.3, 1, 3, 10, 30, 100}, we observe consistent improvements across all methods.

We also note:

• VRN operates on cross-view negatives,
• SimCLR includes such pairs, DHEL does not,
• and gains from VRN are larger when the base loss includes cross-view negatives (e.g., SimCLR).

This suggests VRN is most effective when complementing losses that already involve cross-view negatives.

---

W2. Limitations on using VRN and scenarios where it is benefitial to use it.

We summarize below the scenarios where VRN is most effective, followed by its main limitations.

1. When VRN is beneficial

• Small batch training: From Theorem 5.5, small batches lead to higher variance in negative-pair similarities. Since VRN minimizes this variance, it yields stronger gains in low-batch regimes. Empirical results in Fig.1 support this.
• Robustness to temperature: Contrastive loss is sensitive to the temperature parameter, which affects embedding similarity distributions [R1]. VRN encourages negative similarities toward the optimal $-1/(n-1)$, reducing performance fluctuation. Below, we show SimCLR with VRN ($\lambda=30$) yields more stable accuracy across temperature values on CIFAR-10/100:

Fig.2 Robustness to temperature.

[R1] Wang, et al. Understanding the behaviour of contrastive loss. CVPR2021.

2. Limitations of VRN

• Loss of meaningful structure: Variance in negative similarities can reflect semantic diversity. Forcing uniform similarity may suppress this, as discussed in Sec. 4 (L233–245) and Sec. 7 (L433–436).
• Diminished effect with large batches: As batch size increases, the natural variance stabilizes, reducing the marginal benefit of VRN.
• Hyperparameter tuning: VRN introduces an additional weight $\lambda$, which requires tuning per setting.

---

E1. How many negative pairs fall above the optimal threshold?

We evaluated the ratio of negative pairs whose cosine similarity falls below the theoretical threshold of $-1/(n-1)$, using models pretrained with SimCLR (ResNet-18, CIFAR-100, $t=0.2$).

We generated 5,000 negative pairs from each model by applying random augmentations and measuring cosine similarity from the projector output. The table below summarizes the results:

Table 1 Ratio of excessively separated negative pairs and associated statistics.

We find that over 57% of negative pairs fall below the optimal similarity across all batch sizes. Smaller batches show greater variance and slightly lower accuracy, aligning with Theorem 5.5, which predicts increased variance in negative similarity with reduced batch size.

---

Q1. How many of negative pairs does not satisfy the optimal thershold if a method only uses negative pairs into the loss function?

We interpret your question in two possible ways:
(1) How many negative pairs fall below the optimal similarity threshold when training only with the VRN loss, or
(2) when using a contrastive loss with only cross-view negatives.

Regarding (1), the VRN loss is not intended to serve as a standalone training objective. It is designed to regularize standard contrastive losses, and cannot learn meaningful representations on its own.

Regarding (2), we report statistics from models trained with sigmoid loss (in Example 5.4), which uses only cross-view negatives. In this case, a large fraction of negative pairs have similarities below the threshold of $-\frac{1}{n-1}$, where $n$ is the sample size. This indicates over-separation. When the loss is modified to include both cross-view and within-view negatives, this issue is mitigated, as shown below:

Table 2 Negative pair similarity in models pretrained with sigmoid loss.

Please let us know if this interpretation differs from your intent — we would be glad to clarify further.

---

Q2. Is there any toy experiments?

Please see E1 in our response to Reviewer qfkv.