Enhancing Contrastive Learning with Variable Similarity

We appreciate the reviewer's positive comments for our paper. We address the concerns raised point by point below.

Comment_1: Differences of proposed method with CoCor.

Response_1: Thanks for your comment. We would like to clarify that CoCor and our proposed method differ significantly in both methodology and design philosophy.

CoCor defines augmentation strength based on the frequency or number of transformations applied to a sample. Specifically, a composite augmentation with more operators is defined as a stronger augmentation. Based on this, CoCor proposed a data augmentation consistency, i.e., the similarity between the augmented view and the original input decreases with the strength of the augmentation. However, the learning of this similarity structure in CoCor is supervised by ground-truth class labels, which are required to guide the alignment of augmented views with their semantic categories. This makes the approach label-dependent and less applicable in fully unsupervised settings.

In contrast, our method is based on a different perspective of augmentation. Even though positive pairs are derived from the same data sample, different parameterizations within the same augmentation type will induce variations in the resulting views. Therefore, the similarity between such augmented views should not be assumed constant, but rather should vary in accordance with the applied augmentation parameters. We define similarity not based on the number of transformations, but rather based on the parameter variations within the same augmentation type. For instance, two crops with small scale differences are considered more similar than two crops with large scale difference. Our method predicts this similarity via an augmentation-aware module $\varphi$ and aligns it with the representation similarity in a self-supervised manner. This enables the model to capture semantic consistency from augmentation parameters, without relying on human-annotated labels.

In summary, while both approaches utilize augmentation-related cues, our method is fundamentally different in its definition of similarity, learning strategy, and reliance on supervision. We believe these differences not only constitute a clear methodological novelty, but also broaden the applicability of our approach to a wider range of unsupervised learning scenarios.

Comment_2: Effectiveness of individual augmentation.

Response_2: Thanks for your comment. To better understand the effect of individual augmentation types on the predicted similarity, we conducted an ablation study to evaluate each augmentation’s contribution.

Specifically, we conduct this experiment based on the SimSiam method. We isolate the effect of each of the following augmentations: random cropping, color jittering, Gaussian blurring, and horizontal flipping. When predicting the similarity between two augmented views, we use only the parameter difference from a single augmentation type, effectively assessing its individual contribution to the variable similarity estimation. The following results are obtained:

These results reveal several insights: Crop and color jitter yield stronger performance when used alone, suggesting that their parameter differences offer more informative cues about view similarity. In contrast, blurring and flipping result in slightly lower performance. This may be attributed to their limited influence on the image’s semantic content, as flipping alters only the orientation and blurring affects low-level details without substantially changing object-level features. Consequently, their limited semantic impact makes them less effective in providing informative variation for similarity estimation.

Thanks a lot again for your positive comments on our paper! We sincerely hope that our responses can further improve the clarity of the paper.

Thank you for your appreciation of the novelty, theoretical analyses, and experimental results of our paper! Thanks also for your very insightful and constructive suggestions! Our point-by-point responses are as follows.

Comment_1: Clarity of Notation.

Response_1: We appreciate the reviewer for pointing out this issue. To clarify, we summarize the meaning and role of each similarity term used:

We will introduce the above notations with clearer textual explanation accompanying each equation if the paper is finally accepted.

Comment_2: Parameter Sensitivity.

Response_2: Thanks for your comment. The sensitivity analyses of $\lambda$ and $\gamma$ have already been presented in Tab. 9 and 10 of our manuscript, respectively. The value of $\omega$ follows the same setting as in AugSelf [R1]. To further address the reviewer’s concern, we additionally conduct a sensitivity analysis for $\omega$, while fixing $\lambda = 0.5$ and $\gamma = 1$.

These results indicate that the performance is relatively stable across a reasonable range of $\omega$, suggesting that the method is not highly sensitive to this parameter.

Comment_3: Negative Sampling Extension.

Response_3: Thanks for your comment.

Our method focuses on adjusting the similarity objective for positive pairs, as they are directly generated from the same instance via data augmentations. In this case, the augmentation parameters provide a meaningful and grounded basis for modeling semantic similarity.

Extending the concept of variable similarity to negative pairs is indeed an interesting direction. However, we note a fundamental difference: negative pairs typically consist of different samples, and their similarity arises not only from augmentations but also from inherent semantic differences. As such, using augmentation parameters alone to model the similarity of negative pairs may not be sufficient or reliable.

One potential solution could be inspired by information-theoretic principles: we may estimate the inherent similarity between samples (e.g., using approximations of mutual information), and combine it with augmentation-induced variation to form a more comprehensive similarity measure for negative pairs. This would allow the model to jointly consider intrinsic sample differences and augmentation-induced discrepancies when weighting negative similarities.

We believe this integration requires careful design and is beyond the scope of the current work, but we appreciate the reviewer’s suggestion and will consider it as a future research direction.

Thanks a lot again for your very detailed comments on our paper! We sincerely hope that our responses can further improve the clarity of the paper.

[R1] Improving Transferability of Representations via Augmentation-Aware Self-Supervision, NIPS, 2021.

Thank you for your positive and constructive comments! Our point-by-point responses are provided below.

Comment_1: Similarities and differences with AugSelf.

Response_1: Thanks for your comment. As the reviewer suggested, our proposed method and AugSelf both argue that the augmentation-invariance assumption in traditional contrastive learning may harm semantic consistency. However, the two approaches address this issue from completely different perspectives.

To be specific, AugSelf uses an auxiliary task to only preserve augmentation information in its forward phase. In contrast, our method further explores the augmentation information to explicitly modulate the variable contrastive-supervision signal for both forward and backward phases, leading to three important technical differences:

1. Dynamic Similarity Adjustment: We introduce a variable similarity mechanism that dynamically adjusts the similarity supervision between two augmented views. Nevertheless, AugSelf does not incorporate such a dynamic mechanism, and thus it cannot explicitly adjust the similarity supervision in the corresponding learning objective.

2. Relative Similarity Constraint: We further introduce a relative constraint that enforces the similarity between weakly augmented views to be reasonably higher than that between weakly and strongly augmented views. This relational structure ensures the important semantic consistency, which is absent in AugSelf.

3. Augmentation-Type Aware Similarity Estimation: AugSelf treats all augmentation types equally. In contrast, our method explicitly accounts for the transformation that induces the most significant semantic change and uses the minimum similarity across types as the target. This practice allows the model to focus on the most challenging transformation, leading to robust representation.

We will add the above discussion in the camera-ready version if this paper is finally accepted.

Comment_2: Effectiveness of more epochs.

Response_2: Thanks for your comment.

We would like to clarify that the number of training epochs in our experiments follows the same setting as in previous works [R1, R2, R3], where 200 epochs are commonly used for evaluations on both the ImageNet-100 and ImageNet-1k datasets. Moreover, the results of Tab. 11 in the appendix show that our proposed method remains effective when trained for 800 epochs on the ImageNet-100 dataset, indicating that its effectiveness is successfully demonstrated with more training epochs.

For ImageNet-1k, here we follow the reviewer's suggestion to train our model with 800 epochs by using 8 NVIDIA Tesla V100 GPUs. The training setup follows the baseline method SimSiam, with a learning rate of 0.05, and batch size of 256. The corresponding results are shown in the following table.

As shown in the above table, our method achieves the best top-1 accuracy on the ImageNet-1k dataset among all compared methods. The experimental results demonstrate the effectiveness of our proposed method with more training epochs.

Thanks a lot again for your very insightful comments on our paper! We sincerely hope that our responses can solve your main concerns.

[R1] Contrastive Learning with Consistent Representations, TMLR, 2024.

[R2] Contrastive Learning with Stronger Augmentations, TPAMI, 2023.

[R3] On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learning, ICML, 2024.

Thanks for the authors’ detailed responses. I think the authors have solved my concerns, so I will raise the score to 5.