How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction

We are grateful to you for your valuable comments and constructive suggestions.

Q1: A constant 2 appears suddenly. Inequality (4) holds only when inner product is greater than 0.

A1: Thanks. The constant 2 is not necessary. We have deleted this constant factor. Therefore, inequality (4) holds without the requirement that the inner product term is greater than 0. We have also corrected the corresponding result in Theorem 4.2.

Q2: Line 902-904: why does inequality (3) hold?

A2: Thanks. We have corrected the inequality (3) in the anonymous link: https://imgse.com/i/pEsQ5Pe. We have also corrected the corresponding result in Theorem 4.7.

Q3: Line 923-925: $1/2\epsilon_q^2$ appears suddenly.

A3: Thanks. We have corrected the formula from line 918 to 925 in the anonymous link: https://imgse.com/i/pEsQ5Pe. We have also corrected the corresponding result in Theorem 4.7.

Q4: Some experiments of domain generalization ability.

A4: Thanks. In our opinion, this question is about the domain transfer ability from pre-training data to downstream classification data. However, our work assumed the distribution of downstream classification data is the same as the distribution of pre-training data. Therefore, we were unable to study domain transfer ability. We will attempt to mild this assumption to analyze the impact of SVD on domain transfer ability from empirical and theoretical perspectives.

If our understanding is wrong, please do not hesitate to contact us. We will make some more clear explanations at once.

Q5: Cite some works.

A5: In our modified manuscript, we have made some discussions about these works you mentioned.

[1] proposed the concept of augmented distance and provided some upper bounds revealing the theoretical effect of augmented distance on understream classification performance. Specifically, they found that the classification performance of contrastive SSL is related to three key factors: alignment of positive samples, divergence of class centers, and concentration of augmented data. Theorem 2 in our work provided both upper and lower bound, which can not only give similar conclusions but also reveal some additional factors. Firstly, the term $V(f(x)|y_{\bar{x}})$ implies the alignment of positive samples. Secondly, the term $V_{y_{\bar{x}}^{\neg}}(f(x)|y_{\bar{x}})$ stems from labeling error caused by data augmentation, which is similar to the concentration of augmented data. Thirdly, the term $V(f(x^-)|y^-)$ implies the alignment of negative samples, which is not considered by [1]. More importantly, we further improved the bounds of Theorem 2 via data dimensionality reduction and provided the corresponding theoretical analysis and empirical observations (Section 4.2).

[2] established a theoretical framework for weakly supervised contrastive learning for the first time. Their results revealed that 1) semi-supervised information improves the error bound compared with purely unsupervised contrastive learning by using all labeled samples; 2) joint training of supervised and unsupervised contrastive learning does not improve the error bound compared with purely supervised or purely unsupervised contrastive learning. Although weakly supervised contrastive learning is not the topic of our work, the labeling error considered by our work is analogous to a type of weak supervision, i.e., noisy-labeled information. Therefore, we will extend the theoretical analysis of this work to weakly supervised contrastive learning in our future work. Besides, [1] and our work both gave the suggestion that we should choose a moderate feature dimension $k$, which enhances the credibility of our suggestion.

[1]Huang et al., Towards the Generalization of Contrastive Self-Supervised Learning. ICLR 2023.

[2]Cui et al., Rethinking Weak Supervision in Helping Contrastive Representation Learning. ICML 2023.

Q6: Whether the combination of SVD and moderate embedding dimension would bring positive or negative effects.

A6: Thanks. This question is the concern stated in the remark of Theorem 4.9. We use a moderate embedding dimension to mitigate the reduction of the connectivity for the augmentation graph. According to our empirical observations, if we use a moderate embedding dimension, the positive effect of SVD on the downstream classification performance is not offset by the reduction of the connectivity for the augmentation graph at least. We don’t know whether a moderate embedding dimension can completely eliminate this reduction, which is a limitation of our work. We aim to further explore the impact of the moderate embedding dimension in our future work, which was mentioned in the first point of Limitation (Appendix E).

Q7: "SLT-10"to "STL-10".

A7: Thanks. We have corrected this typo.

Q8: Domain generalization performance of the model?

A8: Please see A4.

We are grateful to you for your valuable comments and constructive suggestions.

Q1: Still no evidence showing whether the labeling error is sufficiently small in this paper.

A1: Thanks for your constructive comment. Our work didn’t guarantee that the label errors are sufficiently small, which was discussed in the second point of the Limitations section in Appendix E (line 984). The primary objective of this study is to demonstrate that the label error under the setting of prior work [1] is not negligible. Our approach further reduced labeling error, as validated in Tables 5 and 6. Achieving sufficiently low label error remains an ongoing research objective.

[1]Wang, Y., et al., Do generated data always help contrastive learning? ICLR, 2024.

Q2: No clear definition of $\epsilon$.

A2: Thanks. We have supplemented their clear definitions as follows. $\epsilon(\alpha_{q})$ is defined as the maximum distance between $f(x),f(x^+)$ when taking the truncated parameter $q$, where $(x,x^+)$ is a false positive sample pair. And, $\epsilon(\alpha_{q^*})$ is defined as the minimum distance between $f(x),f(x^+)$ for any truncated parameter values. In summary, $[\epsilon(\alpha_{q^*}), \epsilon(\alpha_{q})]$ is the range of value of $||f(x)-f(x^+)||$ for $(x,x^+)\sim p(x,x^+, y_{\bar{x}}^\neg)$.

Q3: Why does the formula hold in Theorem 4.9?

A3: The proof of the result for Theorem 4.9 was provided at the end of Appendix C. This result is quite similar to Lemma B.5, with the key difference being the definition of $E(f,W)$. In the proof, we have rewritten the form of this definition as $\underset{\bar{x}\in\bar{D},x\in p(\bar{x}|\bar{x})}{\mathrm{Pr}} \left[g_{f^*,W^*}(x) \neq y_{\bar{x}}\right]$ to facilitate the comparison with Lemma B.5. Obviously, $x\in p(\bar{x}|\bar{x})$ is included in $x\in p(\cdot|\bar{x})$. According to Lemma B.5, the result of Theorem 4.9 holds.

Q4: The improvements on Table 4 and Table 5 appear to focus exclusively on CIFAR-10, which may seem inconsistent given that CIFAR-100 and STL-10 were utilized in the earlier experiments.

A4: Thanks. We have added related experiments on CIFAR-100 and STL-10 (anonymous link: https://imgse.com/i/pEsQI8H).

Q5: The conclusions of this paper may not be universally applicable, as certain factors still involve heuristic choices, including the degree of data inflation, the extent of weak augmentation, the number of dimensions selected for data reduction, and the domain of the datasets used.

A5: Thanks. We have made some explanations about the several factors you mentioned.

1) Data Inflation: Data inflation is not the focus of this paper. Previous work [1] has shown that the more similar the distribution of inflated data is to the one of original data, the better the model performance. Therefore, this paper adopts the optimal inflation setting from [1].

2) Weak Augmentation: We also adopt the weak augmentation strategy suggested in [1].

3) Data Dimensionality Reduction: The SVD truncation parameter used in this work is manually set to verify the effectiveness of SVD, not to obtain optimal model performance. Therefore, we set a large $q$ value in most experiments. In the future, we will adaptively learn the optimal truncation value $q^*$, which is mentioned in the second point of Limitation in Appendix E.

4) Data Domain: The experiments and theoretical analysis of this work are focused on the field of computer vision. Exploring whether the findings generalize to domains such as NLP and graph learning is an interesting and open question. Future work will extend this investigation to NLP and graph learning to assess the broader applicability of the proposed framework.

[1]Wang, Y., et al., Do generated data always help contrastive learning? ICLR, 2024.

Q6: "ultimately improving improve model performance" to "ultimately improving model performance"

A6: Thanks. We have corrected this issue.

Q7: Still no evidence showing whether the labeling error is sufficiently small in this paper.

A7: Please see A1.

Q8: Will the conclusions of this paper also hold for other contrastive learning domains, such as NLP and graph learning?

A8: Thanks. The experiments and theoretical analysis of this work are focused on the field of computer vision. Exploring whether the findings generalize to domains such as NLP and graph learning is an interesting and open question. Future work will extend this investigation to NLP and graph learning to assess the broader applicability of the proposed framework.

We are grateful to you for your valuable comments and constructive suggestions.

Q1: Since the labeling issue ultimately stems from imperfections in the augmentation process, could it be addressed simply by making more careful choices in augmentation? For instance, setting the cropping ratio appropriately or relying on smarter augmentation techniques like AutoAugment.

A1: Thanks for your constructive comment.

Firstly, data augmentation presents a dual challenge in deep learning systems: excessive augmentation introduces label noise due to over-distorted samples, while insufficient augmentation compromises model performance through limited diversity. Secondly, the optimal augmentation method varies across different datasets. Optimal augmentation selection requires non-trivial domain-specific expertise and extensive empirical validation.

Besides, some smarter augmentation techniques like AutoAugment can improve model performance by dynamically adjusting augmentation strategies. However, their time-consuming and resource-intensive nature limits practical implementation. In contrast, this work pioneers a novel paradigm by employing data dimensionality reduction as a pre-process strategy to improve model performance effectively. Empirical and theoretical results validate the effect of dimensionality reduction. We will conduct research on dimensionality reduction to develop a more flexible low-rank approximation method as a new augmentation method to achieve greater model performance, which was mentioned in the second point of Limitation (Appendix E).

Q2: Additionally, I feel there should be experiments to justify how frequently this type of labeling error occurs in real-world scenarios.

A2: Thanks. We have added experiments on multiply augmentation combinations to justify how frequently this type of labeling error occurs in real-world scenarios (anonymous link: https://imgse.com/i/pEsQo2d).

We are grateful to you for your valuable comments and constructive suggestions.

Q1: ImageNet?

A1: Thanks. Considering the size of the original ImageNet is too large, it is hard to complete the experiment on it before the deadline (March 31 (AoE)). We have added experiments on TinyImageNet (anonymous link: https://imgse.com/i/pEsQWVK).

Q2: Discontinuous singular value in Table 1?

A2: We have added experiments for $s_{i,j}$ (link: https://imgse.com/i/pEsasuF).

Q3: Are improvements meaningful?

A3: Although a few experimental results did not demonstrate significant improvement, our all experiments exhibited a consistent trend of progress across different settings. In some cases, the improvement exceeded 1%, even when q=30 (such as columns 3,5,6 in Table 4). Further reductions in the q value may lead to additional performance gains.

Q4: Why q=30 not q=25?

A4: Since the optimal q varies across different experimental settings (Table 2), we fixed q=30 for subsequent experiments to ensure consistent improvement in downstream performance across varying conditions, thereby validating our theory, even though the improvement might be relatively limited in some settings when adopting q=30. Note that, this paper aims not to obtain optimal downstream performance but to investigate how dimensionality reduction affects labeling error.

Q5: Other augmentations?

A5: We have added experiments with Random Erasing, GridMask, and HidePatch (link: https://imgse.com/i/pEs15jA). They also support the claims made in this paper.

Q6: Discussion of dimensionality reduction?

A6: Thanks. We have supplemented the discussion of various dimensionality reduction techniques. Beyond SVD, there are various techniques for feature extraction and dimensionality reduction, including matrix and tensor decomposition (PCA[1], NMF[2]), dictionary learning[3], compressed sensing[4], deep learning (Autoencoder[5], GAN[6]). Since we use SVD as a simple example to study the effect of dimensionality reduction on contrastive learning, we didn’t carefully make a survey about various dimensionality reduction methods. Our empirical observations and theory show that an effective data augmentation based on dimensionality reduction is necessary. We will conduct research on dimensionality reduction to develop a flexible low-rank approximation as a new augmentation to achieve greater model performance, which was mentioned in the second point of Limitation (Appendix E).

[1]Y. Ren, et al., Hyperspectral Image Spectral-Spatial Feature Extraction via Tensor Principal Component Analysis, Geoscience and Remote Sensing Letters, 2017.

[2]M. Chen, et al., Feature Weighted Non-Negative Matrix Factorization, Transactions on Cybernetics, 2023.

[3]P. Song et al., Multimodal Image Denoising Based on Coupled Dictionary Learning, International Conference on Image Processing (ICIP), 2018.

[4]T. Hong, et al., A Complex Quasi-Newton Proximal Method for Image Reconstruction in Compressed Sensing MRI, Transactions on Computational Imaging, 2024.

[5]Y. Shen, et al., DRACO: A Denoising-Reconstruction Autoencoder for Cryo-EM, NeurIPS, 2024.

[6]D. Chen, et al., SSL: A Self-similarity Loss for Improving Generative Image Super-resolution, ACM MM, 2024.

Q7: Experimental settings?

A7: To ensure reproducibility, we have updated the experimental details in Appendix D of our revised manuscript.

Hardware: All experiments are executed on an RTX 2070 GPU with an Intel(R) i7-10750H CPU.

Software: Python 3.7.4 with PyTorch 1.13.1.

Optimizer, epoch, batchsize: We provided them in the second point of Appendix D. Note that we pre-train STL-10 with 50 epochs due to its substantial data volume.

Augmentation: We provided the parameter of every augmentation in the fourth point of Appendix D.

Supplementary experiments in our response: Complete implementation details are included in their respective response.

Should any additional technical specifications be required? Please inform us and we will promptly provide the requested information.

Q8: Cost from SVD.

A8: We have added the cost of SVD on different datasets (link: https://imgse.com/i/pEsQfUO).

Q9: Validation on different backbones and datasets.

A9: We have added experiments on TinyImageNet and ViT, ConvNeXt (link: https://imgse.com/i/pEsQWVK).

Q10: Other augmentation?

A10: Please see A5.

Q11: Define moderate embedding dimensions.

A11: We have added experiments on ViT, ConvNeXt, and BYOL (link: https://imgse.com/i/pEswcf1). We find that the optimal dimension is usually in $[512, 2048]$. The impact of the moderate dimension is valuable to be further explored in our future work, which was mentioned in the first point of Limitation (Appendix E).

Q12: Are the improvements significant?

A12: Please see A3.