ConMix: Contrastive Mixup at Representation Level for Long-tailed Deep Clustering

Response to Reviewer RUUY (1/2)

Thank you for reading our paper carefully. We have carefully prepared our response as follows.

Weakness 1 and Question 2: Could the authors comment on the performance of the proposed approach in the balanced setting.

A: Thank you for this valuable suggestion. Following your suggestion, we further trained and tested the performance of different methods on balanced datasets. The dataset configurations and experimental setups refer to [1]. Specifically, we trained different methods using ResNet-18 for 1000 epochs on CIFAR-10 and CIFAR-20. Since ConMix does not leverage some advanced techniques suitable for balanced clustering, we propose an updated version of ConMix called "ConMix+Propos" that embeds ConMix into Propos [1]. We first train the model with the loss of ConMix for 500 epochs, then train it with Propos for another 500 epochs. The total number of training epochs for this updated version is the same as other methods.

We have reported the results of the balanced dataset in the following tables, and also provided the clustering performance on the datasets with an imbalance ratio of 10 in the parentheses ( $\cdot$ ), for better comparisons.

The experimental results are as follows.

CIFAR-10:

CIFAR-20:

Compared to the baseline methods (SimCLR, SDCLR), ConMix demonstrates significant performance improvements on both the balanced datasets and the long-tailed datasets. Compared to recent deep clustering methods (CC, IDFD, Propos), ConMix shows improvements on long-tailed datasets but may not perform as effective as Propos on balanced datasets. Note that, ConMix still outperforms CC and IDFD on most cases even on the balanced datasets.

However, the updated version "ConMix+Propos" performs the best on both balanced datasets and long-tailed datasets, showing that adding some recent deep clustering techniques on "ConMix" will further improve its performance and robustness.

Moreover, we also notice that existing deep clustering algorithms perform well on balanced datasets but suffer severe performance degradation on long-tailed datasets. We believe this is due to these methods making assumptions that are aligned with balanced datasets. While they can achieve good laboratory performance, they are less suitable for realistic long-tailed distributions. This further underscores the importance of research on long-tailed deep clustering.

The above results and analyses have been added in Appendix C of the updated paper to demonstrate robustness of our work. Thank you.

[1] Learning Representation for Clustering Via Prototype Scattering and Positive Sampling, TPAMI, 2023.

Weakness 2: How are the "stochastically assigned tags" selected?

A: In each batch, we assign a random tag to each input for synthesized representations generation. For example, when the batch size is 512 and the number of synthesized representations $M$ is 100, we generate a sequence of 512 random numbers, with values ranging from 1 to 100, where the generation of these random tags follows a uniform distribution, i.e., for a single input, the probability of being assigned to any specific tag is $\frac{1}{M}$.

In the revised paper, we have modified Line 216 from "We randomly select...with equal contributions" to:

"In each batch, we randomly assign tags within $[1, M]$ to original representations from the same network branch {$v_1, v_2, ...v_N$}. The generation of tags follows a uniform distribution with equal probabilities $\frac{1}{M}$ and original representations with the same tag are used to synthesize one particular representation in the manner of Eq. (3)."

Also, we apologize for the typo in Line 215. $\vert \cdot\vert$ denotes the number of elements in the set. We have fixed it in the revised version. Thank you very much for your kind reminder.

Response to Reviewer RUUY (2/2)

Weakness 3: Confusion about the settings on pairwise ConMix.

A: Sorry for the confusion caused. To control for a single variable in order to verify the improvement of multi-sampling methods over the pairwise method, pairwise ConMix also utilizes a 200-epoch SDCLR warmup. Additionally, pairwise ConMix samples weights from a beta distribution rather than using mean representations. We have clarified these points in updated paper and modified Line 408-409 from "We have also...multi-sample combinations" to：

"We have also conducted pairwise ConMix, where synthesized representations are generated by pairing samples instead of using multi-sample combinations. It samples mixing coefficient from a beta distribution and also utilizes a 200-epoch SDCLR warmup."

We hope this revision can improve readability. Thank you!

Weakness 4: Some measures of variability/statistical significance for Table 3.

A: Thank you for your valuable suggestion. Following your advice, we provide the standard deviation and significance test results. Specifically, as the ablation study in the paper, we conduct experiments on CIFAR-10 with an imbalance ratio of 10. We performed the method 10 times and perform a t-test for significance at the 5% level to compare the results of the proposed method with those of ablation study. ✝ denotes rejection of the original hypothesis and the two results are significantly different. The results in percentage are shown below.

It can be seen that the standard deviations of the ConMix methods are all within 3%, except for the CAA of ConMix w/o warmup, suggesting the robustness of our method. And the adopted ConMix w/ SDCLR warmup significantly outperforms other variants on most metrics.

Question 1: Confusion about Line 220 "...equivalent to implicitly sampling different weights from the beta distribution".

A: We apologize that our explanation may have caused you misunderstanding. In mixup, different weights are sampled from a beta distribution. The multi-sampling strategy of ConMix can also achieve varying weights due to the different numbers of samples with specific tags. For instance, let us assume that there are 3 samples associated with tag 1, and 5 samples associated with tag 2. When obtaining the synthesized representation through averaging, the sets corresponding to these two tags have different cardinalities, leading to different weights for the certain samples involved in the synthesis. Our intention is not to claim that the multi-sampling strategy of ConMix and the weight sampling from a beta distribution in mixup are mathematically identical. We have revised the original sentence to:

"but it also assigns different weights to different samples, similar to how mixup does."

Above is our response. We are grateful for your careful review. We believe that your advice improves our work. We hope our response can be satisfactory. Thank you very much!

Dear reviewer RUUY,

Thanks again for your time and efforts in reviewing this paper and the valuable comments on improving its quality. As the reviewer-author discussion deadline approaches, we hope to hear your feedback about our response. If you have further concerns, we are happy to provide more explanations. Thank you very much!

Regards from the authors.

Dear Reviewer RUUY,

We are very grateful for your willingness to participate in the discussion. Would you please let us know if our latest response has met your satisfaction? Given that the deadline for revising the paper is approaching, with less than two days remaining, we hope to learn whether you have any further concerns. If you have any concerns, we will certainly provide a prompt response and make the corresponding revisions to the paper to improve it. We look forward to your reply with great anticipation. We would be very grateful for your reply amidst your busy schedule. Thank you for the time and effort you have invested in our work.

Regards from the authors.

Response to Reviewer EuZq (1/3)

Thank you for your valuable feedback. We have carefully considered your comments and provided our answers below.

Weakness 1 and Question 1: Conduct additional experiments on large and complex datasets like ImageNet to validate the effectiveness and generalization capability of ConMix.

A: The adopted datasets are commonly used in the recent deep clustering papers [1-3]. As those recent deep clustering method did not perform experiments on ImageNet-1K, we just followed their settings. We have performed the experiments on Tiny ImageNet in Table 5 of the paper.

However, we agree with your opinion that conducting experiments on a larger dataset to validate the effectiveness and generalization capability of ConMix is important. Therefore, we conducted experiments on the long-tailed dataset ImageNet-LT. It is the long-tailed subset of ImageNet-1K, which consists of 115.8K images spanning 1,000 classes, with sample number ranging from 1280 to 5. Following [4], we trained a ResNet-50 for 200 epochs and reported the results of the last epoch. We compare ConMix with three baseline methods (SimCLR, SDCLR, BYOL) and three recent superior deep clustering methods (IDFD, CoNR, DMICC).

The results are in the table below.

The experimental results demonstrate that our method still performs well on ImageNet-LT, proving its effectiveness and generalization capability. At the same time, It may be surprising that recent state-of-the-art methods perform worse than the baseline methods. The reason is that they make assumption that data are balanced distributed, which conflicts with the real distribution of the data. This phenomenon indicates the limitations of current deep clustering methods in handling long-tailed data and highlights the urgent need for research into long-tailed deep clustering.

The experiments on ImageNet-LT have been added in Appendix E.

[1] Contextually Affinitive Neighborhood Refinery for Deep Clustering, NeurIPS, 2023.

[2] Clustering-Friendly Representation Learning Via Instance Discrimination and Feature Decorrelation, ICLR, 2021.

[3] Dual Mutual Information Constraints for Discriminative Clustering, AAAI, 2023.

[4] Prototypical contrastive learning of unsupervised representations, ICLR, 2021.

Response to Reviewer EuZq (2/3)

Weakness 2 and Question 2: Enhance the discussion on method interpretability by providing more empirical analysis regarding its impacts.

A: Thank you for your valuable suggestion. We agree that besides the theoretical analysis, empirical analysis is also crucial for understanding the actual impact of ConMix.

Following your advice, we conducted experiments on CIFAR-10 with imbalance ratios (IR) of 1, 2, 5, 10, 20, 50, and 100. And we provide the empirical analysis from the perspective of the compactness of the learned representations for different classes. Specifically, we train the baseline SimCLR and ConMix on CIFAR-10 under different imbalance ratios and calculate the class-wise similarity for each class. Class-wise similarity denotes the average similarity of samples within the same class and can indicate the compactness of representations of different classes in the feature space. Thus, to facilitate the description, we will refer to this metric as "compactness" below. We categorize the 10 classes of CIFAR-10 into Many, Medium, and Few categories based on the number of samples, following a 3:4:3 ratio. We first calculate the compactness for each class, then compute the average compactness for the three categories.

Moreover, we propose a new metric, denoted as F/My, which represents the ratio of the Few category compactness to the Many's, to measure the balance between head classes (Many) and tail classes (Few). The larger F/My is, the greater the discrepancy in compactness between head classes and tail classes is, indicating a more severe impact of the long-tailed distribution. The results are shown in the below table.

From the table, we can derive the following empirical analysis:

(1) The head classes (Many) typically have smaller compactness values, while tail classes (Few) have larger compactness values. This suggests that due to long-tailed effect, head classes occupy more of the feature space than tail classes.

(2) When the distribution is long-tailed, the compactness for Few category is relatively higher, while the compactness for Many category is relatively lower, leading to a larger F/My ratio. This is a negative effect of the long-tailed distribution. However, ConMix can reduce the F/My ratio compared to SimCLR under different imbalance ratios, indicating its ability to mitigate the impact of the long-tailed distributions.

(3) Regardless of whether the classes are Few, Medium, or Many, ConMix improves compactness across different imbalance ratios. This indicates that samples within the same class become more compact, which is beneficial for clustering.

(4) When the imbalance ratio is 1, the dataset is balanced and differences in compactness across different classes are due to the varying difficulty of learning each class.

The above analysis empirically demonstrates that ConMix benefits long-tailed clustering. We have included the above experimental results in Appendix F of the revised version. Thank you for your suggestion!

Response to Reviewer EuZq (3/3)

Weakness 3 and Question 3: Provide detailed descriptions of the experimental setup and hyperparameter selections to improve transparency and reproducibility of the research.

A：We agree that the transparency and reproducibility of the research are very important and we have described the experimental settings in Section 5.1. Based on your suggestion, we provide more detailed information on hyper-parameter choice and training specifics here.

All experiments, unless otherwise specified, are conducted using ResNet18 with a batch size of 512 for 1000 epochs. As described in Section 5.1.2, we set the first convolution layer with kernel size 3×3 and stride 1, and remove the first max-pooling layer for all experiments on CIFAR-10 and CIFAR-20 due to their small image sizes. For ConMix, we adopt the stochastic gradient descent (SGD) optimizer, whose learning rate is 0.5, weight decay is 0.0001 and momentum is 0.9. We adopt the cosine decay learning rate schedule to update the learning rate by step, with 10 epochs for learning rate warmup. During the 1000-epoch training, for the first 200 epochs, we train only with SDCLR to learn meaningful raw representations for interpolation. Then we only use ConMix to train the model. The temperature $\tau$ in NT-Xent is 0.2. We adopt the data augmentation methods described in [1]. Unless otherwise specified, synthesized representation number $M$ = 100 is used in the experiments.

The above descriptions have been added to the revised paper in Section 5.1 and Appendix A.

Moreover, the code of our method has been submitted to the Supplementary Material, where these settings are already configured. We will also open-source the code after the review, ensuring transparency and reproducibility.

[1] A Simple Framework for Contrastive Learning of Visual Representation, ICML, 2020.

Above is our response. Your suggestions and opinions are very important, and we are grateful for the time and effort you have devoted to improving this work. Thank you very much!

Dear reviewer EuZq,

Thanks again for your time and efforts in reviewing this paper and the valuable comments on improving its quality. As the reviewer-author discussion deadline approaches, we hope to hear your feedback about our response. If you have further concerns, we are happy to provide more explanations. Thank you very much!

Regards from the authors.

Dear Reviewer EuZq,

We are very grateful for your valuable and constructive review comments. Given that the deadline for revising the paper is approaching, with less than two days remaining, we hope to learn whether our previous responses have met your satisfaction and if you have any additional questions or suggestions. If you have any new questions or suggestions, we will certainly provide a prompt response and make the corresponding revisions to the paper to improve it. We look forward to your reply with great anticipation. We would be very grateful for your reply amidst your busy schedule. Thank you for the time and effort you have invested in our work.

Regards from the authors.

Dear Reviewer EuZq,

Thank you for the time and effort you have put into this work. The deadline for revising the paper is less than 12 hours away. Could you please take a few minutes to read our response? We look forward to knowing whether our response has addressed your concerns. If you have further concerns, we are happy to provide more explanations. Thank you very much!

Regards from the authors.

Dear Reviewer EuZq,

As the Reviewer-Author discussion phase is drawing to a close, we kindly ask you to review our revisions and responses once more and reconsider your rating. All the other reviewers' concerns have been resolved, and they all gave this paper a positive score. We eagerly anticipate your feedback. Thank you.

Best regards,

The Authors

Response to Reviewer z1DH (1/3)

Thank you for your constructive comments. We have carefully considered your review and provided the response as below.

Weakness 1: The representation synthesis part is supposed to be represented more intuitively, which may be a little confusing at first reading.

A: Sorry for confusing you at first reading. In simple terms, we assign a tag within $[1, M]$ to each input within every batch, where $M$ is the number of synthesized representations. Then, representations from the same network branch with the same tag are averaged to form new representations in the manner of Eq. (3). The generation of tags follows a uniform distribution with equal probabilities $\frac{1}{M}$.

We reviewed the original paper and made the following revisions to alleviate the readers' confusion.

(1) We modified Line 208-209 from "In SimCLR framework...augmented twice" to:

"In SimCLR framework, each input $x_i$ is data-augmented twice, and the two augmented versions are fed into two different network branches in SimCLR. Given $N$ inputs, the network will output $2N$ representations {$v_1, v_2, ...v_{2N}$}."

(2) We modified Line 216-218 from "We randomly select...with equal contributions" to:

"In each batch, we randomly assign tags within $[1, M]$ to original representations from the same network branch {$v_1, v_2, ...v_N$}. The generation of tags follows a uniform distribution with equal probabilities $\frac{1}{M}$ and original representations with the same tag are used to synthesize one particular representation in the manner of Eq. (3)."

We hope these revisions can improve the readability of the paper. Thank you.

Question 1: The result of pairwise ConMix shown in Table3 on CIFAR-10 is better than the result of ConMix with M=500 presented in Figure 2. Is this reasonable? In my understanding, the former is equivalent to ConMix with a larger M on CIFAR-10.

A: The mentioned results in Table 3 and Figure 2 are two different experiments with different settings. Specifically, pairwise ConMix in Table 3 is used to validate the effectiveness of the multi-sampling strategy compared with pairwise sampling. While the experiment with $M$=500 in Figure 2 is to evaluate the impact of different $M$.

The significant performance drops in Figure 2 when $M$=500 might confuse you. But the shown results are actually reasonable. The reason lies in the batch size being 512. So when synthesizing $M$=500 representations within each batch, the multi-sampling strategy in ConMix almost fails to work effectively. But pairwise ConMix synthesizes 256 (half of 512) new representations and still has good performance.

We have emphasized in Appendix B that the batch size is 512, to facilitate better understanding. We have made the following revision:

"Due to the batch size of 512, when $M$ is set to 500, the multi-sample approach in ConMix diminishes considerably, leading to a drop in experimental performance."

Dear reviewer z1DH,

Thanks again for your time and efforts in reviewing this paper and the valuable comments on improving its quality. As the reviewer-author discussion deadline approaches, we hope to hear your feedback about our response. If you have further concerns, we are happy to provide more explanations. Thank you very much!

Regards from the authors.

Dear Reviewer z1DH,

We are very grateful for your valuable and constructive review comments. Given that the deadline for revising the paper is approaching, with less than two days remaining, we hope to learn whether our previous responses have met your satisfaction and if you have any additional questions or suggestions. If you have any new questions or suggestions, we will certainly provide a prompt response and make the corresponding revisions to the paper to improve it. We look forward to your reply with great anticipation. We would be very grateful for your reply amidst your busy schedule. Thank you for the time and effort you have invested in our work.

Regards from the authors.