Learning from Sample Stability for Deep Clustering

On main results (Claims And Evidence)

We compared four state-of-the-art (SOTA), including IDFD, ProPos, CoNR, DMICC, the baseline BYOL by reproducing them, with the same experimental setup as ours, including backbone, batch size, number of epochs, etc. These experimental settings can significantly affect the model's performance and lead to unfair comparison if non-unified. For other representative methods, we directly adopted the results in ProPos. Below, we provide further clarification.

About the compared SOTA: some existing methods surpass the performance of the SOTA we compared, as shown on the website you provided. However, to ensure a fair comparison, the selected SOTA are all methods based on representation learning for deep clustering. These methods focus on learning discriminative representations and then apply K-means to get the final clustering assignments, consistent with the approach used in LFSS. Many of the higher-performing methods available on that website rely on leveraging the substantial knowledge embedded in pre-trained large foundation models. Given that these methods do not learn representation from scratch and benefit significantly from their pre-trained foundations, comparing against them would be inherently unfair. Thus, we chose not to include these methods in our comparison, ensuring our comparisons remain equitable.

About the reported results of ProPos: Due to the varying experimental settings of the SOTA methods, we adopted a unified setup for fair comparison. We used ResNet-18 instead of ProPos's original ResNet-34 and trained on a single GPU, resulting in slightly lower performance than their original results.

Why not display all from results from ProPos：In fact, ProPos directly cited results from various papers. While many studies adopt this practice, it can lead to unfair experimental comparisons. For instance, a method trained for 3000 epochs may claim superior performance over one trained for only 300 epochs. Given limited computing resources, we believe it is essential to at least reproduce the most relevant and competitive SOTA methods to validate performance fairly.

All the reproduced results we report are credible. We clearly indicate in the paper which methods are cited and which are reproduced, along with the experimental settings used for reproduction. We sincerely hope that these explanations will help you recognize the validity of our approach.

Reducing unstable points (Experimental Analyses)

We focused on the top 10% most unstable samples at the 200th epoch on CIFAR-10 and CIFAR-20, using the stability of the sample at exactly the 10th percentile as the threshold for determining instability. In subsequent epochs of training, a significant proportion of these samples surpassed this threshold and became stable. This indicates that our method can effectively reduce the number of unstable samples. The results are at https://anonymous.4open.science/r/ICML25-0E61/4.1.png.

Essential References Not Discussed

Both papers exclude unreliable pseudo-labels during training, due to inconsistent predictions across consecutive epochs and across multiple models respectively. In contrast, LFSS leverages sample stability at the representation level. It allows for training representations from scratch and can be embedded into multiple frameworks. We clarify that our contribution lies not only in the utilization of sample stability at the represention level or good experimental performance, but more importantly, in uncovering the relationship among sample stability, clustering prediction and network memorization. Meanwhile, the two papers you provided are relevant to our work, and we will cite and discuss them in the final version.

About hyperparameters （Weakness）

Despite multiple parameters, LFSS is stable in parameter choice. Please refer to the first answer to Reviewer 9ozz for details. Thank you.

On excluded points (Q1)

Only cluster loss, for more accurate centers. For other losses, we do not exclude them, allowing more samples to contribute to training.

On the order of executing K-means (Q2)

The approach of excluding unstable samples first and then applying K-means is actually a similar process to performing K-means first and then excluding unstable samples. We trained LFSS with two methods on CIFAR-10 and recorded the clustering accuracy of the non-excluded samples at different epochs, to reflect the precision of the cluster centers obtained by two methods. The results are at https://anonymous.4open.science/r/ICML25-0E61/4.2.png. Two results are roughly similar, suggesting the two methods can achieve comparable outcomes.

On Fig.4 (Q3)

Sorry for the confusion. To clarify, we presented results at the 600th epochs as an example mid-training. Similar effects are seen at final models in https://anonymous.4open.science/r/ICML25-0E61/4.3.png.

Thank you for pointing out the typo and we will revise it.

Thank you for your appreciation of this work. We highly value the insightful comments you have provided, and below we offer our responses.

　Essential references not discussed

Thank you for providing the two recent related articles. After careful reading, we believe that these two papers are highly relevant to our work and should be cited in the introduction and related work sections in our work. The first article discusses the approach of solving deep clustering from a prior perspective, which is relevant to our discussion on constructing different supervisory signals based on prior knowledge. The second article innovatively proposes an unsupervised hard sample mining method and employs external user interaction to enhance clustering performance. We enhance clustering performance by utilizing sample stability, which can also be regarded as an indirect approach to hard sample mining. We will cite the two relevant paper in the final version.

　The application of K-means is limited when it comes to large datasets (Weakness)

Indeed, applying K-means to obtain clustering assignments at every epoch can be very time-consuming, especially on large datasets. However, the cluster-level loss for LFSS is only enabled after η epochs warmup, which means that the network can produce meaningful embeddings and does not necessarily require cluster assignments to be updated in every epoch. We can reuse the clustering assignments obtained from a single K-means run across multiple epochs. We evaluate model performance under different cluster assignment update frequencies on CIFAR-10 in https://anonymous.4open.science/r/ICML25-0E61/3.1.png. Although the strategy of updating at every epoch achieves the best performance, the performance drop is marginal when the epoch interval is set to 10, 50, or 100. This allows us to significantly reduce training time. Therefore, we can achieve near-optimal performance while saving time by reducing the execution frequency of K-means.

Hyperlink to observations (Suggestion)

We will add hyperlinks in Introduction section that point to specific observations to enhance readability in the final version.

Comparison with confidence-based sample selection (Question 1)

The distinction between these two approaches lies in their application scenarios. The confidence-based sample selection method typically requires pre-training a representation model first, then training the clustering head with high-confidence pseudo labels to improve performance. This approach's success hinges on the model's ability to generate good representations and pseudo labels. In contrast, our method is a training-from-scratch approach that can be used for representation model training without the need for pre-training.

We apply the self-labeling method in SCAN [1] on the final model of ours on CIFAR-10, using a threshold of 0.99 to filter out low-confidence pseudo-labels. Some of the low-confidence samples excluded by this method overlap with the unstable samples we selected from the final model. We count the number of these samples and provided their accuracy under K-means clustering as below:

The accuracy of the selected unstable samples is lower, while the accuracy of the low-confidence samples is close to the overall accuracy of the global dataset. This demonstrates that the sample stability-based method can identify misclustered samples more effectively than confidence-based approaches.

Question on self-distillation with noise strategy (Question 2)

self-distillation with noise strategy is not a novel contribution in this work and it is a commmonly used trick. This reparameterization trick is oriented in VAE [2] and we should cite it in our paper.

Minor in Fig.3

We sincerely thank you for pointing out this typo. The framework of LFSS is built upon BYOL, where the predictor head is connected after the online network. We will revise this in the final version.

Heartfelt thanks for your efforts in this reivew.

[1] Scan: Learning to classify images without labels, ECCV, 2020.

[2] Auto-encoding variational bayes, ICLR, 2014.

Thank you for your important questions. Below is our response:

Hyperparameter Sensitivity (Weakness 1)

Actually, we provided an analysis of the sensitivity of four hyperparameters on CIFAR-10 in Appendix E. Here, we further provide a related analysis on three datasets: CIFAR-10, CIFAR-20, and ImageNet-10, along with a guideline to assist readers in applying the method to other datasets. The results are at https://anonymous.4open.science/r/ICML25-0E61/2.1.png for unstable ratio δ, noise intensity σ, balance parameter λ and https://anonymous.4open.science/r/ICML25-0E61/2.2.png for warmup epoch number η.

Unstable ratio δ. Typically, a smaller δ is more effective, while a larger δ may exclude many meaningful representations from contributing to the clustering centers, thereby reducing model performance. In LFSS, we fix δ at 0.1.

Noise intensity σ. Adding noise improves the model's robustness. On different datasets, varying the noise level results in performance differences of about 5%, indicating that both excessively high and low noise intensity do not harm model training. In our experiments, we set σ to 0.01 or 0.001.

Balance parameter λ. The parameter λ balances the different terms in Eq. (10), determining the contribution of instance-level and cluster-level losses to model training. An overly small λ can lead to performance degradation due to the insufficient contribution of these losses. In LFSS, we fix λ at 0.1.

Warmup epoch number η. We set η to 0, 50, 200, 500, and 800 to evaluate the impact of introducing cluster-level loss at different training stages. Early introduction (η = 0 or 50) reduces clustering performance due to generating cluster centers of low quality, while later introduction, when representative cluster centers are well established, achieves better final performance. In our experiments, we set η to 200 or 500.

LFSS on imbalance datasets (Question 1)

To evaluate the robustness of LFSS on imbalanced datasets, we conducted experiments on CIFAR-10, CIFAR-20, and STL-10, using an imbalance ratio of 10. We adoped ResNet-18 with a batch size of 256 and trained for 1000 epoch. Besides the three metrics in the paper, we also used CAA (class-averaged accuracy) for evaluating performance on imbalanced datasets. The results are at https://anonymous.4open.science/r/ICML25-0E61/2.3.png. Compared with state-of-the-art methods, e.g., IDFD, CoNR, ProPos, and DMICC, our method achieves the best performance on all three datasets on all four metrics with the maximum improvement over the second-best method is 7% on ACC. Although none of the methods specifically address the handling of class imbalance, our approach demonstrates greater robustness on imbalanced datasets compared to other methods.

LFSS on non-image data (Question 1)

Consistent with our comparison methods, we use image datasets to validate the effectiveness of our method. But our method is not limited to image datasets. As suggested, we further evaluated our method on two text datasets GoogleNews-T and GoogleNews-S. We replaced the ResNet-18 in LFSS with an MLP. We choose distilbert as the backbone to extract features from original texts. We compared LFSS with text clustering methods such as BoW, TF-IDF and HAC-SD. We use the same experimental setup as ours to reproduce IDFD and ProPos. The results are at https://anonymous.4open.science/r/ICML25-0E61/2.4.png. LFSS shows improvements of about 2-3% on all metrics compared with the second best. Despite not specifically considering the data characteristics and design schemes for text clustering, our method still managed to deliver highly competitive experimental performance. This underscores the superiority of our approach and its robustness across different types of data.

Computional cost (Weakness 2, Question 2)

In fact, adding a predecessor network into BYOL framework does not introduce significant computational overhead, as this network is updated by directly copying the weights from the previous epoch, without requiring gradient computation or backpropagation. We compared the runtime (minute) and memory usage (MB) for three methods based on the BYOL framework: BYOL, ProPos, and LFSS, as follows:

It can be seen that LFSS's computational resource usage is slightly higher than other methods but remains within an acceptable range, all within the same order of magnitude.

Computional complexity of LFSS (Question 3)

The computional complexity of LFSS loss is $O(N^2*d)$, where N is the batch size and d is the embedding dimension. We do not consider the computional complexity of the backbone or cluster assignment, as they are independent of our method.

We hope the above response is satisfactory, and we thank you for the time and effort you devote to this review.

Thank you for your careful review and constructive comments. Below is our response:

Changes in sample stability during training (Weakness 1)

Thank you for your valuable advice. We conducted experiments to investigate the changes in sample stability as training progresses. The experiments were performed on SimCLR, BYOL, ProPos, and IDFD frameworks, consistent with the main text. We also tested the proposed LFSS under the same experimental setup. Following the Observation 1 in the main text, we divided the samples into ten bins based on their stability, ranging from lowest to highest. On the CIFAR-10 dataset, we calculated the mean sample stability of each bin at 200, 400, 600, and 800 epochs for each methods. The results are visualized in https://anonymous.4open.science/r/ICML25-0E61/1.1.png. Generally, as training progresses, sample stability gradually increases. This phenomenon becomes more pronounced once the model training has stabilized after 400 epochs.

Also, the sample stability values differ among various methods due to their differing learning strategies. Methods like BYOL, ProPos, and LFSS, which use Exponential Moving Average (EMA) to update the target network, tend to have smoother network changes and thus exhibit overall higher sample stability compared to IDFD and SimCLR, which do not utilize EMA. However, as Observation 1 demonstrates, the relatively unstable samples in these methods still exhibit lower accuracy compared to the relatively more stable ones, confirming the applicability and robustness of our findings across methods that differ significantly in their characteristics.

Experiments on larger intervals (Weakness 2)

Thank you for your deep thoughts on this work. Following on your suggestion, we conducted experiments on CIFAR-10 and ImageNet-10 to verify whether a larger interval would lead to performance improvement. We chose intervals of the smaller values 5, 10, and the larger value 100 to ensure the thoroughness of the experiments. The results are below：

CIFAR-10:

ImageNet-10:

Advice on hard sample strategy (Weakness 3)

Since unstable samples can be considered hard samples that are difficult to correctly identify, LFSS can also be viewed as a method for improving performance in hard sample mining. Particularly, in our cluster-level loss (Eq. (8)), we exclude unstable samples (hard samples) to ensure more accurate cluster centers for contrastive learning. Without this loss, performance on various metrics for CIFAR-10 would decrease by 4-7%, as described in ablation study. We look forward to further addressing unsupervised hard sample mining from the perspective of sample stability. We are grateful for your kind advice.

Visualization on unstable and stable samples (Weakness 4)

We sincerely appreciate your suggestions for improving this work. We have visualized the distribution of embeddings for both stable and unstable samples in https://anonymous.4open.science/r/ICML25-0E61/1.2.png. We perform t-SNE on final embeddings on CIFAR-10. We mark top 10% stable samples in yellow, top 10% unstable samples in purple and other samples in green. Even though the number of stable and unstable samples selected is similar, there appear to be more purple dots (unstable ones) on the figure intuitively. This indicates that the distribution of unstable samples is more dispersed, while the distribution of stable samples is more concentrated, even resulting in many overlaps. This suggests that stable samples tend to be representative samples with similar properties and characteristics. Meanwhile, we also observe that isolated samples in the gaps between clusters often belong to the unstable ones, which highlights their atypical features.

We are very grateful for your sincere suggestions in improving our work as well as for your recognition of our effort. Thank you.