You Can Trust Your Clustering Model: A Parameter-free Self-Boosting Plug-in for Deep Clustering

We sincerely thank the reviewer for their constructive comments and thoughtful suggestions. We appreciate the recognition of the clarity and simplicity of our method. Regarding the main concerns, we provide detailed responses as follows:

N1&Q1-1. Relation to existing works in noisy label learning with local consistency-based selection.

Thanks for the reviewer's comments. Sample selection is a fundamental and important problem, especially in unsupervised learning and deep clustering. Related work can not diminish the novelty of our method, which introduces clear innovations and provides advantages beyond existing methods designed for the domain of noisy label learning.

MOIT[a1] detects noisy labels by measuring the disagreement between a sample’s original label and the class distribution of its k-nearest neighbors in the global feature space. And SSR[a2] selects clean samples based on label–neighborhood consistency and uses a separate classifier to relabel samples with high confidence. They all need do $k$-NN search in the whole dataset with a selected fixed k, and may have some hyperparameters like consistency and confidence thresholds. However, our method is a parameter-free sample selection method and applies local adaptive k-nn within the mini-batch instead of the whole dataset. To provide a fair comparison, we re-implemented MOIT and SSR sample selection strategies and applied them on top of ProPos, replacing our selection module while keeping the rest of the training unchanged. Notably, SSR combines both $k$-NN filtering and confidence thresholding, which may not generalize to all baseline methods (e.g., those without a classifier head). Therefore, we only adopted its $k$-NN-based filtering component for compatibility.

To further ensure fairness with SSR, we conducted additional CIFAR10 and ImageNet-Dogs experiments on the CDC baseline by replacing our sample selection with full SSR filtering with confidence thresholding:

Moreover, we compared their full pipelines with ours (including both filtering and loss function) on CIFAR10 and observed that the overall performance gain from MOIT and SSR remains consistently lower than ours, even not better than ProPos. Additionally, our method achieves better results with shorter training time, as shown below:

While both MOIT and SSR improve over the baseline, their performance is consistently inferior to our method. We believe this is because our framework is jointly optimized with the selection strategy and a discriminative loss, which is not the case for methods like MOIT or SSR. When their selection is applied using their own loss to train the model, performance does not improve. In addition, global $k$-NN filtering, as used in MOIT and SSR, introduces extra computational cost, while our method remains efficient and scalable without additional complexity or hyperparameters.
Following your suggestion, we will refer to the related works and add the results and analysis in the discussion part of our final paper.

N2&Q1-2. How the method scales with the number of clusters?

Thank you for the valuable comment on scalability. As shown in the main paper (Table 2), our method consistently outperforms baselines on datasets with 10–20 categories, which already demonstrates its strong performance in small- to medium-scale clustering scenarios.

To further demonstrate the scalability of our method to datasets with a large number of categories, we additionally report results on CIFAR100, Tiny-ImageNet (200 classes, more results are in Appendix B), and ImageNet-1K (1000 classes). As shown below, our method consistently improves over the ProPos baseline across all datasets:

These results confirm that our method is scalable and remains effective even when the number of clusters increases from tens to hundreds or thousands. Importantly, our method is theoretically agnostic to the number of classes. What fundamentally limits performance is not the number of categories itself, but the memory constraints that restrict batch size. When batch size is small relative to the number of classes, the chance of finding same-class neighbors per sample decreases. By introducing memory bank, our method compensates for this limitation without relying on strong alteration or architectural changes. And we will include these experiments in our final version.

N3. Additional computational complexity for k-means.

We appreciate the reviewer’s concern regarding the computational overhead introduced by $k$-means clustering. It is important to clarify that $k$-means is not a contribution of our method but rather a commonly used approach for obtaining pseudo-labels in representation-based deep clustering models. Moreover, our method is flexible and does not rely strictly on $k$-means: for clustering-head based models, we can replace $k$-means with a direct SoftMax classification layer to generate pseudo-labels.

As demonstrated in Appendix C, Table 9, for the clustering-head baseline CDC, incorporating our method yields consistent improvements regardless of whether $k$-means or SoftMax is used for label assignment.

Q2. The tradeoff is not always optimal in some sense.

Thank you for pointing this out. We agree that the term "optimal" may overstate the result without theoretical or exhaustive empirical justification. We will revise the sentence in the paper to avoid this claim and now state that our adaptive $k$-NN sample selection achieves a favorable trade-off between the quality and quantity of self-supervision signals, as demonstrated by consistent performance improvements in our experiments.

Thank you for the clarifications and the additional results. I am happy with the response. I can see that the method shows consistent improvement in relation to the ones I referred to and therefore i am raising my score

Thank you for the reviewer’s constructive and thoughtful feedback. Below, we address each of your concerns one by one:

About incremental performance gains in deep clustering.

We would like to emphasize that DCBoost achieves consistent and significant performance improvements across a wide range of strong baselines, despite already high starting points. This indicates that our method is capable of further enhancing state-of-the-art results, which is non-trivial in the unsupervised clustering domain.Importantly, under strictly unsupervised conditions, our method establishes new state-of-the-art performance across multiple standard benchmarks without using any true labels. In this context, the observed gains are both meaningful and substantial.

W1&Q1. The local neighborhood may become less trustworthy under class imbalance or label noise.

Thank you for the insightful comment regarding the potential unreliability of local neighborhood consistency under class imbalance. But we would like to clarify that in our unsupervised clustering setting, the model does not rely on ground-truth labels, and thus is not subject to conventional label noise originating from human annotation errors. Instead, the labels are generated via self-supervised learning or clustering, which may contain inherent uncertainty, but this differs fundamentally from traditional noisy label scenarios. Therefore, while the neighborhood consistency may be affected by imperfect pseudo-labels, the concern of external label noise corrupting local structure is not applicable here. 

Regarding class imbalance, we have added a qualitative analysis of the local $k$-NN neighborhood structure on long-tailed dataset CIFAR10-LT and CIFAR20-LT, where the number of training samples per class follows an exponential decay controlled by a ratio between the number of samples in the most frequent class and the least frequent class. For instance, with ratio=10, the head class contains 5000 samples while the tail class has only 500. As expected, local consistency does degrade slightly compared to the balanced case due to skewed class distributions. However, it still remains significantly more reliable than global structural signals, making it still a valuable cue for identifying high-confidence samples. Specifically, our results show that $k$-NN accuracy remain much higher compared to the current clustering ACC and better than global features, even in long-tailed scenarios: 

To further validate the robustness of our method, we conducted additional experiments on CIFAR10-LT and CIFAR20-LT. The following results demonstrate that our method consistently improves clustering performance over strong baselines:

These results demonstrate that our $k$-NN-based selection strategy is effective and robust, even when local consistency may be affected by class imbalance. Although we do not incorporate any specific mechanisms to address non-uniform data distributions scenarios, our method still yields consistent improvements across different datasets and baselines.

W2&Q2. Generality on the hierarchical clustering or density-based clustering model.

While our method was not originally tailored for hierarchical or density-based clustering—since flat clustering remains the mainstream in deep clustering—we recognize the importance of generalizability across paradigms. To this end, we have conducted additional experiments on CIFAR10 using a hierarchical clustering model. Although DCBoost does not leverage the hierarchical structure explicitly, we only retained the pre-hierarchical representation part of the model and integrated it into our own training framework. Based on these learned representations, we applied our training process. Even without utilizing the full hierarchical pipeline, DCBoost consistently improved the clustering performance, outperforming both the original representation and the full hierarchical clustering baseline:

Extending DCBoost to density-based clustering methods (such as DBSCAN) presents additional challenges. These methods are non-parametric and do not assume a fixed number of clusters, making it difficult to align with DCBoost’s current reliance on pseudo-labels. However, adapting DCBoost to such methods remains a promising direction for future work.

Q3. Insights into real-world scenarios.

We thank the reviewer for highlighting the importance of real-world relevance. To this end, we evaluated our method on PathMNIST, a medical imaging dataset derived from colon pathology slides, involving the classification of different tissue types. Our method shows significant improvements over the baseline ProPos:

Limitation in estimating class counts.

We would like to point out that the vast majority of existing deep clustering methods assume the number of clusters K as a predefined input. Similarly, our method also operates under this common assumption. However, we included over-clustering and under-clustering experiments in Appendix B, Table 8 (e.g., CIFAR-20 with cluster numbers set to 10, 20, 30, 40, 50) that our method remains robust even when the specified cluster number deviates from the true number of classes. In particular, DCBoost consistently outperforms the baseline ProPos across all settings. These results suggest that, although our method does not explicitly estimate the number of clusters, it remains more robust to such inaccuracies compared to baseline methods. We agree that incorporating mechanisms to dynamically estimate or adapt to unknown class counts is a valuable direction and consider this an interesting avenue for future work.

[1] Learning from Sample Stability for Deep Clustering, ICML, 2025.

[2] Contrastive Hierarchical Clustering, ECML-PKDD, 2023.

We sincerely thank the reviewer for the careful evaluation and constructive feedback on our work. Below, we provide detailed responses to each of the key concerns raised.

W1. More self-labelling paradigm.

Thanks for your insightful comment. Our paper has already included comparisons with CoNR, a more recent and relevant representative of this paradigm. Follow the reviewer's suggestion, we additionally implemented self-labeling pipelines based on DeepCluster[1] and SCAN[2] on the ProPos baseline. Specifically, for SCAN, we used pretrained ProPos representations and performed the second and third training phases of SCAN; for DeepCluster, we attached a randomly initialized clustering head and supervised the model using self-generated pseudo-labels. 

The results on CIFAR10 (shown below) indicate that these self-labeling methods do not improve performance over the ProPos baseline. In contrast, our method consistently yields significant gains. This suggests that existing self-labeling paradigms may not always transfer effectively to pretrained models.

One potential reason why DeepCluster and SCAN underperform is their reliance on joint training with a randomly initialized clustering head. Prior work CDC[3] has shown that attaching randomly initialized heads can destabilize pre-trained representations. Therefore, while these methods work well in end-to-end pipelines with consistent optimization objectives, they may degrade performance when grafted onto pretrained models without adaptation. By contrast, our method preserves the pretrained model and only introduces a lightweight asymmetric predictor, which minimizes disruption and maintains training stability. This design enables effective improvement on strong pretrained features without architectural modification.

We hope this clarifies that off-the-shelf application of clustering-head based self-labeling methods is not universally effective, and underscores the novelty and necessity of our method. Full results and analysis will be included in the discussion part of main paper in the final version.

W2. About training cost of adaptive k-NN filtering.

Our adaptive $k$-NN filtering introduces negligible computational overhead. Specifically, the top‑$m$ neighbors are retrieved only once per batch, and scores for all candidate $k$ (from 1 to $m$) are computed based on this single result. The optimal-$k$ selection relies on simple, vectorized label consistency checks over the retrieved indices (see function $find optimal-k-balanced$ in code). This process has a time complexity of $O(B·m)$, where $B$ is the batch size and $m$ is the maximum number of neighbors considered. Notably, this step is independent of the total dataset size $N$. Therefore, our method avoids the inefficiencies typically associated with exhaustive sweeping over multiple k values and remains scalable to large datasets.

To validate this, we have conducted experiments on CIFAR-10 with settings: using no filtering ($k$ = 0), fixed $k=25$, and adaptive selection with $m=25,50,75$. As shown below, training time remains nearly identical across all variants, but get great performance improvements:

The results demonstrate that our adaptive filtering strategy is both efficient and effective, providing performance benefits without introducing additional training cost.

W3. How does the positive loss help with mitigating imbalanced clustering?

The positive loss is designed to promote intra-class compactness among high-confidence samples. However, some classes may dominate this loss simply because they have more confident samples. To address this, we introduce a per-class normalization weight $w_c$, which ensures that each class contributes equally to the overall loss, regardless of its size. Specifically, $w_c$ can be seen as the inverse of the square of the L2 norms of the aggregated features from class c. When a class has more high-confidence samples (thus a larger summed feature norm), its influence on the loss is automatically reduced. In this way, the loss naturally compensates for the imbalance in the number of high-confidence samples across classes.

W4-1. How to compute cluster prototypes?

As described in the paper, cluster prototypes are computed by averaging the features of high-confidence samples within each pseudo-label class in current training batch, followed by L2 normalization. The number of prototypes $c_B$ in a batch corresponds to the number of classes with confident samples present in that batch.

W4-2. What will happen to the clusters that have no sample in a mini-batch?

If a cluster has no sample in a mini-batch when the class distribution is imbalanced, we simply exclude that class from the current loss computation. This design does not affect the stability of training, since the model learns from the cumulative gradients over many batches rather than requiring every batch to contain all classes.

Importantly, we additionally validated the method's robustness on long-tailed benchmarks CIFAR10-LT and CIFAR20-LT, where class sizes follow an exponential decay (e.g., with ratio=10, head has 5000 samples, tail only 500).Our method still consistently outperforms the baselines. This demonstrates that the occasional absence of some classes in individual batches does not lead to significant degradation in overall clustering performance.

We added per-class accuracy analysis on CIFAR20-LT, where C0 is the most frequent (head) class and C19 is the rarest (tail) class. We compared the class-wise accuracy before and after integrating our method:

Our method brings improvements in the majority of classes; importantly, it is not restricted to head classes only. Several tail classes, which are more likely to be absent from some mini-batches, still get noticeable performance gains. Beyond class imbalance, we acknowledge the challenges of scaling to many clusters. Our primary focus is on moderate-scale settings where batch sizes can cover most classes. In response to the reviewer’s concern, we have newly added experiments in a setting where the number of clusters exceeds the batch size. Although our base selection method may face limitations due to batch size constraints, it can still be extended to these challenging settings by incorporating a memory bank to store and reuse features across batches, and adopting a fixed $k$-NN retrieval method to stabilize neighbor selection. We add experiments on CIFAR100, where we intentionally reduce the batch size to 64 (i.e., fewer than the number of classes) to simulate this, and size 256 on ImageNet-1K, a large-scale dataset with 1,000 categories. Because full training of ImageNet-1K is resource-intensive, we used ProPos to train for 30 epochs and added our method for another 10 epochs.

Our method still improves clustering performance across all metrics on both datasets. It is also indicated that using high-quality local information to guide the optimization of global structure remains effective, even when not all classes are present in each mini-batch. We will include the results and analysis in the final version of our paper.

Q1. Are the improved features more generalizable to new data domains or new tasks?

To evaluate whether our improved representations are more generalizable, we follow standard linear evaluation protocols. We freeze the learned features and train a linear classifier on top of them for downstream classification. Specifically, we conduct two experiments:

(1) In-domain generalization: Linear classification on CIFAR10 using features learned on training set of CIFAR10. (2) Cross-domain generalization: Linear classification on STL10 using features learned on CIFAR10.

We report Top-1 accuracy and Top-5 accuracy as performance metrics.

As shown below, our method consistently improves performance over the baseline in both settings, suggesting that the enhanced features are not only more discriminative within the same domain, but also more transferable to related domains.

[3]Towards Calibrated Deep Clustering Network, ICLR, 2025.

[4]Learning from Sample Stability for Deep Clustering, ICML, 2025.

We sincerely thank the reviewer for their valuable comments and constructive feedback. We would like to address each of your concerns in detail, and clarify points that may have caused confusion.

W1&Q1. The description of the dynamic selection mechanism.

Thanks the reviewer for pointing out the need for more clarity regarding our dynamic selection mechanism. We clarify the implementation details as follows:

For BYOL-like architecture (BYOL, CoNR, ProPos): We combine outputs from both the online and target networks.  The similarity matrix $S$ is computed as: $S_{ij}=\cos\left(z^o_i, z^t_j\right)$ to search the $k$-NN. This hybrid approach balances the stability of the target network, and the real-time adaptability of the online network. By using both, the neighborhood consistency reflects both stable semantic structure and recent representation shifts.

For other architectures(e.g., CC, SCAN, CDC): We only use the target network's features for $k$-NN retrieval. ($S_{ij}=\cos\left(z^t_i, z^t_j\right)$). This is because the predictor followed by the online network is randomly initialized, and may introduce noise or unstable representations early in training. To ensure reliable consistency estimation, we rely solely on the more stable target encoder output in such cases.  We will revise the manuscript accordingly to clearly describe these design choices and implementation details.

W2. Experimental setup about batch size and total training epochs.

In fact, we have already provided comprehensive information regarding the experimental setup in Appendix A, including backbone architectures, dataset sizes, training durations, batch sizes, and other relevant hyperparameters. For clarity, we would like to reiterate here that our method uses a batch size of 256 and is trained for 200 epochs based on the existing models. We believe this level of detail is sufficient to ensure the reproducibility and scalability of our experiments. Besides, we have submitted our code in the supplementary file, which makes the reproducibility easy.

W3. Whether the proposed method modifies existing models.

Our proposed method is applied directly on the existing models without modifying their backbone networks or any other components. As detailed in Appendix A, for models already based on BYOL-like architecture such as BYOL, CoNR, and ProPos, neither the backbone nor additional modules are changed or added during our enhancement. For other models like CC,SCAN and CDC, we duplicate their clustering network to form online and target networks, and add only a lightweight MLP predictor after the online network. Still, the backbone networks and original modules remain unaltered. 

Q2. Is dynamic retrieval necessary for every batch?

Yes, our method performs  $k$-NN-based clean sample selection at each training batch. That means, if the batch size is $B$, we compute $k$-NN using only $B$ samples. By default, we perform within-batch retrieval, which is efficient and sufficient for all the results reported in the main paper. Thus, no historical representations are used in these main experiments. 

Instead, to handle rare edge cases (e.g., when a batch lacks sufficient same-class samples), we introduce a memory bank of recent features from the target network. This bank is continuously updated with the most recent representations at each training step. This mechanism is only used in the extended experiment in Appendix B, and does not involve maintaining long-term historical representations.

Q3. The dynamic retrieval parameter m setting.

We clarify that evaluating multiple candidate values of $k$ (from 1 to $m$) does not require performing $k$-NN retrieval multiple times. Instead, we compute the $m$-nearest neighbors only once per batch (e.g., $m=50$), and all scores under different $k$ are evaluated based on this single retrieval result. The function $find optimal-k-balanced$ (see code) performs label consistency checks and computes an favorable $k$ using efficient matrix operations. Therefore, if the number of samples within the batch is $B$, the overall computational complexity of dynamic retrieval remains $O(B·m)$ per batch, so the whole computational complexity of training is not significantly affected by the value of $m$. 

We have also added a new experiment on CIFAR10 varied $m$ (e.g., $m$=25, 50, 75), replaced the dynamic strategy with a fixed $k$=25, and even removed the $k$-NN-based filtering altogether (i.e., $k=0$, using all samples) as follows. All these variants showed negligible difference in runtime. This confirms that our strategy introduces no observable computational overhead, while effectively improving sample selection.

Q4. How to ensure the selection of high-confidence samples from different classes?

To demonstrate that our method selects high-confidence samples across all classes (rather than biasing toward a few), we have shown how clustering precision and recall change across different classes before and after applying our method on CIFAR-10.

Precision(%):

Our method improves the precision in 9 classes and recall in all 10 classes. This confirms that our selection strategy does not favor specific classes, but instead works consistently across diverse semantic groups. The base model already yields relatively high clustering accuracy, and the local consistency sample selection is effective at identifying confident samples within each semantic region of the feature space. It naturally achieves balanced coverage across classes. The results also indicate that our method helps refine the global feature space in a class-balanced manner. We will include this analysis and the accompanying tables in the final version of the paper to further clarify this point.

Q5. Illustrating the trade-off through visualization.

Thank you for the constructive suggestion. We chose $k$ as the x-axis because our method adaptively selects the proper $k$ based on label consistency, and the geometric interpretation provides an intuitive way to compare different $k$ values. We agree that using precision as the x-axis could offer another interesting perspective to visualize the trade-off between quality and quantity, and we appreciate the suggestion and will add the figure in our final version.

Q6. Comparison with existing high-quality sample selection methods.

We have added new experiments comparing our method with several existing sample selection methods SSR[1],MOIT[2] and SCAN[3]. SSR and MOIT are noise-robust sample filtering methods and can directly replace our filtering mechanism, and SCAN is another sample selection in deep clustering. The evaluation covers clean sample accuracy, the number of true and false clean samples selected, and training time. Note that SCAN uses a confidence threshold to filter samples and is not directly applicable to the ProPos-like representation-based method. For a fair comparison, we treat SCAN as another baseline and evaluate both its original method and our sample selection method. As shown, our method is more conservative, aiming to minimize false positives. It achieves the lowest number of incorrectly selected clean samples while maintaining competitive accuracy.

The computational overhead of sample selection alone is difficult to isolate; thus, we report end-to-end training time, including selection and model learning. We directly compare the third stage of SCAN and our method, and report full SCAN runtime (stage 2 + 3) in Appendix C, Table 11. Due to limited resources, SCAN experiments were run on an RTX 3090, while other methods were evaluated on V100.

SSR and MOIT require full-dataset evaluation for sample selection, which increases the runtime. In contrast, our method leverages a $k$-NN-based filter within each training batch, enabling more efficient selection. Although SSR and MOIT improve performance when plugged into our training framework, such gains depend on our model design and loss functions. Used independently, these methods do not yield comparable performance, emphasizing the importance of integrating selection and training cohesively. Compared with SCAN, our method achieves a comparable efficiency but higher ACC.

[1] Multi-objective interpolation training for robustness to label noise. CVPR, 2021.

[2] Ssr: An efficient and robust framework for learning with unknown label noise. BMVC, 2021.

[3] Scan: Learning to classify images without labels, ECCV, 2020.