Utilization of Neighbor Information for Image Classification with Different Levels of Supervision

[W 4.3.3] Many of the baselines included in this paper are not the latest or most competitive, which further limits the impact of the comparisons presented.

While we acknowledge the possibility of missing something, we have tried to add (1) key papers from the earlier literature and (2) almost all the competitive papers from recent literature for comparison purposes. We would highly appreciate it if the anonymous reviewer could point us to the literature that we have missed for comparison purposes.

[W 4.3.1] Additionally, some key baselines, such as DeepDPM [3], are missing,

This baseline is present in Table 01 (4th row of results).

We thank the anonymous reviewer for their time and effort in producing a meaningful review, a very detailed summarization of our work, and a comprehensive note of our strengths. Firstly, we will respond to the reviews along with an updated PDF for the questions that do not require further experimentation. Next, we have a set of clarification questions about the additional experiments/comparisons requested by the reviewer.

[W 4.6] Consistency in formatting could be improved as well; for instance, Table 2 contains results with varying decimal places, with some values displayed to two decimals and others to only one.

When re-using numbers from other works, we are limited by the precision they use. We provide 2 decimal precision whenever available. The 2nd decimal place is required to assess the difference between ViT-B/16 results of TEMI, SCAN, and kMeans.

[W 4.4] The paper’s presentation could also benefit from some adjustments. Placing additional figures, particularly those referenced in the main text, after the summary section disrupts readability.

We request some clarification as to whether this comment is about referring to figures from the appendix in the main text. If not, we would be highly appreciative of specific suggestions on how to improve our presentation and readability.

[W 4.2] The selection methods for positive and negative samples also appear somewhat simplistic. Using the nearest samples as positives and the furthest as negatives might not be a robust strategy. For instance, previous works, including [1], have shown that employing “hard” negative samples (those that are more challenging to distinguish from positives) can significantly improve model performance. Given the existing research on positive and negative sample mining, there are likely many unexplored approaches that could yield interesting results. It would be beneficial to explore how the selection of positive and negative samples could be refined for this task setting to improve efficiency and effectiveness.

Please refer to our response to [W4.1]. We have given an explanation and experimental evidence as to why hard negative samples can improve performance in supervised settings (which is explored in the existing literature) but not in unsupervised or semi-supervised settings (which is explored in this paper). The UNIC’s task setting does not benefit from the existing literature on hard negatives for contrastive learning.

[W 4.1] I have several concerns regarding the technical contributions of this paper. While I agree that positive and negative neighbor mining can improve clustering methods, this approach has already been extensively explored in previous contrastive learning research, such as in [1 ICLR2021] and [2 NeurIPS 2020].

We thank the reviewers for bringing up this question. The existing work in the hard negative literature shows gains for a fully supervised image classification setting. However, these techniques fail to improve UNIC’s problem settings.

We append a MoCHi [2 NeurIPS 2020] row to Table 1 and Table 7. We also append an HCL [1 ICLR2021] row to Table 1. These results show the failure of hard negative mining to improve performance.

Then we add an “Appendix A.14 FAQ: HOW DOES UNIC’S NEIGHBOR MINING COMPARE TO HARD NEGATIVE MINING?” to explain the reasons for this failure, why hard negative mining in literature is orthogonal to our work, and why we specifically need UNIC’s neighbor mining techniques.

[W 4.6] I am curious if authors have tried various representation learning methods (such as MAE, Supervised ViT, iBot, etc) beyond these ones presented in the paper? What kind of self-supervised learning strategy can achieve the best performance? Do you have any insights on it?

We are thankful for this insightful question from the anonymous reviewer. We limited our main text to DINO representation learning methods to stay consistent with the literature. In order to answer this question, we ran additional experiments and reported the results in Table 7: UNIC Clustering Performance with Different Backbones on ImageNet-50.

Here, we present results from both ResNet and ViT backbones trained on a range of SSL techniques. In addition, we report a result from a supervised trained backbone as a best-case scenario.

The insights from these experiments are discussed in Appendix A.15 UNIC WITH OTHER BACKBONES.

W6. There are several typos, such as Line 258 “y ̂_i,y ̂_p,y ̂_p”

Thank you for pointing this out. We have corrected this.

W4. Could the author explain the selection of τ_2 for ImageNet and STL? For CUB, what is the parameter τ_2 set to?

We have updated line 292 to give τ_2 values for all datasets. We append lines 472 to 475 to validate our choices for τ_2 in retrospect. The authors have not tuned τ_2 extensively in this study. We only pick a τ_2 that is intuitively big enough to cover the necessary region of the neighborhood based on the number of classes and example images per class in the dataset.

W2. For a comprehensive clustering ablation, could the author replace the proposed positive mining strategy with a simplified one or existing one and show it in Tab. 3 to validate the effectiveness of the proposed positive mining strategy?

The authors have reported results from simpler mining strategies in Table 6 in the Appendix. The bottom row (7th row) of the table (90.80%) is obtained by our full mining strategy (which includes positive, cleaning, and negatives). The 6th row (89.04%) reports a simple strategy (which includes positives and negatives only). The 5th row (87.12%) reports the simplest strategy, which is to mine positives only.

The authors would like the input of the anonymous reviewer as to whether this ablation should be present here as a separate table or be added to the main Table 6 on the paper.

We thank the anonymous reviewer for their time and effort in producing a meaningful review. We will respond to the reviews along with an updated PDF. Firstly, we will respond to the questions that do not require further experiments. In the meantime, we will run additional experiments to answer the rest of the questions.

While the summary and strengths have pointed out most of our contributions, we would like to emphasize that our strategy of mining negatives, and their analysis of their contribution to improving the accuracy of clustering and GCD tasks (or any tasks) is novel as well.

W1. When encountered with fine-grained classes, proposed methods show significant reduction with a less powerful backbone, e.g., DINOv1. Could the authors analyze the bounds on the backbone’s feature extraction capabilities that make the proposed approach fail? For example, when the basic K-means clustering accuracy drops to 60% or 50%, the proposed UNIC will not work? – We thank the reviewer for this insightful question. We conduct a range of experiments with different backbones to answer this question and report both kMeans and UNIC results for them in Table 7: UNIC Clustering Performance with Different Backbones on ImageNet-50 and explain the results in Appendix A.15 UNIC WITH OTHER BACKBONES. UNIC’s gains over kMeans drop to very low numbers (<1%, or negative) when the feature extraction is underperforming (MoCHi, MoCoV2) where kMeans accuracy itself is low 60%s. UNIC consistently performs well for representations for which kMeans accuracy is over 65%.

W5. For fine-grained image benchmarks, the experiments are only conducted based on the CUB dataset, which cannot demonstrate the generation ability of the model. The author should conduct experiments on other fine-grained like Stanford Cars and FGVC-Aircraft. If the proposed UNIC framework can well-mind clean neighbors, it should also achieve comparable performance with state-of-the-art benchmarks as well.

The authors thank the reviewer for this question. We acknowledge that the only FGCV result on the main text was where UNIC outperformed the literature only for the DINOv2 backbone case. We have conducted additional experiments during the rebuttal period to demonstrate how and when UNIC’s GCD capabilities are generalizable across the SSB suite.

We run experiments on CUB-200, StanfordCars-196, and Aircrafts-100 datasets. We run two experiments for each dataset. First, we experiment with the DINOv1 pre trained Vit-B/16 backbone similar to earlier GCD literature. Then, we follow experiments with SimGCD [ICCV 2023] pretrarined backbone similar to recent GCD literature SPTnet [ICLR 2024]. Please refer to the table below for experimental results.

Our results show that UNIC suffers from the low-quality neighbors mined with DINOv1 backbone. However, UNIC outperforms the existing literature when it is able to utilize SimGCD pretrained backbones and achieve state-of-the-art.

It should be noted that SPTnet achieves its improvement over the previous literature by adding additional prompt parameters to the ViT backbone. UNIC achieves better performance without changing the backbone architecture (Only tuning the last block).

[Q 2.4] For the GCD ablations in Table 4, it is curious why the labeled negative neighbors are inferior to the mined ones (when comparing the results in the second and third rows).

Thank you for pointing out this. We have appended lines 415-418 to explain this scenario. Another section has been appended to the Appendix (A.13) to elaborate on the underlying issue.

[Q 2.3] Moreover, the experiments lack insightful analysis. For example, in Table 3, why the contrastive loss harms the performance of clustering;

Thank you for pointing out this. We have appended lines 409-410 to explain this scenario.

[Q 2.2] The experiments are not comprehensive. It is needed to compare with other positive/negative mining strategies directly.

The authors have reported results from simpler mining strategies in Table 6 in the Appendix. The bottom row (7th row) of the table (90.80%) is obtained by our full mining strategy (which includes positive, cleaning, and negatives). The 6th row (89.04%) reports a simple strategy (which includes positives and negatives only). The 5th row (87.12%) reports the simplest strategy, which is to mine positives only.

The authors would appreciate the anonymous reviewer feedback on whether these studies are enough or if we should compare them to something else specifically.

We thank the anonymous reviewer for their time and effort in producing a meaningful review. We will respond to the reviews along with an updated PDF. In addition, we have a set of clarification questions about the additional experiments/comparisons requested by the reviewer.

[Q 2.1] It is encouraged to clarify the novelty in the proposed UNIC, as its implementation looks very common and similar to some existing works.

The major novelty of this work is in the neighbor mining and cleaning strategies. We edit line 103 of the Introduction to emphasize this.

Re-iterating our response to [W2.1] – According to the best of our knowledge of the literature, there isn’t much work that explores improving positives for this style of clustering and GCD. Utilizing negatives (instead of randomly sampling from the whole dataset) has not been explored in any prominent work.

These ideas are explained in the Figure 02, Equation 3-4.

Other than the overall implementation, we provide strong empirical evidence to justify our design choice on filtering with respect to second-order neighborhood sizes. To this end, Figure 4, Figure [8-11]. In addition, we provide an example visualization in the Appendix Figure 13.

In addition, Appendix A.4 discusses the differences between our methodology from existing work. Furthermore, Appendix A.5 specifically answers the question about why our neighbor mining is novel.

The authors would greatly appreciate it if the anonymous reviewer could point us to which literature overlaps with our design. In that case, we will be able to provide better differentiation between our work and the literature.

[W 2.1] The main contribution in this work lies in how to mine positive and negative neighbors. However, the implementation details in the proposed UNIC are not innovative a lot. There have many related methods about improving the positive and negative candidates for both clustering and GCD field. It is unclear about the main difference and contribution proposed in UNIC. Also, it is unclear why UNIC can achieve significant improvements over prior works.

According to the best of our knowledge of the literature, there isn’t much work that explores improving positives for this style of clustering and GCD. Utilizing negatives (instead of randomly sampling from the whole dataset) has not been explored in any prominent work. We would be extremely grateful if the anonymous reviewer could point out such literature, which we could incorporate into our related work sections.

Our reasoning for why UNIC achieved significant improvements in our novel contributions – (1) positive cleaning strategy, (2) the use of negatives, and our improvements to existing techniques – (3) tuning the backbone, (4) getting rid of the DINO-like contrastive loss. We provide evidence of all these design choices individually and collectively contributing to the performance gain in Table 6 of the Appendix.

[W 1.1] My main concern is over the formulation of the learning algorithm. Particularly, I find it difficult to understand Eq 6. The entropy here is computed between two scalar values rather than a distribution. Is this standard binary cross entropy / log loss? This would be more easily understood if written out in full, in my opinion (given that this is not a multi-class entropy problem).

Thank you for pointing out the mistake. We corrected the wording to emphasize on a,b being scalars (probabilities) instead of being vectors (probability distributions). The edits are visible on lines 262 and 275.

We thank the anonymous reviewer for their time and effort in producing a meaningful review. We will respond to the reviews that do not require further experimentation first. In the meantime, we will start running the additional experiments for incorporating UNIC Neighbor mining for the GCD baseline and testing the performance of UNIC on other datasets as suggested by the reviewer.

We want to emphasize two minor suggestions to the summary.

[S1.1] One of the major improvements of our method over existing work (which mostly uses simple nearest neighbor search for mining positives) is that we have a refining step that can greatly reduce the noise level of the positive neighbors. We emphasize this in Figure 2, Equation 4. In addition, further justification is given in Appendix A.3 and Figure 13.

[S1.2] This is an attempt to clarify the utility of negative neighbors. Consider one reference image. Intuitively, we penalize the model for predicting the wrong clusters/classes. Existing work assumes every image except the reference image is of a different cluster than it (i.e. all other images are negatives). However, this assumption gives rise to many false negatives because there ought to be images of the same cluster as the reference. We propose our own neighbor mining algorithm to reduce these false negative neighbors. This is emphasized in Figure 2, Equation 3, Equation 6 (L_neg term). The effectiveness is shown in Figure 8 (mid), and Table 6.

[W 1.4]I believe the presentation of this paper could be improved. e,g: Table 2 is small and difficult to read; Table 6 in uncentered, etc.

Thank you for the suggestions. We increased the font size of Table 2 and centered Table 6. Please refer to the updated PDF.

[W 1.2] As mentioned in "Strengths", the method is simple and can be easily extended. Did the authors consider adding the positive/negative mining strategy directly to existing methods (e.g it might fit naturally on top of the GCD baseline).

The authors thank the reviewer for this question, which enables us to justify some of our novelty – which goes beyond cleaning neighbors for existing pipelines.

Firstly, almost every one of the GCD works in the literature treats the labeled set and the unlabelled set differently. The labeled set is trained on cross-entropy loss between the predicted class and ground truth class labels. The unlabelled data is used for training a contrastive component. There is no notion of neighbors in this work.

The contributions of UNIC include, (1) getting rid of the cross-entropy loss that works directly using the ground truth class labels, (2) adding positive and negative neighbors for both labeled and unlabelled data, (3) cleaning positive neighbors, (4) multiple loss functions.

Getting rid of the need for ground truth class labels explicitly in the training is the reason why UNIC can perform both GCD and image clustering. These labels are implicitly used for positive neighbors for old classes in the GCD case. However, the UNIC pipeline itself is agnostic to whether these labels are used or not.

It is impossible to answer the question “How will UNIC’s neighbor mining improve the GCD baseline?” without significant changes to the GCD baseline code, which in turn would make the implementation very different from the GCD baseline proposed in their paper.

The authors were unable to locate some work in GCD literature where adding our neighbor mining and cleaning was a natural extension.

In contrast, most clustering literature has a notion of neighbors. Therefore, we have conducted experiments by fitting our mining strategies to existing works in a very natural fashion. The results and discussion are presented in A.4 FAQ(1) IS THIS BETTER THAN SCAN+DINO?

[W 1.3.1]The authors could have evaluated on more datasets. For instance, it is common practise to evaluate on the full "SSB" suite (including Stanford Cars and FGVCAircraft) in GCD.

We thank the reviewer for the suggestion to include more experiments and strengthen our paper. We run experiments on CUB-200, StanfordCars-196, and Aircrafts-100 datasets. We run two experiments for each dataset. First, we experiment with the DINOv1 pre trained Vit-B/16 backbone similar to earlier GCD literature. Then, we follow experiments with SimGCD [ICCV 2023] pretrarined backbone similar to recent GCD literature SPTnet [ICLR 2024]. Please refer to the table below for experimental results.

Our results show that UNIC suffers from the low-quality neighbors mined with DINOv1 backbone. However, UNIC outperforms the existing literature when it is able to utilize SimGCD pretrained backbones and achieve state-of-the-art. It should be noted that SPTnet achieves its improvement over the previous literature by adding additional prompt parameters to the ViT backbone. UNIC achieves better performance without changing the backbone architecture (Only tuning the last block).