DebGCD: Debiased Learning with Distribution Guidance for Generalized Category Discovery

Q7. Using CLIP in GCD

Thanks for your insightful comments. Firstly, the label bias problem we aim to address arises from the design of previous GCD methods: unlike known categories, unknown categories only receive soft supervision (see Fig.1). In these methods, label bias remains an issue regardless of the pre-trained backbone network used. Secondly, there have indeed been attempts to leverage CLIP for tackling GCD, such as CLIP-GCD [1], and GET [2]. CLIP provides strong representation capabilities and demonstrates good zero-shot transfer performance. However, it requires expensive large-scale pretraining and poses a risk of data contamination, complicating the distinction between seen and unseen classes. Additionally, it struggles with instances that fall outside the text vocabulary, particularly with the emergence of previously unseen classes. Thirdly, despite these concerns, we can still try to apply CLIP to both our method and the baseline, and we observe consistent performance improvements with the DINO backbone. As shown in Table A11, our D2G achieves an average improvement of 8.2% in ACC across 'All' categories on the SSB benchmark, achieving a highest average ACC.

Table A11. GCD performance using CLIP.

[1] Ouldnoughi, Rabah, Chia-Wen Kuo, and Zsolt Kira. "Clip-gcd: Simple language guided generalized category discovery." arxiv, 2023.

[2] Wang, Enguang, et al. "GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery." arxiv, 2024.

Q1. Reason for using OOD techniques to solve GCD

Thanks for your insightful comments. The motivation for utilizing OOD techniques stems from the inherent semantic shifts present in the GCD task, which involves both known and unknown classes within unlabelled data. Although the objectives of the GCD task and OOD techniques differ, OOD can supply valuable semantic distribution information that can guide the training of GCD on unlabelled data. Additionally, in other open-world tasks such as open-set semi-supervised learning [1] and universal domain adaptation [2], OOD techniques have been shown to offer significant advantages.

Q2. Motivation for using MLP

Thanks for your insightful comments. Please refer to Q1 in general response.

Q3. Performance improvement

Thanks for the comments. On the fine-grained SSB benchmark, our method achieves the highest ACC, clearly surpassing the previous SOTA (64.4 vs. 61.4, as shown in Tab.2). In terms of generic datasets, our method outperforms others on three out of four datasets regarding ACC on 'All' categories, being only 0.7% lower than the best performance on CIFAR10, where the results are nearly saturated.

Q4. Few comparison methods on ImageNet-1K

Thanks for your insightful comments. Indeed, very few methods have reported results on ImageNet-1K due to the high computational costs involved. We have carefully gathered all publicly available results for ImageNet-1K and found that only SimGCD has reported results, which are inferior to ours.

Q5. Ablation on the number of layers in MLP and loss weight

Thanks for your insightful comments. Please refer to Q1 and Q2 in general response.

Q6. Limited performance improvement on some datasets

Thanks for your insightful comments. To address this concern, we provide quantitative analysis on the improvements brought by our method. Particularly, we examine the baseline model’s prediction by categorizing the errors into four types based on the relationship between the predicted and ground-truth classes: "True Old", "False New", "False Old", and "True New". For example, "True New" refers to wrongly predicting a 'New' class sample to another 'New' class, while "False Old" indicates predicting a 'New' class sample as some 'Old' class.

From this perspective, our debiased learning method primarily aims to mitigate the label bias between 'Old' and 'New' classes, thereby reducing the likelihood of 'New' class samples being predicted as 'Old'. Consequently, this reduction in bias leads to a decrease in "False Old" predictions and indirectly improves other types of error. In Table A10, we present the ratios of the four prediction error types for SimGCD with the DINO backbone across three datasets in the SSB benchmark. The results reveal that error distributions vary significantly across datasets, influenced by the dataset's characteristics, the classification of known and unknown categories, the baseline design, and the pretrained backbone network used. Notably, the Stanford Cars dataset exhibits the highest number of "False Old" samples, explaining why our method demonstrates the most substantial performance improvement on this dataset. In contrast, the CUB dataset shows the fewest "False Old" samples, indicating relatively limited potential for performance enhancement. We have incorporated this analysis into our revised version.

Table A10. Error analysis of SimGCD on different datasets in the SSB benchmark.

[1] Yu, Qing, et al. "Multi-task curriculum framework for open-set semi-supervised learning." ECCV, 2020.

[2] Saito, Kuniaki, and Kate Saenko. "Ovanet: One-vs-all network for universal domain adaptation." ICCV 2021.

Dear Reviewer Q4NM,

We greatly appreciate your valuable time and effort in reviewing our paper. We understand that this may be a busy period for you. As the discussion phase draws to a close, we kindly request your feedback on our responses. If you have any additional comments or questions regarding our paper, we would be more than happy to discuss them with you in detail.

We look forward to your reply.

Best regards,

Authors

Q1. Multi-run results

Thanks for your insightful comments. We have included multi-run results for CUB, Stanford Cars, FGVC-Aircraft, CIFAR-10, CIFAR-100, ImageNet-100, ImageNet-1K, and Herbarium19, as shown in Table A8. Despite achieving significantly higher performance, we observe that the variance is even smaller than that of SimGCD (refer to Tab.6 in the Appendix B.1 of the SimGCD paper). We have added this table to the Appendix in the revised version.

Table A8. Multi-run results of D2G.

Q2. Training time and computational costs

Thanks for your insightful comments. We provide the following information below. During inference, all proposed components will be removed, resulting in the computational cost of D2G being equivalent to that of SimGCD. During training, since the main computational cost arises from the backbone network (ViT-B/16), the difference in training time and computational cost between SimGCD and our method is negligible when the same blocks are tuned. In the table below, we present the FLOPs required for a single forward step and the elapsed time for a single training iteration on the CUB dataset, utilizing a batch size of 128 with a NVIDIA V100 GPU. The performance comparison when tuning the same blocks can be found in Q3 of Reviewer YCCi.

Table A9. FLOPs and time costs during training.

Q3. Strategies to reduce potential overfitting

Thanks for your insightful comments. We have incorporated several strategies in the baseline, such as entropy regularization and weight decay, which are effective in reducing overfitting. Therefore, we did not introduce any additional strategies for our debiased learning approach. As indicated by the performance on the held-out validation set (see Fig.5 in the supplementary materials), the model did not exhibit overfitting to the training set.

Q4. The effect of label debiasing

Thanks for your insightful comments. We would like to clarify that while the label debiasing strategy is effective, the auxiliary classifier serves as our vehicle for implementing debiased learning (refer to Table 4 and lines 482–484 in the paper). As shown in Tab.4, applying the debiased loss to the original classifier leads to a decline in performance, as indicated in row 2 compared to row 1. This decline is primarily due to the reliance on the original GCD loss for that classifier, which still results in a biased supervision signal. Comparing row 3 with row 1, we observe a notable improvement of 4.7% in 'All' ACC over the baseline, achieved through our debiased learning with the auxiliary classifier. This highlights the necessity of our auxiliary classifier, which is designed to be debiased in order to facilitate effective debiased learning and optimize the shared GCD feature space. Furthermore, if we do not utilize the label debiasing strategy, the auxiliary classifier—being a core component of this strategy—should be excluded entirely. We have included this clarification in the revised version.

Q1. Brown dashed line in Fig.1

Thank you for pointing this out. However, we have observed that the document displays normally on our end, across various PDF readers and browsers. We would appreciate it if the reviewer could provide more details about the issue encountered, as we would be happy to investigate and work on a resolution.

Q2. Hyperparameters in Eq.14

Thanks for your insightful comments. Please refer to Q2 in general response.

Q3. Comparison with InfoSeive

Thanks for your insightful comments. Infosieve is a hierarchical encoding method specifically designed for fine-grained GCD, which may work well for certain datasets. In contrast, our method does not incorporate specific designs tailored for fine-grained datasets; instead, it aims for broader improvements across both generic and fine-grained datasets. Notably, our method significantly outperforms Infosieve on the SSB benchmark (64.4 vs. 60.5, see Tab.2), showing only a slight performance gap (3.1% lower on 'All' categories) on one of the three datasets. On Stanford Cars and FGVC-Aircraft, our method demonstrates considerable advantages, achieving improvements of 9.6% and 5.4% on 'All' categories, respectively. Additionally, on all generic datasets (see Tab.3), our method consistently surpasses Infosieve, with improvements of 2.4%, 4.7%, and 5.4% on CIFAR10, CIFAR100, and ImageNet-100, respectively, on the 'All' categories.

Q4. Ablation studies on more datasets

Thanks for your insightful comments. We have included additional ablation results for the other two datasets in the SSB benchmark (CUB and FGVC-Aircraft), as well as the generic dataset ImageNet-100, in Table A7. In this table, the letters a, b, c, and d represent Debiased Learning, Auxiliary Classifier, Semantic Distribution Learning, and Distribution Guidance, respectively. Our results indicate that directly applying debiased learning to the original GCD classifier may lead to a drop in performance (see comparisons between (1) and (2)). In contrast, when using an auxiliary classifier, we observe an improvement in performance (comparing (1) and (3)). Furthermore, the joint training of the debiased classifier and the OOD detector yields additional enhancements (comparing (3) and (5)). Finally, the introduction of distribution guidance leads to further performance improvements. These findings are consistent with those obtained on the Stanford Cars dataset, as shown in Tab.4 of our paper. We have incorporated these results into the revised version, as suggested.

Table A7. Additional ablation studies on CUB, Aircraft and ImageNet-100.

Q1. OVA vs self-entropy

Thanks for your insightful comments. Firstly, we agree that self-entropy may seem like a more intuitive approach than the OVA design for OOD detection. In common practice, the maximum score or logit on categories from a closed-set classifier can serve as a good indicator of OOD. However, the situation is different for the GCD classifier. There is an entropy regularization term in the loss function (see $H(\overline{p})$ in Eq.2), which aims to prevent trivial predictions. The SimGCD paper indicates that this regularization term helps mitigate prediction bias between seen and novel classes. Nevertheless, we find that it also results in the classifier's predictions on known categories being less confident than those of a closed-set classifier, thereby degrading the OOD detection performance of the GCD classifier. Secondly, our framework does not aim to strictly partition the in-distribution (ID) and OOD samples, as such strict separation could introduce errors to training. Instead, our goal is to utilize the distribution certainty of each sample. A significant drawback of self-entropy-based OOD methods is the necessity to manually establish a threshold for rejecting "unknown" samples, which relies on validation or a pre-defined ratio of "unknown" samples. This approach is impractical and complicates the definition of unified certainty scores for all samples. In contrast, OVA-based methods eliminate the need for threshold searching, allowing us to simply use a threshold of 0.5, which facilitates the definition of a simple certainty score as presented in Eq. 10. Thirdly, to further validate the above hypothesis, we evaluated the OOD performance of the GCD classifier by using the maximum score on known categories as the ID score. The AUROC results are shown below. Compared to the results in Tab.13, we observe that this approach is less effective than the OVA classifier, as shown in Table A5. Moreover, when we train the OVA classifier, debiased classifier, and GCD classifier simultaneously, the three tasks can mutually benefit both OOD and GCD performance, as illustrated in Tab.4(row 1 & row 4) and Tab.13(row 1 & row 2) in the paper. We have included this discussion in the revised version.

Table A5. The OOD performance of different methods.

Q2. Supervision on uncertain samples

Thanks for the comments. We would like to clarify that Eq. 10 indeed imposes weaker supervision for samples identified as uncertain by the OOD detector. For these uncertain samples, their OOD score $s_i$ will be close to 0.5, resulting in their distribution certainty score $d_i$ approaching 0. Consequently, Eq. 10 imposes a weaker pseudo one-hot supervision on these samples.

Q3. Performance comparison with the same tuned blocks

Thanks for your insightful comments. We would like to present the performance of training the last one or two blocks. As demonstrated in Table A6, this does not result in significant improvements for SimGCD. In contrast to SimGCD, our framework incorporates additional tasks, including OOD detection and debiased learning. We empirically observed that increasing the number of trainable parameters can improve performance on specific datasets, particularly those that are fine-grained. Similar strategies have been employed in previous methods, such as Infosieve[1]. We have included this comparison in the revised version.

Table A6. Performance comparison of SimGCD and D2G with different tuned blocks.

Q4. Performance on Herbarium19

Thanks for the comments. In fact, there is a noticeable performance improvement (see Tab.7) over SimGCD baseline. Our method achieves an ACC of (44.7, 59.4, 36.8), while SimGCD records an ACC of (44.0, 58.0, 36.4) for the 'All', 'Old', and 'New' categories, respectively. It is important to highlight that the Herbarium19 dataset poses unique challenges due to its long-tailed characteristics, which complicates performance for both SimGCD and our method.

Q5. Key to performance improvement

Thanks for your insightful comments. We agree that the CLS token plays a crucial role in enhancing our performance. This improvement is achieved through the optimization of the entire embedding space and classifier via our debiased learning approach. In Fig.7, we visualize the cross-attention between the CLS token and patch embeddings, revealing that the maps generated by our model predominantly focus on the object of interest while effectively ignoring spurious factors and background clutter. The attention maps demonstrate that the CLS tokens in our method exhibit stronger discriminative power compared to the patch tokens, thereby validating that CLS tokens are more effective for distinguishing between both seen and unseen classes. We have incorporated this explanation in the revised version as suggested.

Q6. Debiased loss on the original GCD classifier

Thank you for raising this question. Indeed, we considered this solution at the very beginning of our project. However, we found that applying the debiased loss to the original classifier still results in a biased supervision signal, primarily due to the reliance on the original GCD loss for that classifier. In fact, incorporating the debiased loss into the original classifier leads to a decline in performance (see row 1 & row 2 in Tab.4). This highlights the need for a second classifier, which is debiased by design, in order to facilitate debiased learning and optimize the shared GCD feature space. We have included this explanation in the revised version as suggested.

Q7. Impact of MLP prior to OVA module

Thanks for your insightful comments. Please refer to Q1 in general response.

[1] Rastegar, Sarah, Hazel Doughty, and Cees Snoek. "Learn to categorize or categorize to learn? self-coding for generalized category discovery." NeurIPS 2024.

Dear Reviewer YCCi,

We greatly appreciate your valuable time and effort in reviewing our paper. We understand that this may be a busy period for you. As the discussion phase draws to a close, we kindly request your feedback on our responses. If you have any additional comments or questions regarding our paper, we would be more than happy to discuss them with you in detail.

We look forward to your reply.

Best regards,

Authors

We have revised the paper and would like to invite the reviewers to take a look. Following the reviewers' suggestions, we have made the following major updates:

Section 4.1: Added a detailed discussion about self-entropy and OVA for OOD scenarios, along with a clearer motivation for using the MLP.

Section 5.2: Provided additional discussion with InfoSieve.

Section 5.3: Expanded the discussion on debiased learning for the GCD classifier.

Appendix I: Included a more comprehensive discussion about the CLS token.

Appendix J: Added ablation studies conducted on additional datasets.

Appendix K: Provided hyperparameter analysis regarding the number of MLP layers, loss weights, and the number of tuned blocks.

Appendix L: Included results from multi-run experiments.

Appendix M: Added an analysis of prediction errors.

Please let us know if there are any further concerns.