Adaptive Knowledge Transfer for Generalized Category Discovery

We sincerely appreciate the thorough evaluation and constructive feedback provided. We hope that the following response could address your concerns.

Weakness

(1) Thank you for your clarification. We aim to convey a clear overview of explicit knowledge transfer within GCD problems. While we instantiate knowledge generation, alignment, and distillation using a known-class pre-trained model, AL and CSM, and nMI, our framework allows for flexibility and alternative methods can be employed for these components. For instance, as outlined in the paper, the contrastive loss we utilize can be replaced by other knowledge distillation methods. The primary objective of this writing is to elucidate the explicit knowledge transfer framework for GCD problems, highlight the core challenges associated with knowledge transfer in GCD problems, and offer insights for future research.

(2) We respectfully disagree with you on three points:

1. Knowledge distillation is not the most important component of our method. It holds equal importance with other components, AL and CSM. The ablation study in Tab.4 demonstrates that each component significantly contributes to the model's improvement. As discussed in the paper, not all knowledge in known class data is beneficial for novel class learning. This indicates that nMI does not always enhance the model's performance, as supported by the evidence in Tab.12 in Appendix G. This underscores the importance of AL and CSM.
2. We would like to clarify that we do not claim novelty in negative sample generation within the paper, which is widely used across various contrastive learning domains. All the other negative sample generation strategies that avoid class collision issues like [2, 3] can also be applied to our method.
3. Negative sample generation is not an important component in our method. As shown in the table below, the model's performance without the "negative sample generation" technique remains satisfiable. The reason we opted to use "negative sample generation" is to address the class collision issues of novel classes, as evidenced in the presented results.

Question

We would like to clarify that Fig. 6 illustrates the average outcome of the Channel Selection Matrix generated for each sample in the corresponding class. It can well reflect whether CSM can produce a general selection method for a specific class. However, it does not represent the number of open channels. Additionally, it's worth noting that the values represented by the color depth of each subfigure in Fig. 6 differ, with the maximum value of 0.002 on ImageNet being much smaller than observed on several other datasets. Here, we present the average number of open channels on novel classes in the table below, indicating that fewer channels are transferred in generic datasets, while more channels are transferred in fine-grained datasets.

[1] Openmix: Reviving known knowledge for discovering novel visual categories in an open world (CVPR 2021)

[2] Neighborhood contrastive learning for novel class discovery (CVPR 2021)

[3] Zheng, Mingkai, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. "Weakly supervised contrastive learning." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10042-10051. 2021.

We sincerely appreciate the thorough evaluation and constructive feedback provided. We hope that the following response could address your concerns.

Weakness

(1) We would like to clarify the novelty of our method and its close relationship with the GCD problem in three points:

1. Our framework represents a novel approach to addressing GCD challenges, specifically designed for explicit knowledge transfer in GCD tasks. To our best knowledge, no similar framework has been proposed previously for GCD tasks. As discussed in the introduction, knowledge generation, knowledge alignment, and knowledge distillation in our framework aim to generate known class knowledge representation, facilitate more targeted knowledge transfer, and perform knowledge transfer, respectively. Each component is specifically designed for the task of explicit knowledge transfer in GCD.
2. The techniques proposed in our work are intricately aligned with this innovative framework. Our proposed AL and CSM are unique solutions crafted specifically to tackle knowledge transfer challenges in the GCD problem. AL aligns the known-class representation space to the joint representation space, and CSM selects meaningful channels, enabling more targeted knowledge transfer. To the best of our knowledge, no such techniques have been proposed before.
3. We utilize the contrastive loss to maximize the mutual information. The contrastive loss we introduce is not a traditional one; rather, it uniquely combines AL and CSM, tailoring it to the intricacies of the GCD problem. This novel contrastive loss enables effective knowledge transfer between known and novel classes, unlike traditional knowledge transfer methods.

(2) We contend that the use of ReLU activation is a deliberate choice based on its simplicity and efficiency. And we believe that simplicity is not a drawback but rather an advantage, contributing to the ease of implementing our proposed framework and inspiring future work in designing more sophisticated mechanisms for better knowledge selection.

(3) We would like to point out that the goal of GCD is to learn a model, that can better classify known classes and cluster novel classes, concurrently. Since most existing methods can adapt the model's learning focus by adjusting the weight of $\mathcal{L} _ {u}$ in Equ (1), it is not adequate to evaluate the performance of the model based only on a single metric in the GCD problem. A case in point is the CUB dataset, where the DCCL method exhibits superior performance on novel classes compared to all other methods, yet its performance on known classes is inferior. This observation suggests a potential bias in the DCCL method towards novel class data. Unfortunately, comprehensive analysis is difficult as DCCL has not released its code. When considering all metrics comprehensively, our model is also better than other models on CUB.

Question

Please refer to the weakness section.

We sincerely appreciate the thorough evaluation and constructive feedback provided. We hope that the following response could address your concerns.

(1) Firstly, concerning AL, its objective is to transform the representation space of known classes into a joint representation space that is more advantageous for novel classes. However, because the transformation introduces novel classes, it may have a negative impact on known classes, depending on the relationship between known and novel classes, which is dataset-specific. We further conduct an ablation study on the coarse-grained dataset in Appendix G, where we also observe a performance drop in known classes and an increase in novel classes when applying AL. The potential negative impact of AL underscores the importance of the design of the CSM. It is noteworthy that the features obtained after both AL and CSM can be considered beneficial for novel class learning. This observation is further supported by the results of the ablation study. Secondly, we would like to clarify that our model integrates not only AL but also nMI and CSM. The utilization of all three components collectively results in notable improvements across all known and novel classes in all datasets. This underscores the method's robust generalizability.

(2) Apologies for any confusion. In Table 5, we assess the model with a frozen known-class pre-trained encoder, adapter layer, and cluster head (denoted as 'AL') to evaluate the quality of the Adapter Layer. The adapter layer and cluster head are learned by $\mathcal{L} _ {base}$. When the number of layers is set to 0, it means we only learn a cluster head. "Ours" denotes our final model with all components. Importantly, when the number of layers is set to 0, "Ours" directly applies nMI without AL and CSM modules.

We sincerely appreciate the thorough evaluation and constructive feedback provided. We hope that our response could address your concerns.

Weakness

We opt for the ReLU activation function due to its simplicity and efficiency. Following your insights, we have conducted an analysis to count the number of dead neurons in the model, and the results are presented in the table below. Fortunately, our final model utilizing ReLU does not exhibit this issue. We believe this is potentially due to our truncated norm initialization. In the subsequent discussion, we will delve into a comparative analysis of the model's performances when employing ReLU and GeLU.

Question

(1) We appreciate your attention to the activation function choice. As depicted in the table below, adopting GeLU resulted in marginal changes in model performance. These experimental findings indicate that both GeLU and ReLU deliver comparable performances, providing evidence that there are few or no dead neurons in our model. Combining this with the empirical results mentioned above, we maintain that ReLU remains a straightforward and effective choice under a good initialization.

(2) In Figure 2, the 'Cls' in the lower branch and the upper branch both can classify labeled and unlabeled data. The 'Cls' in the upper branch is learned based on $\mathcal{L} _ {base}$ in Equ (1). The loss term $\mathcal{L} _ {u}$ in $\mathcal{L} _ {base}$ is the self-labeling loss. Specifically, it utilizes the predictions of 'Cls' in one view of the image to generate pseudo-labels for the other view of the same image. This mechanism enables the model to assign categories to unlabeled data. As a result, the 'Cls' component can classify both known and novel classes. For a more in-depth explanation of $\mathcal{L} _ {u}$, we invite you to refer to Appendix D. Your interest is greatly appreciated.

(4) Firstly, as evidenced in the table above, directly performing knowledge transfer without AL and CSM may not consistently enhance model performance. This phenomenon confirms that not all supervised knowledge is beneficial. This reaffirms the necessity of introducing AL and CSM to filter and refine the transferred knowledge. Secondly, [3] only do experiments on CIFAR and ImageNet datasets. We follow their method and extend their experiments to iNat21 datasets and find that their conclusions may not hold for iNat21 datasets, emphasizing the dataset-specific nature of their findings. As shown in the table below, the novel class performance of UNO is consistent with our assumption on the data splits, and self-supervised uno, which abandons supervised knowledge, achieves inferior results. The results indicate that

1. Supervised knowledge is still beneficial in the hard split of the iNat21 dataset.
2. completely eliminating supervised information is not an optimal choice thus emphasizing the importance of transforming and filtering beneficial components in our proposed knowledge transfer framework.

(5) Firstly, we have to clarify that we do not utilize the "dynamic conception generation"[10] technique. Secondly, the cluster size regularization technique is widely employed in unsupervised learning[5, 6] and GCD[7, 8, 9], and it is not our contribution. It ensures meaningful clustering by preventing the model from grouping all data into a single cluster. Therefore, without cluster size regularization, our model and most other GCD models will decrease greatly. Here, we also present the model's performance without cluster size regularization (CSR) in the table below. This table confirms the importance of cluster size regularization as discussed earlier. Nevertheless, our method can still contribute to performance improvement even without cluster size regularization.

[1] Zhao et al, "Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation", NeurIPS 2021.

[2] Gu et al. "Class-relation Knowledge Distillation for Novel Class Discovery", ICCV 2023

[3] Li, Ziyun, et al. "Supervised Knowledge May Hurt Novel Class Discovery Performance." arXiv preprint arXiv:2306.03648 (2023).

[4] Oord AV, Li Y, Vinyals O. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748. 2018 Jul 10

[5] Asano YM., Rupprecht C., and Vedaldi A. "Self-labelling via simultaneous clustering and representation learning." In Proc. ICLR, 2020.

[6] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. "Scan: Learning to classify images without labels." In Proc. ECCV, 2020.

[7] Fini, Enrico, et al. "A unified objective for novel class discovery." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[8] Wen, Xin, Bingchen Zhao, and Xiaojuan Qi. "Parametric classification for generalized category discovery: A baseline study." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[9] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.

[10] Pu, Nan, Zhun Zhong, and Nicu Sebe. "Dynamic Conceptional Contrastive Learning for Generalized Category Discovery." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

We sincerely appreciate the thorough evaluation and constructive feedback provided. We hope that the following response could address your concerns.

Weakness

(1) We appreciate your feedback and have addressed the issues in the revised version of the paper.

• “BL” is the baseline model. In detail, the baseline model has the same architecture as the final model and only uses $\mathcal{L} _ {base}$ in Equ (1) to learn known and novel classes. The loss can influence both the encoder and the classifier. Meanwhile, “Clu.” denotes that we directly cluster the features generated by the pre-trained known-class model. The model of “Clu.” is also the model with the same architecture as the final model and uses the same $\mathcal{L} _ {base}$ to learn known and novel classes. But now, the encoder is frozen, meaning the loss can only influence the classifier. In Appendix H, we also discuss the details of clustering.
• This baseline model is the same model “BL” that we discussed above.
• As crNCD is originally designed for NCD problems, we have adapted it to the GCD setting using a widely employed architecture utilized in other GCD methods. Additionally, to mitigate the influence of different self-labeling designs, we employ the same $\mathcal{L} _ {base}$ (Equ.1) as in our method, instead of using the original $\mathcal{L} _ {opt}$ from crNCD. The original $\mathcal{L} _ {opt}$ in crNCD did not perform well in the GCD setting, as indicated in the table below. It is important to note that we have made no modifications to the core loss $\mathcal{L} _ {rKT}$.

(2) We would like to clarify the distinctions between our method and those proposed in [1,2]. We note that a detailed discussion of [2] has been provided in the related work section.

• Here we summarize the three main differences between our methods and [2] for clarity.
  1. Our paper proposes a novel explicit knowledge transfer framework that introduces a fresh perspective for advancing knowledge transfer and [2] is only one of the concrete implementations of our proposed framework. Compared with the limited form of knowledge transfer in [2], our framework is more flexible and effective.
  2. [2] performs knowledge distillation on the output space, while we conduct it in the feature space which contains more information. [2] does not employ knowledge selection to filter out harmful knowledge or knowledge alignment to transfer knowledge more efficiently; instead, they use a simple weight function to amplify knowledge distillation when class relations are strong.
  3. [2] is specifically designed for the NCD problem and it achieves poor results in GCD (table above).
• Regarding [1], the primary objective of their loss function $\mathcal{L} _ {sKLD}$ is to enable the model to capture both global and local information within an image. In contrast, our method is specifically designed to facilitate the transfer of known class knowledge to novel class learning. Notably, [1] employs a two-branch learning framework, with one branch focusing on local part-level information and the other on overall characteristics. They transfer knowledge between these two branches. They do not require knowledge alignment because they perform knowledge distillation in the same label space. In contrast, our approach employs an upper branch dedicated to extracting valuable known class knowledge and proposes AL and CSM to do effective knowledge transfer in different representation spaces. These fundamental differences in the purpose and methods distinguish our work from [1].

(3) Thank you for your suggestion. In response, we provide the derivation of infoNCE from mutual information and the details can be found in Appendix A.