Composing Novel Classes: A Concept-Driven Approach to Generalized Category Discovery

Thank you for your detailed and thoughtful feedback. We deeply appreciate the effort you put into reviewing our work, and we value your insights. In the following, we have carefully addressed each of your concerns to the best of our ability.

W1: Experiments with ResNet Model trained from scratch

We appreciate the reviewers' suggestions. Following your suggestions, we have conducted experiments using ResNet. We compare our approach with the strong baseline, SimGCD, and present the results as follows:

Our method achieved remarkable improvement in the novel classes on three datasets. Note that for simGCD, the reported results on ImageNet100 represent the best performance we have achieved so far. It is highly time-consuming to tune for the best hyperparameters.

Furthermore, as you mentioned, fine-grained datasets are considered novel for DinoV1. The superior performance of our method on these datasets also substantiates the effectiveness of our approach.

W2: prior knowledge of the number of clusters

Yes, this is a common limitation inherent in GCD. Concurrently, as shown in Table 6, our approach still significantly outperforms previous methods with the estimated number of clusters. Moreover, our method does not show a substantial decrease in performance when compared to scenarios where the number of clusters is known, demonstrating the robustness of our method.

W3: Contribution

We believe our major novelty does not lie in the application of covariance loss to GCD. The novelty of our paper can be summarized in the following two points:

1. Novel Learning Framework. We establish a novel framework based on concepts, analyzing the relationship between novel and known concepts. We classify these concepts into two main categories: derivable and underivable concepts. This analysis leads to our proposal of a novel three-stage modeling framework for learning those concepts.

2. Two effective techniques. In addition to our model framework, we introduce two methods to enhance the concepts learned by the model. First, we incorporate covariance loss into GCD to improve the learning of concepts. Second, we propose a concept score normalization technique that ensures the model learns derivable and underivable concepts in a more balanced manner.

Those two points are novel in the GCD problem and greatly improve the results.

W4: English writting

Thank you for your advice. We have fixed this error in our new version.

W5: Definition of Concepts

Please refer to General Response.

W6 & Q2: motivational insights of derivable concept

Thank you for your feedback. We believe the motivational insights claimed in the introduction section are convincing. Please first refer to General Response for Derivable and Underivable Concept and Derivable Concepts’ Utility.

As depicted in Figure 1, in the conventional GCD setting, we train a single linear layer and a cosine classifier on top of a pre-trained model that has been trained on known classes. This classifier is then used to cluster novel classes and classify known classes. Unlike SPTNet and crNCD, which incorporate intricate designs and train more parameters, our linear classifier-based approach delivers performance that is on par with these methods on the CUB and ImageNet100 datasets and surpasses them on the Scars dataset. The exciting results achieved by our simple design of generating derivable concepts demonstrate our motivation.

We appreciate your suggestion and are open to conducting further control experiments and qualitative analyses. Could you please specify the types of experiments and analyses you believe would be most beneficial to validate the influence of 'derivable concepts'?

W7: notation is unclear

$l$ is the number of known class concepts $l = |C^{k}|$. $l’$ is the number of novel class concepts $l’ = |C^u|$. Thanks for your advice. We have changed $z_i$ to $\mathbf{z}_i$

Q1,Q3: Concerns related to concept

Please refer to General Response.

Q4 & Q5: Concept and Semantic Visual Cues

The concepts are not equivalent to semantic visual cues. Fig. 2, represents "concepts" as semantic visual cues to aid understanding. The visualization of learned concepts is shown in Appendix M. Additionally, following your suggestion, we change the color of the wheel and illustrate the relationship between concepts and semantic visual cues in Appendix M.

Q6: Application

For users, we can adapt the existing method [1] to estimate the number of classes, and then discover novel classes. Once we can estimate the number of classes, we can apply GCD to various settings, especially in scenarios, where exists massive unlabeled data and labeling them is costly. In addition, it has been applied to cell discovery [2].

We sincerely hope the response above addresses your concerns. If not, please feel free to raise any remaining issues, and we will be more than happy to address them.

[1] Sagar Vaze et al, Generalized Category Discovery

[2] Brbic et al. MARS: discovering novel cell types across heterogeneous single-cell experiments

Thank you for providing the clarification and additional experiments.

My concerns are partially addressed in the rebuttal, such as the comparison with non-pretrained backbone (e.g., dinoV1 or dinoV2). The additional experiments added by the Authors show the robustness and good performance of the method proposed in this work.

However, my major concerns remained:

(1) Unclear and misleading definition of "Concepts", "Known class concept", "Derivable concept", and "Underivable concept". This concern is major because it is the foundation of the main motivation of this work. The definition, explaination and justification about "Concept" and "Why concept helps" in the context of GCD is too subjective. If the justification holds, qualitative experiments are needed to support the claim.

More importantly, the Authors used "We represent "concepts" as semantic visual cues in Fig. 2 just for understanding." in the Global Response to justify the concern related to "Concept and Semantic Visual Cues". I could not agree with this answer because: If the Concepts in this work are not Semantic Visual Cues (e.g., attributes or parts as present in this manuscript), this representation is misleading and does not correspond to the facts. As a reader, the message I received from Fig. 2 is: "concepts are learned compositional semantic cues. That's why the concepts learned from known might be useful for novel classes". However, this is not in line with the actual concept definition in this work.

(2) My question The authors claimed, “This observation suggests that existing approaches may struggle to capture all the derivable concepts useful for knowledge transfer.” (P2#L65) —> Could you please further explain why the results in Table 1 support this claim? Why is the relatively lower performance of SPTNet and crNCD attributed to a "struggle to capture all the derivable concepts"? is not answered. From what observation or experiments the Authors can get this conclusion? In addition, the Authors described " Unlike SPTNet and crNCD, which incorporate intricate designs". I do not think SPTNet and crNCD have intricate designs. And, this is not related to my qustion.

(3) The justification about Novelty. Such two stage learning framework has been widely used with different loss function and architectural designs from the very beginning of GCD and NCD literature because the Base/Novel split setting [1, 2]. Even the foundation NCD work AutoNovel (RankStats) [1] used such learning framework. Therefore, this framework used in this work can not be considered as a novelty. Thus, the novelty of this work is limited to the loss function.

In conclusion, the idea for tackling GCD presented in this work is intriguing and holds potential for further exploration. I acknowledge it. However, the current manuscript requires substantial improvements in clarity, soundness, and novelty. Additionally, the claims made in this study need to be more thoroughly substantiated, as they are not sufficiently convincing in the present revision.

Therefore, I decided to keep my initial rating. Thank you for the Authors' responses and effort made during the rebuttal stage, which I believe will make this manuscript strongger in its future revision.

[1] Han, K., Rebuffi, S. A., Ehrhardt, S., Vedaldi, A., & Zisserman, A. (2021). Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6767-6781.

[2] Wang, H., Vaze, S., & Han, K. (2024). SPTNet: An efficient alternative framework for generalized category discovery with spatial prompt tuning. arXiv preprint arXiv:2403.13684.

We thank the reviewer for prompt response and would like to further address the remain concerns

Q1: concepts:

1. We employ the term "concept" to denote the semantic patterns learned from data at a subcategory level. Some of these patterns are semantic visual cues that are comprehensible to humans (such as parts & attributes), while others may not be easily interpretable but are stable visual patterns. For instance, prior works [1, 2, 3] have shown that inputs maximizing a neuron’s activation include both recognizable visual semantic cues and abstract visual patterns. In the comments, we clarify that 'the concepts are not equivalent to semantic visual cues', which means our concepts include but not limit to semantic visual cues. We apologize for any confusion that may have arisen.
2. We want to emphasize that the effectiveness of our framework is built upon the property of those learned subcategorical patterns instead of whether we denote them as 'concepts'. Our framework operates on the premise that known and novel classes share some common subcategorical semantic patterns, an assumption widely accepted in the field of NCD and GCD, and that novel classes possess some unique semantic patterns. We adopt this terminology by following [1, 6] and agree that this term may have varying meanings under different contexts. To avoid confusion, we can easily replace 'concepts' with other terms such as 'semantic patterns', as it refers to a stable pattern learned from data.
3. Our Figure 2 is primarily for illustrative purposes.

[1] Nguyen A, Dosovitskiy A, Yosinski J, et al. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks[J]. Advances in neural information processing systems.

[2] Erhan D, Bengio Y, Courville A, et al. Visualizing higher-layer features of a deep network[J].

[3] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks.

Q2: struggle to capture all the derivable concepts

Given the semantic patterns learned from known classes, we define 'derivable concepts' as a linear combinations of those patterns, which can be modeled as a linear layer. We refer to those patterns of the novel classes that cannot be represented as such linear combinations as 'underivable concepts'.

In Fig. 1 and Tab. 1, we use a simple linear layer to combine known class concept scores to generate new concept scores. Based on our definition of derivable concepts, we can generate them by adding a linear layer after the encoder of a known class pre-trained model. We observe that this linear model outperforms SPTNet and crNCD on certain datasets. We believe this performance improvement arises because our linear method captures concepts that SPTNet and crNCD do not.

To verify this, we conduct an additional experiment. Specifically, we analyze the responses of all 768 neurons in the encoder of our linear method, SPTNet, and crNCD using 100 randomly selected samples from the Stanford Cars dataset. The responses of the neurons are converted into probability distributions on these 100 samples using the softmax function. Since concepts are directly linked to neurons, if the concepts learned by two neurons are similar, their probability distributions across the 100 samples should also be similar. To quantify this similarity, we employ the Kullback-Leibler (KL) divergence to compare the probability distributions of our linear method with those of SPTNet and crNCD. For each neuron in our linear method, we calculate the minimum KL divergence with respect to all neurons in SPTNet and crNCD, respectively.

The table below shows the distribution of neurons in our methods based on their minimum KL divergence values against SPTNet and crNCD. Notably, compared to SPTNet, at least 43 neurons in our method are entirely distinct from those in SPTNet (KL divergence > 1.0), and an additional 77 neurons exhibit differences (KL divergence between 0.5 and 1.0). A similar trend is observed when comparing our method with crNCD. These findings demonstrate that our linear method generates some concepts not captured by either SPTNet or crNCD. Thus, we infer that “existing approaches may struggle to capture all the derivable concepts useful for knowledge transfer”. We hope this additional experiment addresses your concern.

Finally, we would like to apologize for the irrelevant statement: "Unlike SPTNet and crNCD, which incorporate intricate designs." Thank you for pointing this out.

Q3: Novelty

Our framework consists of three distinct phases, each designed to efficiently learn derivable and underivable concepts. Here’s a detailed description of each phase:

1. Representation Initialization: We begin by initializing representations for known classes, similar to existing methods [1,2]. This sets the foundation for further learning.
2. Learning Derivable Representations: This phase introduces a novel approach not found in existing work[1,2]. We employ a generator (linear) layer specifically to generate derivable concepts. The effectiveness of this approach is demonstrated in Table 1, highlighting its significance in our framework.
3. Simultaneously Learning Derivable and Underivable Concepts: In this final stage, we fine-tune both the representation and the classifier. Unlike previous methods [1,2] that directly tune representations, which are ineffective in learning derivable concepts (as shown in Fig 1 and Tab 1), we have designed a contrastive learning strategy to preserve the derivable concepts learned in the second phase. This strategy is crucial, as it prevents the model from losing these learned concepts. Additionally, we adopt an expansion (linear) layer to broaden the concept space and propose a concept normalization technique to better learn underivable concepts.

This structured approach ensures that our framework effectively captures and utilizes both derivable and underivable concepts, enhancing overall performance.

[1] Han, K., Rebuffi, S. A., Ehrhardt, S., Vedaldi, A., & Zisserman, A. (2021). Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6767-6781.

[2] Wang, H., Vaze, S., & Han, K. (2024). SPTNet: An efficient alternative framework for generalized category discovery with spatial prompt tuning. arXiv preprint arXiv:2403.13684.

Thanks the Authors for providing further clarification on the definition of "concepts" and novelty.

I would like to clarify that, I understood well what does "concepts" represent in this work, from a technical perspective. And I acknowledge its effectiveness. My major concern is that, to clarify this, the current manuscript indeed needs major revision, especially the Introduction part. This should be motivated from technical perspective, with experimental observation as support (like you did in the paper and with the additional experiments).

The current writing is too subjective. As you mentioned, "Some of these patterns are semantic visual cues that are comprehensible to humans (such as parts & attributes), while others may not be easily interpretable but are stable visual patterns. ", if this is the case, showing a visualization in the Introduction to motivate and explain your motivation from technical perspective would be very clear. However, this really needs major revision.

I stress again, I like the idea of using these fine-grained relevant and irrelevant middle-level semantics to improve GCD/NCD. This is novel.

However, the major novelty is limited to the losses. This is the fact that can be hardly updated during the rebuttal period. As you listed in your response, NCD/GCD usually starts with supervised representation (learned on known or dino initialized). So Step 1 is common practice in NCD/GCD work by default, because this is the core idea of NCD/GCD. This should be included in your contribution. Further, Step 2 and Step 3 are widely used paradigm with different loss function designs or knowledge distillation schemes, since essentially, NCD/GCD is a transfer learning task. Thus, from framework design perspective, I really cannot consider Step 2 and 3 as your contribution.

The major novelty in my understanding is: 1) the idea of leverage derivable and derivable concepts to improve GCD; 2) the proposal of the loss functions to achieve this intuitive idea. I consider it is limited still. Thus, I keep my initial score.

Thank you!

Writting

Thank you for your thoughtful feedback. We have added additional visualizations of concepts in the appendix. Additionally, we would like to clarify the logical flow of our Introduction to demonstrate that our paper is structured, objective, and aligned with its primary focus. Our reasoning is as follows:

1. The model can learn concepts: This is well-supported by extensive literature [1, 2, 3, 4, 5, 6]. While we agree that visualizations and in-depth analyses are valuable, they are not the primary focus of this work. A detailed exploration of concepts is more appropriate for studies dedicated to visualization and interpretability. Our primary objective is to address the Generalized Category Discovery (GCD) problem effectively. For this reason, we believe including concept visualizations as technical evidence in the appendix is more suitable than placing them in the main text.
2. Similar semantic patterns of novel and known class concepts: Our assumption—common in many GCD studies—is that some novel class concepts share similar semantic patterns with known class concepts. This foundational assumption does not require additional substantiation. Building on this, we categorized concepts into two types: derivable and underivable concepts.
3. The necessity of derivable concepts: The necessity of derivable concepts: As demonstrated in Table 1, we highlight that a model focusing on capturing derivable concepts can achieve performance comparable to or even surpass current SOTA methods [7,8] on certain datasets. To investigate the underlying reasons behind this performance, we conducted an additional experiment based on your suggestion. The results reveal that some neurons in our model exhibit distinctly different activation patterns compared to those in existing SOTA methods, indicating that our approach captures unique concepts absent in prior methods. This observation, combined with the evidence from Table 1, leads us to conclude that current SOTA methods fail to fully capture derivable concepts. Furthermore, the results also demonstrate that these concepts are crucial for enhancing model performance. We would also like to apologize for the subjective phrasing in our original manuscript, particularly the statement: “This observation suggests that existing approaches may struggle to capture all the derivable concepts useful for knowledge transfer.” Based on your feedback, we have refined this section in the updated version of our manuscript. We hope the revised text addresses your concerns and provides a clearer justification for our approach.
4. Our model proposal: Building on the above assumption and findings of this experiment, we propose our model, which is specifically designed to better learn both derivable and underivable concepts. We hope this clarification addresses your concerns and provides a clear overview of our paper’s logical flow. Thank you again for your valuable insights, which have been instrumental in improving our work.

[1] Lee J H, et al. From neural activations to concepts: A survey on explaining concepts in neural networks.

[2] Erhan D, et al. Visualizing higher-layer features of a deep network.

[3] Nguyen A, et al. Understanding neural networks via feature visualization: A survey[J]. Explainable AI: interpreting, explaining and visualizing deep learning.

[4] Nguyen A, et al. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

[5] D. Bau et al, Network Dissection: Quantifying Interpretability of Deep Visual Representations

[6] Matthew D Zeiler et al. Visualizing and understanding convolutional networks.

[7] Peiyan Gu, Chuyu Zhang, Ruijie Xu, and Xuming He. Class-relation knowledge distillation for novel class discovery.

[8] Hongjun Wang, Sagar Vaze, and Kai Han. Sptnet: An efficient alternative framework for generalized category discovery with spatial prompt tuning.

Novelty

As you mentioned, “Step 2 and Step 3 are widely used paradigms with different loss function designs or knowledge distillation schemes, since essentially, NCD/GCD is a transfer learning task.” We respectfully disagree with this characterization. To the best of our knowledge, our framework is the first in GCD to adopt a multi-stage methodology that explicitly separates the learning of derivable and underivable concepts. In NCD, methods such as AutoNovel [1] and crNCD [3] typically follow a two-stage process: supervised pretraining on known classes, followed by joint training on all classes. In contrast, recent GCD approaches, such as GCD [5], SPTNet [2], and SimGCD [4], generally employ a single-stage learning model, bypassing supervised pretraining entirely. These approaches attempt to learn derivable and underivable concepts simultaneously, which, as shown in Table 1, is less effective. In detail, our derivable-underivable separate learning methodology distinguishes itself from existing works in three key aspects: Purpose, Framework, and Loss Function.

1. Purpose:

• Step 2: The primary objective is to focus exclusively on learning derivable concepts, a goal that has not been explored in any existing GCD research [1, 2, 3, 4, 5].
• Step 3: The objective is to retain the derivable concepts learned in Step 2 while simultaneously learning new underivable concepts. This purpose stems directly from our strategy of separating the learning processes for derivable and underivable concepts, making it unique in the context of GCD research[1, 2, 3, 4, 5].

2. Framework:

• Step 2: We introduce a novel linear layer (generator layer) to explicitly model the relationship between derivable and known class concepts. This layer design is distinct from existing GCD approaches[1, 2, 3, 4, 5].
• Step 3: The framework expands the linear layer to effectively incorporate underivable concepts. Model growth is not a common practice in existing GCD methods [1, 2, 3, 4, 5].

3. Loss Function:

• Step 2: Our loss function incorporates a novel covariance loss, which you acknowledge as a significant contribution.
• Step 3: We utilize contrastive knowledge distillation loss to retain derivable concepts and introduce a concept score normalization mechanism to balance the learning of derivable and underivable concepts—addressing a unique challenge in this phase. The overall loss function is specifically designed for Step 3 and is distinct from any current GCD work [1, 2, 3, 4, 5].

These innovations collectively establish the novelty of our Step 2 and Step 3 designs. If you still believe that Steps 2 and 3 are widely used, we would greatly appreciate it if you could point us to specific studies or works that support this claim.

[1] Han, K., Rebuffi, S. A., Ehrhardt, S., Vedaldi, A., & Zisserman, A. (2021). Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6767-6781.

[2] Wang, H., Vaze, S., & Han, K. (2024). SPTNet: An efficient alternative framework for generalized category discovery with spatial prompt tuning. arXiv preprint arXiv:2403.13684.

[3] Gu et al. Class-relation Knowledge Distillation for Novel Class Discovery, ICCV 2023.

[4] Wen et al. Parametric classification for generalized category discovery: A baseline study, ICCV2023.

[5] Vaze et al. Generalized category discovery, CVPR2022.

Dear Authors,

Thanks for thr further clarification.

As a reader, I still find the writing is subjective without experimental support, especially for the definition of "concepts" in this work. This is also pointed out by Reviewer Au3V.

Regarding the novelty, the proposed losses decide the shape (stages) of the current framework. So, I cannot consider the framework as an individual novelty. Further, I did not say prior work used exact the same framework as this work. It is similar. One more stage designed for the corresponding loss does not make the framework per se novel.

Thank you!

Reviewer wv58

Thank you for your thoughtful feedback. We sincerely appreciate your effort and have addressed your questions below.

W1: More explanation

As depicted in Figure 1, following the conventional GCD setting [1,2], we train a single linear layer and a cosine classifier on top of a pre-trained model that has been trained on known class data. Unlike SPTNet and crNCD, which incorporate intricate designs and train more parameters, our simple approach delivers performance that is on par with these methods on the CUB and surpasses them on ImageNet100 datasets and the Scars dataset in Table 1.

[1] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. [2] Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A baseline study.

W2 & Q2: definition or description of "concept"

Please refer to General Response.

W3: definitions of key variables m and n

m is the number of concepts in the second stage model and n is the number of all concepts in our final model. Due to the model design and our ``concept’’ definition, m is also the output dimension of the generator layer and n is also the output dimension of the expansion layer. The ablation study of n and m are provided in Appendix G and H.

W4 & Q3: Mode Size

The generator and expansion layers increase the model size, and it is unclear if the performance gains are due to the larger model rather than the proposed method. The baseline models need to be scaled similarly to ensure fair comparisons. I suggest conducting additional experiments with scaled baseline models to isolate the effect of their proposed method from the impact of increased model size.

The parameter counts for the models are illustrated above. At n=5m, our model's parameter count is nearly on par with those of SimGCD and SPTNet. However, at n=2m, our model boasts a substantially smaller parameter count compared to the others, except InfoSieve, which adopts k-means to cluster all classes. In our final model (n=5m, m=1536), the parameters of the linear and classifier layers are calculated as 768×7680+7680×number of classes. On the other hand, SimGCD and SPTNet employ a DINO head, which is composed of four MLP layers.

The results indicate that when n=2m, our method also outperforms previous approaches, except on the CUB dataset, where we are slightly inferior to InfoSieve. These findings suggest that our improvements are not mainly due to an increase in parameter count, but rather the inherent effectiveness of our method.

We sincerely hope the response above addresses your concerns. If not, please feel free to raise any remaining issues, and we will be more than happy to address them.

Q1 Clarification on Line #41-31

In crNCD, it has been observed that the existing strategy of sharing encoders can disrupt the existing class relations, impeding the effective transfer of knowledge between labeled and unlabeled data. We hypothesize that the reason for this disruption can be susceptible to the label uncertainty of unlabeled data. Therefore, we assert that 'sharing encoders make it challenging to transfer knowledge between labeled and unlabeled data and can be susceptible to the label uncertainty of unlabeled data.' To reduce misunderstanding, we will revise the original statement and remove our hypothesis: 'However, as demonstrated in crNCD (Gu et al., 2023), the strategy of sharing encoders undermines meaningful class relations, complicating the transfer of knowledge between labeled and unlabeled data.'

Thank you for your detailed feedback, which is crucial for enhancing the quality of the manuscript. Below, we address the concerns raised and hope to resolve them effectively.

W1: Batch Size

Thank you for your feedback. The experiment results regarding the hyperparameter $B$ are presented in the table below. As illustrated, the model performance deteriorates when the batch size is too small. There are two potential reasons for this. First, a small batch size may not provide a reliable approximation of the covariance between concepts $i$ and $j$ for $C_{i,j}$. Second, a reduced batch size can adversely affect the regularization constraint in the unsupervised loss term $\mathcal{L}_{u}$ (refer to Eq. 9 in the Appendix), leading to poor clustering performance. Furthermore, the table indicates that a larger batch size benefits the model's performance. In this study, we adhere to the settings of previous works [1, 2] and select a batch size of 128.

[1] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery.

[2] Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A baseline study.

W2: covariance loss

The covariance loss can be applied across all dimensions, not just limited to the top-m dimensions. As evidenced in Table 5, during the second stage of our experiments, we incorporate the covariance loss across all dimensions (dim=m) and discover its beneficial effects. However, upon reviewing the results in Table 8, we observe that including covariance loss in the third stage (dim=n) yielded only marginal improvements. Consequently, we have elected to exclude this loss in the third stage of our approach. We have adjusted the relevant description to avoid misunderstanding.

W3: implement detail of the set of negative samples

As in traditional contrastive learning [1], we treat all other instances as negative samples without any hard example mining strategy. In detail, the memory buffer contains 2048 negative samples.

[1] He et al. Momentum Contrast for Unsupervised Visual Representation Learning.

W4: Explain of Stanford Cars

As shown in Tab. 5 of the paper, the model's impressive performance on Stanford Cars can be primarily attributed to the effectiveness of $\mathcal{L}_{cov}$. This loss function proves beneficial across nearly all datasets, and it is particularly impactful for Stanford Cars. This effectiveness may be linked to the specific characteristics of the dataset itself.

We have included the visualization of Stanford Cars (t-SNE and concept) in Appendix M and N.

W5. CUB performance w.r.t EL dimension

An overly large expansion layer can cause the model to focus excessively on learning underivable concepts, which are difficult to learn and prone to devolving into noise. As the dimension $m$ of the expansion layer increases, the model's propensity to capture noise also increases, which can disrupt the model's existing knowledge and degrade its performance. While concept score normalization can mitigate this issue to some extent, it is not sufficient to completely offset the impact of a significantly large expansion layer. As a result, the model may still learn a considerable amount of noise, leading to a noticeable decline in performance.

To test this hypothesis, we conduct an additional experiment with an even larger expansion layer, specifically one with 20m dimensions. As the results in Table 12 demonstrate, the model's performance deteriorates across nearly all datasets.

I observed that, after I highlighted and disagreed the following quoted statement from the Authors' initial Global Response, the Authors EDITED their Global Response and DELETED this part WITHOUT providing any clarification or notification:

"Concept and Semantic Visual Cues: We represent 'concepts' as semantic visual cues in Fig. 2 just for understanding."
— Authors' initial Global Response, as recorded in the Revisions history. Can be found by clicking the Revisions button.

I would like to bring this to the attention of other Reviewers, ACs, and Viewers as a reminder that my concern regarding this statement remains valid and can still be found in the initial Global Response through the revision history.

It is concerning that the Authors chose to remove this part of their response without addressing it explicitly, while I understand that this issue may influence the evaluation of the paper. Such actions may mislead ACs and other Reviewers about the entire review process and the concerns raised.

This situation highlights the advantage of a fully open-review system: it ensures transparency and traceability. I strongly encourage the Authors to engage directly with reviewer feedback, addressing doubts and concerns constructively rather than avoiding or obscuring them.

Reviewer wv58

******* After Author's clarification

The Authors clarified that, this part of their response was deleted in the global response, but moved to my individual response, while I was writing my response based on their initial version. This is a misunderstanding caused by the not updated response when I was writing. THE ABOVE CONCERN IS WELL-SOLVED NOW after Author's clarification. Thus, the above is not a concern anymore.