Semantic-Enhanced Prototypical Network for Universal Novel Category Discovery

1. To balance the trade-off between smaller and larger patches, this paper employs a 2D pooling operation on the patches. However, it's worth noting that such a pooling operation may result in the loss of finer-grained information. To further investigate this trade-off, I think it is necessary to conduct corresponding experiments.

2. Treating an image as a composition of patches is a common operation in computer vision tasks [1-2]. To enhance the novelty of this paper, further modification strategies should be applied. I think that certain filtering strategies could be effective in such task, particularly for addressing various meaningless patches (e.g., the background).

3. The basic idea of PEB is that a concept word can be utilized to describe one image. However, what I find interesting is whether PEB can be meaningful in a multi-classification task. Further experiments could be conducted in such a scenario to explore the effectiveness of PEB.

4. The pre-trained parameters could potentially impact the experimental results. To provide a more comprehensive illustration of the effectiveness of the PR strategy, I recommend comparing PR with other methods in the experiments.

5. Based on the Z-score, PR distinguishes the misaligned prototypes from others. However, it may make erroneous decisions when dealing with intra-class diversity and inter-class similarity. Regarding these difficult samples, experiments should be conducted to further illustrate the effectiveness of PR.

6. The overview in Figure 3 should be revised to include additional details about your pipeline. I found it confusing to understand the descriptions of PCL and PEB in this figure. Furthermore, it is advisable to carefully review the symbols (e.g., C_n) used in this paper to minimize unnecessary comprehension difficulties for the reader.

[1] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF international conference on computer vision. 2021. [2] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).

• The motivation of the new task seems unclear to me. For example, why holding out some private known classes is significant in the real world? Also, what does this "universal" mean?
• The computational requirement of the proposed method seems to be much more demanding than current SOTAs. For example, as another ViT-based approach, GCD only needs to train the last block of the ViT, but training the whole ViT model seems a necessity for the proposed method, if I understand correctly.
• The writing is somewhat hard to follow. There are many standalone modules and their own acronyms, making the paper not easy to understand. Also, the proposed method seems complicated, and the narration of the methodology is in lack of high-level intuitions and justification of the design choices.

1. The novelty of the proposed method is limited. The proposed method can be decoupled into two parts: 1. Prototypes Alignment. Apart from filtering private-known classes, the rest of this part is very similar to [1]. It is unclear how the proposed method differs from [1] in prototype alignment. 2. Patch-Entropy Balance for Semantic Enrichment. As explained below, $\mathcal{L}_{PEB}$ is not specifically designed for UniNCD, and using unsupervised learning loss in NCD is not new.

2. The motivation for introducing two parts is unclear. First, the Patch-Entropy Balance for the Semantic Enrichment module appears less relevant to the UniNCD task. In particular, the proposed loss can be applied to any representation learning problem. Second, the ablation shows that without $\mathcal{L}_{PEB}$, the SEPNet is worse than many other methods like PromptCAL and DPN w/PR. It seems to indicate that the Prototype Refinement is less necessary for the overall framework.

3. The robustness of the threshold hyperparameter is unclear. What if the split ratio between private-known classes and common-known classes is not 3:1? It is more convincing to provide some experiments when the split ratios changes for validating the effectiveness of the model.

4. In traditional NCD & GCD tasks, there are many experiments conducted on fine-grained datasets such as Standford Cars, CUB-200-2011 and FGVC-Aircraft. Can the authors also provide similar experiment results on these fine-grained datasets?

5. The paper lacks clarity in multiple places:

• What is the cluster label $c^u_{i,j}$ in Eq.6b?
• What does $U_c$ represent in Eq.7?
• The meaning of $x^j_i$ is unclear in Eq.8.
• There are several typos in Eq.12, Eq.14, and Eq.16.
• What is the definition of known category ratio in Appendix E? Does the ratio between private-known classes and common-known classes keep constant while the known category ratio changes? Moreover, what does "PARSE" represent in Fig 7?

[1] Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, QianYing Wang, and Ping Chen. Generalized category discovery with decoupled prototypical network. In Proceedings of the AAAI Conference on Artificial Intelligence, number 11, pp. 12527–12535, 2023.