MTMC: Generalized Category Discovery via Maximum Token Manifold Capacity

(1) It is unclear that why need self-attention to get weighted sample centroid. More experiments need to verify the effectiveness of self-attention in equation 1.

(2) The technical novelty is limited. Maximizing the nuclear norm looks like a general method which can be used in many tasks, such semi-supervised classification. The authors need more explanation about the relationship between GCD and nuclear norm.

(3) In many settings, performance improvements are not significant. And recent methods are not compared , such as [1] and [2].

(4) Limited theoretical analysis about the effectiveness of MTMC mentioned in the first contribution.

[1] SPTNet: An efficient alternative framework for generalized category discovery with spatial prompt tuning

[2] Solving the Catastrophic Forgetting Problem in Generalized Category Discovery

1. The authors claim that the tokens of a class can represent intra-class patterns, as shown in Figure 1. However, these tokens are actually features of different patches within an image at the image level. It is not clear how these patch-level features can effectively represent sub-classes at the class level. More detailed analysis or additional experiments are needed to demonstrate this capability.

2. The method for estimating the number of clusters K is not clearly presented. Additionally, in Figure 2, the results are somewhat confusing and do not show an improvement over previous methods in class estimation like GCD/GPC. Do you use the same method with GCD or GPC? Only the best results in Figure 2 should be clearly highlighted in bold.

3. There are some typographical errors in the paper. For example, in line 197, it seems that "equation 4" should be "equation 3."

1.The introduction of manifold capacity to GCD needs to be explained more specifically. Besides, the method relies on maximizing the nuclear norm to measure the class token manifold capacity. However, maximizing the nuclear norm could introduce computational overhead, especially for large-scale datasets. This raises concerns about the practical efficiency of the approach.

2.While the paper discusses the limitations of embedding quality in datasets like CIFAR100 and Herbarium19, a more comprehensive analysis of scenarios where MTMC may underperform would strengthen the evaluation. While MTMC emphasizes enhancing the manifold capacity, the paper does not provide a detailed comparison with other state-of-the-art manifold learning techniques, such as locally linear embedding or manifold regularization methods. This omission makes it difficult to comprehensively assess its advantages in various contexts.

3.The paper could include more detailed ablation studies to explore how variations in hyperparameters, such as λ (the coefficient for MTMC loss), affect performance across different datasets.

1. The presentation is not good. There are a number of confusing or misleading expression / claims.

• While it sounds very interesting to introduce the term [cls], but the reviewer is quite confusing because another concept called the class token manifold extent, or the extend of the sample centroid manifold, is introduced. Is it is essentially the manifold capacity? If yes, why not make the naming consistently?

• In L197-200: There seems a mistake. Eq. (4) cannot be an optimization objective. Should it be Eq. (3)? The reviewer supposed it is the case. However, the interpretation is weird: "when the sample centroid manifold is maximized, it implicitly minimizes each [vis] manifold, thereby enhancing the intra-manifold similarity." From the optimization perspective, the reviewer cannot find the equivalence between maximizing Eq. (3) and minimizing each [vis] manifold.

• Similarly, it sounds also misleading in L230: "After training, the manifold capacity increases, compressing the manifold of each sample within the cluster and promoting repulsion between them."

• L393-395: It is vague to connect the "richer and complete representation within each class" and the "distinction bewween different classes". The reviewer cannot see this point. Just a simple instance. In neural collapse, each class is converged to a singleton but different singleton can also be far from each other. From the perspective of Fisher discriminative score, is it having the most discrinimative ability? On the other hand, if the "valume" of each class is expanded, is there a danger to overlapping?

• L420-423: The reviewer was confused what is trying to express.

• L430-L431: The definition of the norm is unclear. It seems not correct to have $\sum_j \lambda_j =1$.

• L463: Is it correct? The reviewer cannot see that claim.

• L466: Does it really a Frobenius norm, or a nuclear norm eventually computed for and illustrated in Figure 4?

• L484-485: The logic behind the sentence is not direct or clear.

2. Though introducing the von Neumann entropy looks intriguing, it suffers from numerical issue because it is more likely some of the eigenvalues are vanishing. Moreover, the definition depends on the rank $k$. However, the rank $k$ is unkown. It is unclear how the performance changes with respect to the parameter of the rank $k$.

3. It is not clear how to estimate the number of categories $K$.

4. The experiments are insufficient. In Figure 3, while it looks flat with respect to the parameter $\lambda$. However, it was a disguise. The range of the parameter $\lambda$ was set extremely tiny, say, 0.001, 0.002, 0.003, 0.004. What about a large range, e.g., [0,1]? This is another implicit parameter $k$ for the "rank". Is it fair to use the groundtruth value or how to estimate the rank? What about the performance with respect to the parameter $\hat k$, which is the estimated numerically?

1. Although this method performs well in terms of performance, its novelty is somewhat limited. The primary innovation lies in applying nuclear norm maximization to the class tokens in ViT for the GCD task. However, the design does not explicitly address the specific needs of the category discovery task but rather resembles a feature expansion enhancement for the general ViT framework.

2. The complexity of Singular Value Decomposition (SVD) is quite high. Suggestion: If methods such as randomly initialized SVD could be adopted to effectively reduce complexity while maintaining the component’s effectiveness and generalizability, it would further validate the effectiveness of the perspective introduced in this paper.

3. The final loss function appears to focus only on the sum of singular values, which may lead to a concentration of feature expansion in a single direction. Suggestion: It may be beneficial to consider the distribution of singular values or add some constraints to better align the method with the requirements of the category discovery task.