SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuning

Authors claim that SPT enables the model to focus on parts of objects, but the way it is designed, there are learnable parameters around each patch. Are the learned parameters, after convergence, sparse in nature? Are there more non-zero values around discriminative regions of objects and zero around patches belonging to background? The SPT setup in its current form is not very interpretable and the experiments certainly do not validate the claim that SPT is better than Bahng et al. because it enables the model to focus on object parts. Can this claim be validated/negated if, instead of around a patch, SPT is applied as learnable horizontal or vertical stripes in the image (maintaining the same number of parameters as the original SPTNet).

Thanks for the comments. To validate the sparsity of our learned prompts, we visualize and analyze the learned prompts in Appendix E. It can be seen that after learning, the majority of learned prompts are activated, so the learned prompts are not sparse.

As discussed in General Response -Q1, SPT serves as a learned data augmentation technique while data augmentation is very important for contrastive learning in GCD. Therefore, more activated prompt parameters means a stronger augmentation; while sparsely activated prompt parameters means a weaker augmentation, which is unlikely to enhance the representation learning much. The effectiveness of the learned spatial prompts is also validated by the attention maps, as shown in Fig. 4(b), which demonstrate more diverse attention maps across different heads (top right subfigure).

We follow the suggestion to apply learnable horizontal and vertical stripes separately and the results are shown in Table S2, with the "width" of the prompt, $s$, set to 2, leading to 0.2 "data" parameters. Note that the $s$ for SPT-Net is set to 1, leading to 0.1M learnable `data’ parameters. If we set $s$ for the stripe veraints, there will be 0.03M learnable parameters. Hence, we choose $s=2$ in this experiment for better capacity of the stripe variants. As can be seen from Table S2, Despite having slightly more parameters than SPTNet, these two variantes are still outperformed by SPTNet, indicating that the improvement is due to the design rather than parameters.

Table S2: Evaluation on SSB. Bold values represent the best results.

Table-4 is not presented efficiently in my opinion and needs more attention. From rows 6 and 7, without Global prompting, SPT-S is better than SPT-P. But there is no experiment which uses Global + SPT-S with alternate training in the table.

Thanks for the suggestion. We added the experiment for Global + SPT-S and rearranged rows in Table-4 to show the influence of different components step by step in the revised paper. We create Table S3 here by selecting relevant results from Table-4 in the main paper for convenience. The comparison of SPT-P and SPT-S with global prompting reveals that SPT-P outperforms SPT-S, indicating that SPT-P is also more effective than SPT-S when coupled with the global prompting and alternative training.

Table S3: Evaluation on SSB. Bold values represent the best results.

In Fig. 3(a), for each k, are epochs adjusted accordingly? I believe this is important because authors report that with smaller k, the model underfits. But what about the experiment of training the model parameters to convergence, followed by training the SPT to convergence (k=1). This experiment is crucial to understand why the alternate training is required. Please provide the details of this experiment.

Thanks for the suggestions. We have provided the the suggested experiments and analysis in General Response - Q3.

Lack of a detailed explanation of why the proposed method significantly improves GCD performance…The paper does not thoroughly describe how this relates to the GCD problem and why it leads to better performance.

We thank the reviewer for the comment. Indeed our design is specialized for GCD. Please refer to General Response - Q{1,2,3} for further discussion, more qualitative analysis, and more quantitative study to demonstrate the specialty of our method for GCD.

The reasoning behind why SPTNet outperforms Global Prompt in GCD is supported only by experimental results and not by direct consideration of object parts, with insufficient evidence contrary to Vaze et al. (2022)'s findings.

Thanks for the suggestion. We have followed the suggestion to provide direct consideration of objects parts by qualitative analysis. Please refer to General Response - Q2.

The paper needs analysis of how alternative training induces changes and what benefits it has over end-to-end or completely separate two-stage learning strategies in terms of Expectation-Maximization (EM) learning aspects.

Thanks for the suggestion. We have followed the suggestion to provide additional quantitative analysis in General Response - Q3 to further demonstrate the effectiveness of our EM-inspired alternative learning design. Additionally, in Appendix E and F, we qualitatively demonstrate the impact of alternative training on both data parameters and model parameters. Our results reveal that compared to end-to-end training, alternative training effectively prevents prompt to be zero and thus ensures the diversity of augmentation for contrastive learning. We also provide theoretical analysis from the perspective of EM for the reason using alternative training in Appendix G, which shows that end-to-end training is not equivalent to alternate training. Meanwhile, our alternative training design separates the training model parameters and data parameters into different training steps, resulting in reduced training time and a manageable number of learnable parameters in each training step. Consequently, this approach alleviates the training difficulty.

The paper could strengthen its reasoning and analysis linking the simple strategy proposed to the task of GCD. While it demonstrates a substantial performance increase with a straightforward approach, there is a lack of analysis or reasoning provided in the paper, making it difficult to directly correlate the performance improvements with their causes.

We appreciate the suggestion. We follow this suggestion to include more reasoning and analysis, both empirically and theoretically for GCD. Particularly, we provide more quantitative analysis on the alternative training in General Response - Q3, qualitative analysis on the spatial prompt learning in Appendix E, qualitative analysis on the learned representation in Appendix F, and theoretical analysis in Appendix G. All these consistently demonstrate the effectiveness of our design for GCD. Additionally, the results and discussion in Table S2, S3, and S4 for addressing the comments of Reviewer E8qj further strengthens the reasoning of our design and links to GCD.

It does not seem to be a specialized method for GCD. This reviewer recommends explaining in more detailed reasoning to choose this strategy

We thank the reviewer for the comment. Indeed our design is specialized for GCD. Please refer to General Response - Q{1,2,3} for further discussion, more qualitative analysis, and more quantitative study to demonstrate the specialty of our method for GCD

Presenting the visualization of representation during training at each switch to show the representation is enhanced as the objective of their approach.

Thanks for the suggestion. We follow the suggestion to conduct visualization to verify the effectiveness of our alternate training strategy for representation learning on CIFAR-10 dataset (Please refer to the newly added Appendix F). Specifically, we present the model representations at the $20^{th}$, $100^{th}$, $200^{th}$, and $400^{th}$ epochs before switching . With the increase of epoch, we observe clearer boundaries between semantic categories and increased compactness within each cluster. This confirms that our alternate training strategy leads to more robust representation. In addition, we provide more discussion in General Response -Q1, more qualitative analysis in General Response -Q2, and more quantitative analysis in General Response -Q3.

Their framework can be utilized for zero-shot learning tasks such as open-set recognition and open-vocabulary semantic segmentation, which evaluate the model on both seen and unseen classes, not only GCD. This reviewer agrees that their results show efficiency and efficacy. This reviewer believes that the experiment results on the closely related task strengthen their study.

We appreciate the reviewer's recognition of the efficiency and efficacy of our framework and the suggestions on potential applications of our framework beyond GCD. Indeed, as discussed and analyzed in General Response - Q{1,2,3}, the strengths of our methods are designed on purposely to favor the GCD problem. Other prompt tuning approaches (Jia et al., 2022; Bahng et al., 2022) have been developed to enhance fully supervised learning. However, our patch-based design is to realize the unique ``part transfer’’ insight for GCD. We will consider further exploring the other possible applications of our design in the future work.