W4: Analysis on subspace. Thanks for the suggestion! We have now included the qualitative visualization in the revision using Paella [4] to synthesize images based on different (hard & learned) prompts. Please refer to Figure 5-11 in the appendix of the revision or the links below. The visualizations in Figure 5-8 show that soft prompts learned by subspace modelling capture different intra-class variations, such as fine-grained attributes in terms of colour, texture and depiction styles. This explains why our SuPr improves over CoOp, which is stuck with learning only dominating concepts. Also, walking in the subspace across different subspace bases shows interesting transitions along different attributes, showing the wealth of semantic information learned in each subspace, as shown in Figure 9-11.

• Text to image synthesis using different prompts.

• Subspace walking --- color attribute.

• Subspace walking --- texture attribute.

• Subspace walking --- depiction styles.

[4] Dominic Rampas, Pablo Pernias, Marc Aubreville. A novel sampling scheme for textand image-conditional image synthesis in quantized latent spaces. arXiv preprint arXiv:2211.07292, 2023.

W3: Main technical contribution. Please first refer to the response to Reviewer $kdeh$ regarding 'novelty in subspace modelling'. May the take-home messages be refined from our clarification.

Clarification for Message (1): 'dividing soft prompts into subgroups' should not be treated as a main contribution, as this is also just what CoOp-Ensemble requires for modelling. Our first main contribution is subspace modelling with multiple groups of soft prompts. No prior prompt-learning approaches considered subspace modeling to represent categories. This is a great contribution, as learning a subspace classifier leads to better extrapolation/generalization than a prior vector/prototype based class representations such as CoOp. We will show in our revision that a subspace classifier captures the intra-class variability for a class rather than a single dominating point, such that it improves generalization.

Clarification for Message (2): 'regularizing soft prompts with hard prompts' can be enabled in various ways, such as strong alignment between soft/hard prompts. Excessive alignment can harm an adapted VLM's performance on the base classes, as observed in [2,3]. However, our contribution is the specific approach to regularize the modelled soft-prompts-based linear subspaces by forcing them to span hard-prompt embeddings. Our improved VLMs can be tailored well for base classes while maintaining generalization on unseen classes.

Explanation for insights into why the proposed method works: Please refer to the response to the following concern. Also, we compute the prediction scores for all the test samples for some datasets and visualize the top $10$% prediction-confident samples in Figure12-14. To better understand the selected samples, we cluster them into three clusters using K-means. From the results, we can see that the test samples that simulate the vector classifiers are less diverse than the subspace ones, indicating the issue of learning only the dominating concepts of vector classifiers. Among them, we can also see the samples predicted right using subspace classifiers but not by vector classifiers.

• Prediction-confident test samples of Freckled/Petunia/Ostrich for vector v.s. subspace classifiers. Samples with red boxing are wrong predictions, and samples with blue boxing are predicted right with subspace classifiers but not with vector classifiers.

[2] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledgeguided context optimization. In CVPR, 2023.

[3] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In ICCV, 2023.

Response to nXYV

W1: Analysis about "Is subspace modeling useful" in Section 4.4. We understand this is a confusion about our ablation study. Please refer to the response to all for this concern.

W2: SVD for subspace modelling. 1) Restrict $U_c$ orthonormal without SVD: Good point. SVD was used to ensure the orthonormality of $U_c$ during optimization. Placing this restriction on $U_c$ directly was also considered in our preliminary implementation. However, $U_c$ was formed by the soft-prompt embeddings, i.e., it is generated dynamically (as soft prompts update iteratively), making this constraint hard to be in place for $U_c$. We will keep investigating this in our future research. 2) What happens if $U_c$ is not an orthonormal matrix? Indeed, having an orthonormal matrix $U_c$ for linear subspace modelling is unnecessary. We can use the vanilla support points from the soft-prompt embeddings and employ least square linear regression to model a linear subspace as per [1]. We tried this during development, and it performed similarly to SVD - just 0.23\% weaker. Thus we stayed with SVD for simplicity and slightly better empirical performance. 3) The subspace is no longer disentangled/independent when $U_c$ is unconstrained? Yes. However, we would like to clarify that the orthogonality constraint imposed by SVD affects the bases that define each linear subspace, but it does not affect whether the subspaces are orthogonal to each other. IE: There is currently no inter-subspace independence/orthogonality constraint. We also tried regularising the subspaces to be orthogonal to each other during development, but this negatively affected performance.

[1] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning, 2001.

Response to SWVW

W1: Inconsistent Result from EuroSAT dataset. Please refer to the response to all for this concern.

W2: Ablation study. Please refer to the response to all.

W3: Confusing explanations. Sorry for providing any confusing explanations. Please kindly let us know which parts confuse you. We will clarify them in a revision.

Response to kdeh

W1: Performance. Please refer to the response to all for this concern.

W2: Novelty in subspace modelling. Our method SuPr differs from typical subspace modelling methods in the literature by the following aspects: 1) We introduced a novel hard-prompt-based regularization to guide the modelled subspace to span hard-prompt embeddings. This differs from the typical orthogonality regularization commonly used [1-3] to decouple the modelled subspaces of different classes. We also experimented using orthogonality regularization, which induced bad performance. This indicates prompt-based subspace modelling is different from conventional subspace methods. Orthogonalizing them is not beneficial. 2) We proposed an ensembling method to improve our linear subspace modelling by learning multiple linear subspaces with different hard prompts.

[1] You Chong, Daniel Robinson, and René Vidal. Scalable sparse subspace clustering by orthogonal matching pursuit. In CVPR 2016.

[2] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. Adaptive subspaces for few-shot learning. In CVPR, 2020.

[3] Devos Arnout, and Matthias Grossglauser. Regression networks for meta-Learning few-Shot classification. In AutoML 2020.

W3: Missing LASP in Table 3&4. Initially, we noticed a flaw in the LASP paper, as the mean of their reported results did not match their reported mean. Thus, we abandoned their results in Table 3. And, LASP did not provide results for the evaluation for Table 4. Nevertheless, we have now re-implemented LASP for both settings in Table 3/4 and included the results in the revision. The results show that our method consistently holds its superiority over LASP.

Cross dataset transfer (HOS):

Few shot learning (HOS):

Response to Keaq

W1: Comparison with making all parameters learnable. We have now included the results of making all parameters learnable during fine-tuning CoOp and SuPr. The results below show that both methods have gained degraded performance when freeing all parameters for training. We attribute this observation to using limited training samples, making over-parameterized models easy to overfit. However, our SuPr still improves over CoOp by about 3.0\% accuracy in this situation.

W2: Extra training cost? a) Number of trainable parameters: Our method builds on top of training multiple sets of soft prompts divided from the single set of soft prompts from CoOp, i.e., SuPr has the same size of trainable parameters as CoOp. SuPr-Ens has total parameters = the number of ensembles $\times$ number of parameters of soft prompts, which slightly introduces extra parameters. However, the parameter size of soft prompts is tiny; thus, the extra parameter amount is still small enough. b) Computational cost: The following table shows the training time for adapting different models to ImageNet. The results show that our SuPr and SuPr-Ens do not introduce substantially more computational cost from the baseline ($32$mins of CoOp). CoCoOp requires substantially more cost, $25.5$ times longer than the base unit. The next is ProGrad, which has $2.38$ times the cost of CoOp. Our SuPr and SuPr-Ens scale the training time up to $1.5$ and $1.78$ times of CoOp, which is comparable with the recent SoTA methods, KgCoOp ($1.06$) and LASP ($1.31$).