Text-Guided Visual Prompt Tuning for Vision-Language Models

W1. The author introduces too many variables and formulas in the method introduction section, which may cause some difficulties in understanding the author's method.

Thank you for your valuable feedback. We acknowledge that the introduction of multiple variables and formulas in the method section might make the explanation dense and potentially challenging to follow. To address this, we will revise the section to improve clarity and accessibility by streamlining the presentation, providing intuitive explanations alongside key formulas. We hope these revisions will enhance the readability and comprehension of our method.

W2. The author's motivation is derived from the analysis of unimodal prompts, leading to the conclusion of using text to guide visual prompts. However, the visual prompts obtained through the author's method are also unimodal. Therefore, I believe it would be beneficial to add the performance of the new visual prompts in Figure 1 to demonstrate the effectiveness of the author's method.

We sincerely appreciate your insightful feedback. In response, we have presented the performance of the optimized visual prompt. The results (seen in response to reviewer a1Dz Q2) reveal that the text-guided visual prompt achieves remarkable improvements on both base and novel classes, underscoring the efficacy of our approach in significantly enhancing the generalization capabilities of visual prompts.

W3. Guiding the representation generation of visual prompts through text has already been applied in MaPLE. Additionally, the author's cross-attention calculation appears to be similar to the parameter-free attention in CALIP[1].

[1] CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention

Thank you very much for your feedback. The concerns you raised are addressed in the common response and my reply to Reviewer a1Dz's Weakness 1. We hope these explanations resolve your concerns.

Q1. Referring to weakness 2, I believe it would be beneficial to include the optimized visual features 1 based on the author's method to demonstrate its effectiveness.

The details of this response can be found in W2.

Q2. In the description of the method, it would be helpful to reduce the introduction of new variables and include pseudocode to aid in understanding the author's approach.

We greatly appreciate your thoughtful feedback and suggestion. In response, we will revise the method description to reduce the introduction of new variables and will include pseudocode in the supplementary materials to facilitate a clearer understanding of our approach. We believe this will enhance the comprehensibility and accessibility of our method.

Q3. Personally, I think that given the lack of novelty in the method presented, the paper should reduce the length devoted to describing the method. Instead, it could analyze what causes the differences in generalization capabilities between visual and text unimodal prompts, or explore whether encoder-only VLMs can be extended to decoder-only VLMs. Such analyses would make the work more impactful.

Thank you very much for your thoughtful comments and suggestions. I believe that the novelty of our method has already been highlighted in the common response and the responses to the previous questions. Furthermore, in our response to Reviewer a1Dz, we conducted a more detailed experimental analysis of the motivation behind our method. These additional experiments further validate that our approach significantly enhances the generalization performance of visual prompts. Nevertheless, we greatly appreciate your perspective, and we agree that further exploration into the generalization capabilities of different modalities, as well as the potential for targeted prompt design in decoder-only VLMs, will be valuable. These areas will certainly be the focus of our future work to further advance and refine the method.

Q2：-This paper uses Figure 1 to express his motivation, aiming to demonstrate that the generalization ability of visual prompts is not as good as that of text prompts; However, only two small datasets, Eurosat (remote sensing) and DTD (texture), were displayed. Both datasets are small and very fine-grained. It is interensting that do similar experiments on large dataset, such as ImageNet, or the mean of all 11 datasets;

Thank you very much for your suggestion. Based on your feedback, we conducted more detailed experiments, and the results are presented in the table below.

The results demonstrate that the generalization performance of visual prompts is weaker than that of text prompts, with the gap becoming more pronounced as the dataset difficulty increases. Additionally, the table also shows the performance of visual prompts after incorporating our proposed TGVP. It can be observed that the generalization performance of visual prompts improves significantly, even surpassing the generalization performance of textual prompts.

Q3: -In some experimental implementation details, such as line 273, the setting of the number of layers for the visual prompt and the comparative experiment on the number of layers are missing; 260 line EMA method, lacking setting of λ hyper parameter;

In our work, the number of layers for both visual and textual prompts is consistently set to the first 9 layers. This configuration aligns with prior works, such as PromptSRC and MaPLe, which also adopt a similar "Deep Prompt" strategy.

For the λ parameter in the EMA mechanism, we uniformly set it to 0.5 across all experiments.

To provide a comprehensive analysis, we also conducted additional ablation studies discussing the impact of different λ values. The results demonstrate that as the parameter λ increases, the strength of textual knowledge guidance intensifies, leading to a significant improvement in the model's generalization performance on novel classes. However, retaining a portion of the original visual prompt token information proves beneficial for enhancing the model's overall performance across both base and novel classes. To balance the model's performance on base and novel categories, we selected λ=0.5 as the optimal value.

W1: Cross-modallity attention has been done in CALIP, and the projection from text prompt to visual prompt in Figure 2 has also been done in MaPLe, which looks a bit like incremental;

[1] CALIP: Zero Shot Enhancement of CLIP with Parameter free Attention (AAAI 2023)

The distinctions between our method and approaches such as MaPLe have been elaborated in detail in the common response. The primary innovations of our method are as follows: our method introduces several targeted improvements to enable more comprehensive cross-modal interaction and improved generalization to unseen categories, which constitute the core contributions of our work:

1. Text Embedding as a Cross-Modal Information Source: For the first time, we propose leveraging the text embeddings—output from the text encoder and rich in high-level semantics—as the textual information source for cross-modal interaction. This ensures a more comprehensive and semantically robust exchange of information.
2. Text-Knowledge Guidance Module: We propose a novel Text-Knowledge Guidance Module, which can dynamically transfer textual knowledge to guide the generation of visual prompts. This makes the visual prompts semantically aware and adaptable to both seen and unseen classes, thereby enhancing the generalization capability of the model.

While our method may share some superficial similarities with the cross-attention design in CALIP, there are significant differences in terms of design objectives, methodology, and implementation details:

1. Difference in Objectives

• CALIP aims to achieve bidirectional feature enhancement between textual and visual features through a parameter-free cross-attention mechanism. In contrast, our method focuses on selecting the most relevant text category knowledge as guidance for a specific vision task. This demonstrates a significant difference in the functional objectives:
  • CALIP emphasizes feature complementarity and bidirectional fusion.
  • Our method is task-driven, prioritizing task-specific relevance

2. Key Design Differences

• CALIP employs a fully parameter-free attention mechanism, directly applying SoftMax to $F_t$ and $F_v$ .In contrast, our method incorporates several critical steps:
  • Top-k Selection: Our method computes a similarity matrix and selects the top-k most relevant text category tokens for each visual prompt. This selection process does not exist in CALIP.
  • Temperature Modulation Mechanism: Our method uses a temperature parameter τ\tauτ to control the sharpness of the similarity distribution, enhancing task adaptability. CALIP does not include such a mechanism.
  • Weighted Feature Aggregation: Our method aggregates top-k guidance using attention weights, producing new task-specific text guidance. CALIP does not involve such a selection and aggregation process

3. Highlighting Innovations

• The unique aspect of our method lies in its "most relevant text category selection" strategy, which is crucial for solving multimodal fusion challenges in specific vision tasks:
  1. Explicit Guidance Selection: Instead of indiscriminately fusing textual and visual features, our method focuses on selecting task-relevant text categories through top-k filtering.
  2. Hierarchical Task Guidance: Our method emphasizes the selection of specific guidance for different levels of vision tasks, whereas CALIP's attention mechanism lacks such hierarchical design.

Q1：-Regarding cross-modalality attention on Idea, CALIP: Zero Shot Enhancement of CLIP with Parameter free Attention (AAAI 2023) has already been done (very similar); This paper lacks this reference;

The details of this response can be found in W1.

We sincerely appreciate your valuable comments and suggestions. Below, we provide detailed, point-by-point responses to address your concerns. We hope these replies effectively resolve the issues you have raised.

W1：The paper seems very similar to MaPLe (Khattak 2023a) and PromptSRC (kHATTAK 2023B) in that they all jointly learn visual and/or text prompts. The related work section briefly mentions them but does not really discuss them adequately.

We have elaborated on this point in detail in the common review section and hope it addresses your concerns effectively.

W2：In the base-to-novel generalization experiment (table 1) the average improvement under the HM column (harmonic mean of base and novel classes) is 1.27% over 11 dataset. However, a closer look reveals that this improvement is mostly due the EuroSAT dataset which shows 8% improvement . Excluding that dataset, the average improvement over the remaining 10 dataset is only 0.35% which is a very marginal improvement .

Regarding the notable performance improvement of our method on the EuroSAT dataset, compared to other image classification datasets discussed in the paper, we identify the following key characteristics of EuroSAT: it contains a limited number of categories (only 10) and poses a greater challenge for zero-shot recognition by pre-trained models (e.g., the CLIP zero-shot classification accuracy is relatively low). We attribute these challenges to the nature of EuroSAT as a dataset of satellite remote sensing images, which are of low resolution (32×32) and exhibit a substantial domain gap from natural images. Many images consist of pure color blocks that are difficult to discern even for humans, further complicating the task for pre-trained models.

Our method addresses these challenges effectively by dynamically injecting text-based category knowledge into the visual prompt, thereby enhancing the intra-class compactness and inter-class separability of the visual features generated by the visual encoder. The effectiveness of this approach is further illustrated in our latest visualizations.

W3：In Table 4 about the domain generalization, the TCP method is missing. Considering that TCP seems to be among top performing methods in other experiments (Table 1-3), including the results of TCP in table 4 will be helpful.

Thank you for your suggestion. We have reproduced TCP and included its results in the domain generalization experiments.

Q1：See my comments above. Also, the proposed method shows a strong performance on the EuroSAT dataset across various experiments. Performance on the other 10 datsets are relatively much lower. A discussion on what is special about the EuroSAT dataset would be insightful.

The details of this response can be found in *W2.

We sincerely appreciate your valuable comments and suggestions. Below, we provide detailed, point-by-point responses to address your concerns. We hope these replies effectively resolve the issues you have raised.

Weakness 1: Thank you for your valuable feedback and suggestions. I believe the novelty of our method has already been clearly outlined in the unified response. Regarding your point about the proposed method's performance being relatively modest, we would like to emphasize that the experimental setups in the field of prompt tuning are already quite challenging. In this context, performance improvements over existing methods are typically modest, often under 5%. However, our approach demonstrates consistent improvement across 11 different image classification tasks, with notable gains of approximately 8% on datasets that differ significantly from natural images, such as EuroSAT. Additionally, in Table 6 of the paper, we compared our method with state-of-the-art approaches that utilize LLMs, and the results show that our method achieves the best performance even when augmented with LLMs. Based on these observations, we believe that the improvements demonstrated by our method are significant within the context of the field.

Weakness2: We sincerely appreciate your insightful feedback and suggestions. As outlined in the experimental section, our method has been evaluated across a diverse array of standard experimental setups in the prompt tuning domain, and we have conducted comparisons with both the most recent and seminal methods in the field. Nonetheless, we acknowledge your valuable point regarding the inclusion of ensemble learning approaches. In response, we have expanded our evaluation to incorporate ensemble learning techniques as additional baselines. We believe that this enhancement will not only strengthen the foundation of our work but also position it more effectively within the broader research landscape.

Q1: What is the function of the "project" component in your model? How would altering its structure impact performance?

In our work, the “project” component is designed to transfer text-embedding, which contains high-level semantic information, into vision prompt token space for further cross-modality interaction. The projector is constructed using a simple "Linear+ReLU+Linear" structure, with its primary structural variation determined by the dimensionality of the intermediate layer, $D_{\text{dim}}$. We conducted ablation studies on $D_{\text{dim}}$, and the results demonstrate that both excessively low and excessively high values for $D_{\text{dim}}$ negatively impact the model's final performance. Based on these findings, we selected $D_{\text{dim}} = 128$ as the optimal parameter.

Q2: Could the authors consider adding the ensemble baselines, such as the WiSE-FT method, to provide a more comprehensive comparison? WiSE-FT: Robust fine-tuning of zero-shot models

Thank you for the insightful suggestion. We will incorporate WiSE-FT into the baseline in the experiment of domain-generalization to provide a more comprehensive and robust comparison in our revised submission.

Q3: Could the authors clarify the definition of "P" in Equation (7)? Additionally, could you explain the process for obtaining T^{topk}{j} and I^{topk}{j}?

In Equation (7), "P" represents the visual prompt tokens. Regarding the selection process for $T_{topk}$, we compute the attention map between the visual prompt tokens and the text embeddings through dot product. From the $N_c$ category text embeddings, we select the $top_k$ categories most relevant to the visual prompt tokens based on this attention map. Subsequently, the text embeddings of these $top_k$ categories are utilized to inject textual information into the visual prompts.

To further validate the motivation and effectiveness of our method, we conducted more extensive experiments across 11 downstream datasets. Below, we present some representative results. It is evident that the text-guided approach enables the visual prompt to achieve remarkable improvements on both base and novel classes, highlighting the efficacy of our method in significantly enhancing the generalization capabilities of visual prompts.

Notably, in these experiments, no prompt was applied to the text encoder; the guiding textual information consisted solely of the text embeddings output by the original CLIP text encoder. This further underscores the exceptional performance of our method in optimizing the effectiveness of visual prompts.