A PATCH LEVEL PERSPECTIVE OF PROMPT TUNING

Thank you for your valuable feedback.

None of the referenced works or losses explicitly target the integration of patch-level information or the alignment of consistency across views in a multi-modal context for VL prompting, both of which are central to our method. Our architecture introduces fundamental advancements by integrating patch-level relationships and inter-view consistency across vision, text, and predictions, establishing a robust multi-level alignment.

• Vision: Unlike PromptSRC, which focuses on global consistency and does not consider patch-level relationships, our method introduces intra-view regularization to align patches within the anchor view. To address variability across views, we add an inter-view consistency loss to align patches between a newly introduced augmented view and zero-shot anchor views, enabling superior generalization.

• Text: For inter-view consistency in text, we generate view-specific prompts for the augmented and anchor views, conditioned on patch clusters. This approach ensures sensitivity to finer image details in the text branch, further reinforced by a cross-view text consistency loss.

• Prediction Consistency: For holistic alignment, we combine patch features with the class token to enrich the vision representation before generating predictions. A KL-divergence loss regularizes these predictions, reinforcing consistency across views and complementing the alignment of vision and text features.

Neither PromptSRC nor CoPrompt incorporates patch-level information or addresses inter-view consistency. [1], which was independently developed and released publically on September 26, 2024, just days before the ICLR deadline, could not be incorporated or referenced in our work. While [1] uses clustered patch tokens for text prompts, it relies on a single view and does not account for inter-view variances by maintaining consistency across anchor and augmented views. Additionally, it does not integrate patch information into both text and predictions and underperforms compared to PromptSRC.

The losses described in our method are purpose-built for multi-modal prompting and are distinct from those in referenced works. For instance:

• Eq(5), (11), and (13) maintain inter-view text, vision and logit along with intra-patch-level consistency, addressing finer-grained variability beyond the global consistency discussed in [5].

  • Eq (5): $(\mathcal{L_{\text{intra-view}}} = \sum_{i=1}^{M}(1-\operatorname{sim}(\bar{P}^i_{an} - \hat{P}^i_{an})))$
    This differs from [5], where we use patch-level representations ($P_{an}$) compared to class tokens ($V_{an}$), as employed in [5].
  • Eq (11): $(\mathcal{L_{\text{view-tailor}}} = | \mathrm{stop\_grad}(\hat{T}\_{an}) - \hat{T}\_{aug} |)$ This differs from [5], where we regulate cross-view text features generated from two different views, $\hat{T}\_{an}$ and $\hat{T}\_{aug}$. In contrast, [5] applies the regulation to a text feature conditioned without image information, $T$, and its corresponding zero-shot text feature, $T\_{\text{zero-shot}}$.
  • Eq (13): $(\mathcal{L_{\text{inter-logits}}} = \mathcal{KL\text{-Div}}( \mathrm{stop\_grad}(\mathrm{softmax}(\mathrm{sim}({\hat{T}\_{an}}, {\hat{V}\_{an}}))), \mathrm{softmax}(\mathrm{sim}({\hat{T}\_{aug}}, {\hat{V}\_{aug}}))))$ This equation is applied to predictions generated from the anchor view and the augmented view. The purpose of this loss is to ensure that predictions, which are sensitive to fine-grained features from two different views, learn to attend to similar fine-grained concepts. This is in contrast to [5], which applies this loss to make prompt predictions, $\mathrm{sim}({T\_{prompt}}, {V\_{prompt}})$, similar to zero-shot predictions, $\mathrm{sim}({T\_{zero}}, {V\_{zero}})$, while focusing only on global concept learning.

• Eq (12): $\hat{V} = V + \alpha \star \operatorname{Avg}(\hat{P})$ captures rich patch-level details that are often overlooked in adapter-based methods, such as those in [2]. Specifically, we use patches $\hat{P}$, enriched through a convolutional projection, to enhance the class tokens $V$. This approach ensures that the final vision features $\hat{V}$ become more sensitive to patch-level information. In contrast, adapter-based methods [2] focus on adapting class token features $V$ through a linear projection, which is to improve global semantics.

• Eq(6) $\mathrm{crossview-feat}(\hat{P}^i_{\mathrm{aug}}) = \bar{P}^j_{\mathrm{an}}, \quad j = \arg\max_{k \in \{1, \dots, M\}} \mathrm{sim}(\hat{P}^i_{\mathrm{aug}}, \bar{P}^k_{\mathrm{an}})$ and Eq(7) $\mathcal{L_{\mathrm{inter-view}}} = \sum_{i=1}^{M} \big( 1 - \mathrm{sim}(\hat{P}^i_{\mathrm{aug}}, \mathrm{crossview-feat}(\hat{P}^i_{\mathrm{aug}})) \big)$ differ from FILIP [3], which focuses on global image-text alignment and does not account for inter-view patch consistency between different image crops, text, and predictions. Specifically, we apply Eq(6) and Eq(7) to bring the prompted patch tokens $\hat{P}\_{an}$ closer to the zero-shot anchor patches $\bar{P}\_{an}$. This enforces patch consistency across views. In contrast, in [3], for a given global vision embedding $V\_i$, a text $T\_j$ other than the corresponding text $T\_i$ is considered a positive pair for image-text pretraining. However, they do not account for patch-level relationships across different views, nor do they consider the intricacies of prompt tuning.
• Eq(9) $\{b^1, b^2, \dots, b^{nt}\} = \mathrm{Voronoi-Clustering}(\bar{P})$ and Eq(10) $\hat{G} = \{\hat{g}^1, \hat{g}^2, \dots, \hat{g}^{nt}\}; \quad \hat{g}^i = g^i + b^i$ derive prompts from rich patch-level details, unlike the quantized bias used in [4], achieving more flexible and expressive representations. Specifically, Eq(9) and Eq(10) use Voronoi clustering of patches $\bar{P}$ to generate bias. In contrast, [4] uses a quantized EMA codebook for NLP tasks and does not consider vision information, particularly patches, to generate biases.

This deliberate design bridges a critical gap by addressing patch-level variability and multi-modal inter-view consistency, enabling our method to effectively tackle fine-grained variability while ensuring robust generalization. Our approach distinguishes itself from incremental improvements or adaptations of existing methods, making it both fundamentally novel and impactful.

3. The hyperparameters $\lambda_p, \lambda_1, \lambda_t$ were selected using a methodology similar to [2], as partially detailed in Table 14. Despite the added complexity introduced by view-tailored text prompts, the training time remains comparable to CoCoOp (generates a different prompt for each image) while achieving significantly improved performance. Specifically, our method delivers improvements of 5.18%, 5.72%, and 5.42% on base, novel, and HM metrics, respectively, for the base-to-novel task. These notable gains in performance and generalization highlight the effectiveness and efficiency of our approach.

4. The descriptions for Eq(9), Eq(10), have been incorporated in view-specific prompts para. Additionally, for improved clarity, we welcome specific examples where the use of symbols was unclear, and a new Figure 1 with more detailed caption has been provided. It is worth noting that Eq(1) and Eq(2) are directly taken from PromptSRC [5] as part of the preliminaries.

[1] Mutual Prompt Leaning for Vision Language Models. Int J Comput Vis (2024) [2] Consistency-guided prompt learning for vision-language models. ArXiv, abs/2306.01195, 2023. [3] FILIP: Fine-grained Interactive Language-Image Pre-Training. ICLR 2022 [4] Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding [5] Self-regulating Prompts: Foundational Model Adaptation without Forgetting

We have updated our paper to further refine it:

• We clarified the differences with PromptSRC and CoPrompt in the third paragraph of the introduction.
• The related works section now includes [1], highlighting how it differs from our approach.
• Additionally, we updated the relevant methodology paragraphs for Equations (6), (7), (9), and (10) to emphasize the key differences from [3] and [4].
• We are planning to provide the whole code including vornoi clustering and other parameters used after acceptance.

Thank you for your valuable feedback.

1. Our method introduces a distinctive novelty by going beyond existing global models like PromptSRC and DePT, focusing on integrating patch-level information and enforcing inter-view consistency across vision, text, and prediction levels. Patch integration is critical because, while CLIP provides globally consistent representations across image crops, patch-level information varies significantly. Ensuring that patches within a view align with CLIP’s pretrained knowledge through an intra-view patch loss adds regularization but is insufficient for generalization across diverse datasets. Leveraging patch relationships across crops is essential for promoting consistency, enabling improved generalization across domains (inter-view patch loss). This further allows us to propagate similar cross-view consistency to text features and predictions, making them sensitive to patch-level variations. A summary of novel components is below:

• Vision Consistency: We extend PromptSRC’s global consistency loss to patches within the anchor view, introducing intra-view regularization as an initial step. To enable generalization across diverse views, we incorporate an inter-view consistency loss between a newly introduced augmented and zero-shot anchor views. By regularizing these inter-view patch relations, patch representations are aligned more closely with zero-shot CLIP outputs, reinforcing cross-view consistency for robust generalization.

• Text Consistency: To align text features across views, we generate view-specific prompts for the augmented and anchor views and introduce a consistency loss to ensure alignment between them. Since both views originate from the same image, relying on a single global class token would lead to overly similar prompts and trivial solutions. To address this, we derive unique biases for each view from clustered patches, encouraging meaningful divergence between prompts while maintaining robust alignment.

• Prediction Consistency: To achieve cohesive inter-view alignment, we integrate patch features with the class token to enrich vision representations before generating predictions. This integration captures nuanced details across views, leading to more expressive predictions. A KL-divergence loss is then applied to regularize predictions across views, complementing the consistency established in vision and text levels.

2. Regarding DePT, it introduces a classification head alongside the standard predictions based on text-vision similarity. In our approach, when integrating DePT, we incorporate this classification head during training while maintaining all other components of our method.
3. Kindly refer to the text consistency explanation under Point 1 above for the reasoning behind Voronoi clustering.
4. We have provided results for our method in Table 11 and provided it below, demonstrating its effectiveness when combined with CoCoop and CoPrompt, where it further improves their performance. Additionally, KgCoCoop and CoCoop become almost identical when our method is applied, while CoPrompt is an extension of MaPLE.

4. We did not include Bayesian Prompt in Table 1 due to space constraints, as it does not outperform our method in any of the 11 subtables. For Table 3 on domain generalization, DePT does not provide results. Similarly, KgCoop and ProGrad are excluded from Tables 1 and 2 due to space limitations and their inferior performance compared to our method.
5. We have fixed the issue in Figure 1 in our updated manuscript.

We have updated our paper to further refine it and address your concerns:

• We provided a clearer explanation of the differences with PromptSRC and CoPrompt in the third paragraph of the introduction.
• We improved the explanation of clustering for bias in the view-tailored text paragraphs.
• For DePT, we added more details about its integration in the "Additional Implementation Details" section of the appendix.

Thank you to the authors for detailed rebuttal. The given clarifications are satisfactory. I would like to increase my score to 6.

Thank you for your valuable insights. We appreciate your attention to detail and would like to clarify few details.

1. We appreciate the reviewer’s observation regarding the additional loss functions. Each loss function is purposefully designed to address specific challenges associated with integrating patch-level information and enforcing inter-view consistency across vision, text, and predictions. These losses ensure the method achieves a level of generalization and robustness that previous approaches fail to deliver.

• Vision consistency: Intra-view patch loss regularizes consistency within similar views, while inter-view patch loss enforces relationships across different views. Intra-view consistency adds an additional layer of regularization by aligning with zero-shot CLIP patches. While CLIP is trained to produce consistent global representations across crops, patch-level information can vary significantly. The inter-view patch loss addresses this variability by preserving relationships between patches across views, ensuring closer alignment with zero-shot CLIP outputs and enhancing generalizability. This inter-view consistency is further extended to the text branch and used to generate predictions.

• Text consistency: View-specific text prompts are initialized using patch-level information, and a text prompt consistency loss ensures coherence across views.

• Prediction consistency: Predictions from vision and text features are regularized to cohesively maintain inter-view alignment.

  Previous methods fail to utilize this critical patch-level information, focusing solely on patches for vision branches while ignoring patch relationships across views. This approach is insufficient for improving generalizability, as our extensive ablation studies clearly demonstrate. Each loss component is validated, showing that removing any one significantly impacts performance.

  Additionally, the method’s modular design ensures seamless integration into standard training pipelines. To facilitate accessibility and adoption, we will release our implementation upon acceptance, enabling the community to replicate our results and leverage these carefully designed components to enhance generalization and robustness effectively.

2. To provide experimental justification, we have showcased the following component-level design choices. The results below compare a model with a similar parameter count to ours but trained without incorporating our proposed components. These findings demonstrate that merely increasing the number of parameters does not lead to improved performance, underscoring the critical value of our approach.

We further show individual vision, text, and prediction-level component design choices justification at the experimental level. The results below demonstrate the incremental benefits of our vision-level component design. Starting from the baseline PromptSRC (anchor view), each proposed component contributes to consistent performance improvements. The inter-view patch loss achieves the most significant improvement, highlighting its critical role in enhancing inter-view consistency and overall performance.

The results below demonstrate the impact of our text-level component design. Introducing view-tailored text conditioned on CLS tokens provides a modest improvement, while conditioning on patches further enhances performance. Adding the consistency loss achieves the best results across all metrics, highlighting the importance of enforcing alignment across view-specific text prompts.

The results below demonstrate the effectiveness of our prediction-level component design. Adding TextAdapter provides a modest improvement by leveraging adaptive text-driven features, while ConvProj further enhances predictions by refining vision representations. Finally, the consistency loss achieves the best performance across all metrics, highlighting its critical role in aligning predictions across views, generated from augmented and anchor features.

3. We appreciate the feedback regarding Figure 1 and have revised it to highlight our losses, components. The figure caption has also been expanded to include more comprehensive details, ensuring better understanding and clarity.

4. CasPL [1] is not directly comparable to our model due to its use of the larger CLIP-L/14 model for distilling adaptable prompts. CasPL employs a two-stage process: first, adaptable prompts are learned from the larger CLIP-L/14 model through unsupervised learning; second, a separate set of boosting prompts is trained using prior prompt-tuning methods. In contrast, our approach does not rely on distillation from larger models and instead utilizes CLIP-B/16, aligning with other comparable prompt-tuning methods. Additionally, since CasPL was released on September 26, 2024, very close to the ICLR deadline, it was not feasible to refer it in our paper.

5. Equation 5 is applied to prompted anchor view patches and zero-shot anchor view patches and does not appear to require correction. However, please let us know if we have overlooked any specific issues. Regarding Equation 9, Voronoi clustering can be applied regardless of order, as the prompts are randomly initialized, and the clustering outputs are added as a bias. These prompts are then passed through the attention mechanism, which has order invariance.

[1] Cascade Prompt Learning

We have updated our paper to further address your concerns:

• The experiments mentioned above are included in the appendix, specifically in Tables 15, 16, 17, and 18, with detailed explanations provided in the corresponding paragraphs.
• We revised the related works section to address [1], clearly outlining the differences between our work and the referenced paper. Additionally, we explained why comparing the two would be unfair.

Dear Reviewer,

Thank you for your detailed review of our submission. As the deadline for the rebuttal period approaches, we kindly request your feedback on our responses to ensure that we have addressed your concerns. If further clarification or additional experiments are needed, we would be happy to provide them promptly.

If our responses have resolved your concerns, we would sincerely appreciate it if you could consider updating your scores accordingly.

Your input is critical to the final evaluation of our work, and we look forward to hearing from you soon.

Dear Reviewer,

Thank you for your response and for reconsidering your score. We appreciate your acknowledgment of the effort and potential of our method. We understand your concern regarding the perceived complexity of the algorithm and would like to provide a simple explanation to clarify the necessity and uniqueness of each component.

1. Why are multiple components required?
In few-shot learning, simultaneously regularizing vision, text, and predictions is critical but challenging (e.g., PromptSRC, CoPrompt). A simple regularization strategy would fail to account for the interdependencies between these modalities and variations across views. Integrating patch-level information into all three modalities is the only viable approach to achieve robust alignment and generalization. Without such a design, the model risks underperforming in real-world few-shot scenarios. (Experimental justification: Refer components table in above comment)

2. Why are multiple patch losses necessary?
A single inter-view patch loss is insufficient as it could lead to trivial solutions, such as all patches in one view aligning to a single patch in another view. The intra-view patch loss mitigates this by ensuring that patches within the same view maintain meaningful relationships, complementing the inter-view loss. This dual approach is essential to effectively inject patch-level information while preserving cross-view consistency. (Experimental justification: Refer vision-level table in above comment)

3. Why introduce view-tailored text?
To regularize the text branch and ensure cross-view consistency, fine-grained patch-level information must be incorporated into text prompts. This design ensures that text prompts are tailored for each view, capturing finer image details. Regularizing these prompts across views is essential, as they stem from the same image. A design without this mechanism would fail to incorporate patch-level information coherently at the textual level. (Experimental justification: Refer text-level table in above comment)

4. Why is prediction-level consistency necessary?
As demonstrated by PromptSRC, prediction-level consistency enhances cohesive alignment across modalities. However, applying this consistency solely to class token vision features leads to trivial solutions. By incorporating patch-level information into the predictions and enforcing consistency across predictions from different views, our method ensures meaningful and robust consistency, avoiding oversimplified outcomes. (Experimental justification: Refer prediction-level table in above comment)

Why this approach is essential:
Our design choices are not just logical—they are necessary. Each component addresses a critical gap in existing methods and collectively forms a framework capable of solving the unique challenges of few-shot learning. Simplifying the design further would compromise performance and fail to address key issues such as cross-view consistency and sensitivity to patch-level information.

Impact and Future Directions:
This approach sets a new direction for the field of prompt learning by demonstrating the value of integrating patch-level information for adapting VL models. While the perceived complexity may seem significant, it is an inevitable outcome of addressing these challenges comprehensively. We believe the broader research community will refine and build upon this foundation over time, as has been the case with similar advancements in other domains.

We kindly request that you reconsider your score based on this explanation. Additionally, if there are further points requiring clarification, we would be happy to provide additional details or experiments.

Thank you for your time and thoughtful engagement.

Thank you for your valuable feedback.

1. We would like to highlight that all experiments are conducted with different seeds, consistent with previous methods. These methods report only the mean and do not include the standard deviation, making replicability and comparison of standard deviation challenging. While we acknowledge that the performance on base-to-novel generalization and domain generalization shows a slight margin of improvement, it aligns with the trend of marginal gains observed in previous methods. In contrast, our main results on the base-to-novel task demonstrate a significantly better average improvement.
2. While we agree that the choice of hyperparameters is complex, it enables more effective tuning for individual datasets. For instance, a dataset with significant patch variation can benefit from a higher $\lambda_p$, while larger values for $\lambda_t$ and $\lambda_l$ are better suited for datasets requiring stronger inter-view consistency at a global level.
3. In Section 3.2, the cropping is performed using PyTorch's RandomResizedCrop, which does not ensure alignment with the patch size of 16. This makes it infeasible to calculate correspondence directly based on cropping parameters. Instead, we determine correspondence by identifying the most similar patch at the feature level.
4. The local and global views are defined by the crop ratios of different views. Both the augmented view and the anchor view are global views by default. In Table 10, we aim to demonstrate the effect of having more than one augmented view and the impact of using a much smaller crop size. This experiment does not correspond to Eq(10) since the number of patch clusters is not varied.

We have updated the paper to include a description of using different seeds, detailed in the "Additional Implementation Details" section of the appendix.

Thank you for your valuable feedback.

While we believe that the paper has multiple components, carefully ablation studies have been performed to justify our choices. Evenmore ablations for simple choices could have performed, though we believe it would take exponential time for doing it.

1. Inter-view and intra-view consistency alone do not suffice for achieving optimal performance; their combination proves most effective as shown below. The results without inter-view and intra-view consistency are derived using only a single anchor view. Intra-view consistency adds an extra layer of regularization by aligning with zero-shot CLIP patches. While CLIP is designed to produce consistent global representations across different image crops, patch-level information can vary significantly. To address this, our inter-view patch loss enhances generalizability by preserving the relationships between patches across two views, ensuring closer alignment with zero-shot CLIP outputs. With the intra-view patch loss in place, the inter-view patch loss introduces an additional layer of non-triviality by enforcing consistency across views. Conversely, the presence of intra-view patch loss prevents solutions from straying too far from the foundational CLIP zero-shot patch knowledge, ensuring the inter-view patch loss does not lead to unintended deviations. Additionally, this inter-view patch knowledge is successfully extended to enforce consistency in the text branch and to refine predictions, validating the effectiveness of our approach.

2. We adopt convolutional projection as it provides superior local context compared to transformers. While transformers are effective for capturing global context, we aim to aggregate local contextual information to model finer concepts. Additionally, attention mechanisms in transformers are more memory-intensive than convolutions due to self-attention, even though the runtime remains similar. The gains shown in Table 9 highlight the advantages of this approach over MLP adapters.
3. As highlighted in point 1, the patch-level relationships across views are crucial for enhancing generalizability. Building on this insight, the only viable way to extend this principle to the text modality is by conditioning text with patch-level information. While clustering is indispensable for achieving the fine-grained conditioning required, it neither impacts training efficiency nor imposes additional memory overhead. This makes it a highly effective strategy, yielding significant gains in performance and generalizability.
4. We have applied L2 normalization to both our patch and class embeddings, a technique proven effective in prior works. This normalization enhances the stability and alignment of embedding..
5. Our work is designed to incorporate patch-level information while also leveraging patch relationships across views as a cue for regularizing text, vision features, and predictions. Simply focusing on integrating patches in isolation, whether in vision, text, or predictions, is insufficient, as demonstrated in Table 4. Additional results shown below compare a model with similar parameters to ours but trained without any of our proposed components. These findings highlight that simply increasing the number of parameters does not improve performance, emphasizing the importance of our approach.

We have updated our paper to address your concerns in greater detail:

• We included an additional ablation study in the appendix (Table 15) to justify the effectiveness of our proposed components. In this study, we trained models with identical parameters, both with and without our components. The results demonstrate a significant performance improvement when our components are included.