Don't Paint Everyone with the Same Brush: Adaptive Prompt Prototype Learning for Vision-Language Models

We thank the reviewer for the positive and constructive comments. Please find our response to each comment below:

W1. As stated in the paper, Prototype learning traces its roots to classical models such as K-Nearest Neighbors (Peterson, 2009) and Learning Vector Quantization. Though some new aspects (adaptive attention, decorrelation, etc) are introduced in this paper, the technical novelty seems still limited.

Re W1. Thank you for your comment regarding the technical novelty of our work in the context of prototype learning. We acknowledge the foundational roots of prototype learning in classical models, but we believe our approach brings significant advancements to this domain.

A key innovation in our work is the integration of Vision-Language Models (VLMs) into prototype learning. This represents a considerable shift from traditional methods that predominantly utilize visual data. By leveraging textual descriptions through VLMs, our model is able to generate prototypes that capture a more diverse and comprehensive representation of each class. This is particularly crucial in scenarios with limited visual data, such as few-shot learning, where traditional prototype models may exhibit biases towards certain styles or attributes due to sample scarcity. Furthermore, the introduction of adaptive attention and decorrelation in our approach addresses key limitations in existing prototype learning methods. Adaptive attention allows our model to dynamically focus on the most relevant aspects of each prototype, enhancing the accuracy and robustness of classification. Prototype decorrelation, on the other hand, ensures diversity among the prototypes, preventing overlap and redundancy that often plague traditional prototype-based methods.

These advancements are not merely incremental; they represent a significant leap in the field of prototype learning. Our method overcomes challenges inherent in few-shot scenarios and extends the capabilities of prototype models to handle a wider variety of classes more effectively.

In summary, the integration of VLMs for prototype generation, coupled with adaptive attention and decorrelation, constitute the core technical novelties of our work. These contributions significantly enhance the versatility and efficacy of prototype learning, marking a notable advancement in the field. We believe that these points, as elaborated in our paper, underscore the novelty and significance of our approach in pushing the boundaries of prototype learning.

W2. The paper addresses the adaptive attention of prototypes. This does work but is also somewhat a straightforward point. The paper does not tackle the adaptive attention of words inside a prototype. The importance is verified in the failure case analysis in the experiments.

Re W2. Thank you for your insightful suggestion regarding the exploration of word-level adaptive attention within prototypes. While this is indeed an intriguing aspect, our current approach using CLIP presents certain limitations in this regard. Specifically, the visual-language interaction in CLIP occurs at the embedding space level, which restricts our ability to dissect and analyze the semantic meanings of individual words within the prompts.

Furthermore, in our approach, we do not constrain the length of the prompts to preserve their generative nature. This variability in prompt length poses additional challenges in consistently analyzing word-level attention across different prototypes. Consequently, our research primarily focused on adaptive attention at the prototype level. This decision was guided by the objective to enhance the overall classification performance and robustness by dynamically focusing on the most relevant prototypes for each class.

However, we recognize the potential value of investigating word-level attention and its impact on model performance. In future research, we aim to explore methodologies that could allow for more granular attention analysis, perhaps by employing different models or techniques that can facilitate word-level semantic grounding in the context of visual-language tasks. We believe that such explorations could uncover new insights into the intricate interplay between textual and visual elements in prototype-based learning, further advancing the field.

Q1. What's the learnable part of prompt prototypes in Figure 2?

Re Q1. Thank you for your query regarding the learnable part of prompt prototypes in our model, as depicted in Figure 2. The learnable aspect primarily consists of textual features. For instance, in the ViT-B/16 configuration, these textual features have a dimension of 512. The dimension of the entire learnable weights for prompt prototypes is calculated as the product of the class number, the number of prototypes per class, and 512. These textual features are updated and refined during the training process, dynamically adjusted to better capture the nuances of each class.

We thank the reviewer for the valuable comments and answer them, as numbered, below:

W1. The experiments follow the base-to-new setting in CoCoOp. However, the base-to-new setting is more about generalization ability. Besides, the performance of the base-to-new setting is very sensitive to the hyperparamers, especially for epochs. It is because the performance of this setting requires a balance between alignment and generalization, which can be achieved by reducing the epochs. When tuning the training epochs of CoOp, it will also achieve good performance. It is suggested to use the few-shot learning setting in CLIP and CoOp, which is more fair and supportive to demonstrate the effectiveness of the proposed methods.

Re W1. Thank you for your valuable suggestion regarding the incorporation of few-shot learning settings. We have indeed conducted experiments in this setting and reported the performance in our response to W2.

However, our primary focus on the base-to-new setting stems from our objective to critically evaluate the generalization ability of our methods, particularly in scenarios involving new classes. This setting is essential for understanding how well a model can adapt to novel categories, a key aspect of our research. While methods CoCoOp and MaPLe claim improvements in generalization on new classes, our findings indicate that with descriptive prompts, CLIP can already demonstrate superior performance on new classes than these context optimization methods. We provide a detailed discussion on this validation in our paper.

Furthermore, our analysis specifically addresses the issue of overfitting towards base classes and explores the potential of fine-tuning prototypes for both base and new classes with multiple prompts. This approach is aimed at enhancing generalization in new classes.

Regarding the impact of training epochs on CoOp, our experiments reveal significant insights. As shown in the tables for datasets like DTD, Flowers, and UCF, reducing the number of training epochs does not necessarily improve performance on new classes. In fact, we observe a consistent drop in new class performance with CoOp, highlighting a potential limitation of this method in achieving a balanced performance.

These findings are crucial as they demonstrate the comparative advantages of our approach, particularly in maintaining a balance between alignment and generalization across different class settings. We believe that these insights add significant value to the field and underscore the effectiveness of our methods.

DTD

Flowers

UCF

We thank the reviewer for the valuable comments and answer them, as numbered, below:

W1. As an important contribution, the Attention weighting and L_dec only gain limited performance improvements, which degrades the contribution to the community. The overall compared methods are also very limited.

Re W1. We appreciate your insights on the performance impact of L_dec and the breadth of our comparisons. It's important to note that L_dec improves the Harmonic Mean (HM) performance by 0.31% on ImageNet, which is significant given the dataset's size. We've also tested L_dec's contribution on other datasets, observing larger impacts without tuning its coefficient. This underscores the broader applicability and effectiveness of L_dec across various datasets.

W2.There is some confusion in the layout of tables and images.

Re W2. Thank you for pointing out the layout concerns. We understand that clarity in presentation is crucial for comprehensibility. We will thoroughly review our document to improve the placement and clarity of tables and images, ensuring they align intuitively with the corresponding text sections. We would also appreciate specific instances from the reviewer where the confusion is more pronounced, to better address these areas.

W3. Although using multiple prompts as category prototypes can help capture visual differences, in practice, not every visual sample closely matches each prototype.

Re W3. Thank you for the comment. We recognize that not every visual sample will closely match each of our multiple prototypes. However, this is an integral aspect of our design. The use of multiple prompts as category prototypes is intended to capture the broad spectrum of visual variations within a class.

In our method, the matching mechanism is designed to be flexible and robust. It assesses the affinity of a visual sample to the range of prototypes, rather than seeking a one-to-one precise match. This approach ensures that our model can generalize well across diverse samples within a class, even when direct matches to specific prototypes are not feasible.

Moreover, we have implemented strategies to further refine this matching process. These include the decorrelation loss, max/mean logits balancing that assesses the degree of similarity between samples and prototypes, ensuring that each sample is associated with the most representative prototypes for its class.

Dear Reviewer fKhC,

We would like to express our sincere gratitude for your encouraging remarks about our work, particularly for highlighting aspects such as its intuitiveness, completeness, clear organization, and notable advantages. Your kind words are genuinely appreciated and give us a sense of accomplishment.

In response to the weaknesses you identified in your review, we have provided detailed responses and hope that these have adequately addressed your concerns regarding our paper.

As the discussion phase is about to close in the next 12 hours, we are keenly awaiting any further feedback you may have regarding our responses. We remain prepared to answer any additional questions or provide clarifications as needed, ensuring a thorough and timely exchange of information.

Once again, thank you for your invaluable input and the time you have dedicated to reviewing our work. Your insights are greatly valued.

Best regards, The Authors

We thank the reviewer for the valuable comments and answer them, as numbered, below:

W1. The authors keep claiming that CLIP only uses one prompt, but in CLIP paper section 3.1.4, they discuss how they use 80 prompts to improve the performance without sacrificing test time speed (unlike APPLe which is slower with more prompts). The authors should definitely compare their method to CLIP with 80 prompts as a baseline.

Re W1. We thank the reviewer for pointing out the usage of 80 prompts in the CLIP model. It is true that CLIP's performance on ImageNet benefits from these prompts. However, our work focuses on the limitations of these generic prompts, particularly for fine-grained datasets. These prompts, as limited in the official CLIP repository[1], are less effective for such datasets due to their lack of specificity. For example, prompts like 'a plastic {}' or 'a {} in a video game' do not capture fine-grained distinctions well. It is worth noting that the CLIP performance reported in our paper uses the customized prompts as indicated in the CLIP paper, e.g., “A photo of a {label}, a type of pet.” for OxfortPets.

We acknowledge that these context prompts can indeed improve the performance. We have evaluated the base-to-new setting on ImageNet in the table below (CLIP-80-emb). As for the embedding ensembling method that is raised in question 7. We have tested the performance between embedding ensembling and logits ensembling methods. It can be seen that without fine-tuning, the performance results between the two methods are relatively similar. However, if we want to further fine-tune the textual features, embedding ensembling method tends to overfit to base classes. This phenomenon has been discussed in Figure 3 (impact of prototype number).

As for the test time speed, logits ensembling indeed causes more computation overhead, but only for dot product between the visual and textual features. Because the textual features of the 80 generic prompts also need to be inferred through the language transformer 80 times. The dot product between visual and textual features needs relatively less computational resources.

[1] https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb

W2. The presentation can be improved: It needs to be clarified how training-free works. I think the authors should more explicitly describe it. My understanding is that training-free = 50 prototypes only (the second row in Table 4). Correct me if I am wrong. The description of the training process is also vague. Section 4 omits details on how prototype features are fine-tuned. It seems to me that the text encoder and the prompts are only used to initialize the prototypes. Correct me if I am wrong.

Re W2. Thank you for your valuable feedback regarding the clarity of our training-free approach and the overall training process. We agree that these aspects require more explicit descriptions and have revised our paper accordingly.

The training-free version indeed utilizes only 50 prototypes. These prototypes are initialized by the CLIP text encoder using specifically designed prompts that correspond to the categories or classes in our dataset. This initialization is critical as it provides a foundation for our zero-shot classification approach. This training-free method contrasts with our trained approach, where after the initial prototype initialization, we further fine-tune the prototype features. The fine-tuning process involves adjusting these features to better align with the specific nuances of our dataset, thereby improving the classification performance.

To address the reviewer’s concern about the vagueness in our training description, we have expanded Section 4 to include a detailed account of this process. This includes how the prototypes are initialized, the role of the text encoder and prompts in this initialization, and the subsequent fine-tuning steps, if applicable. Specifically, we explain how the text encoder and prompts serve as a starting point, and how image samples from our dataset are used to refine the prototypes for improved accuracy and applicability.

Dear Reviewer tgxP,

First and foremost, we would like to extend our gratitude for the positive feedback you have provided on our work. Your kind words are both encouraging and appreciated.

In response to the points of weakness you highlighted, we have previously submitted detailed, point-by-point responses. We hope that these have satisfactorily addressed the concerns about our paper.

As the discussion phase is approaching its conclusion in the next 12 hours, we eagerly anticipate any additional feedback you might have on our responses. Should there be any further questions or clarifications needed, we are ready and willing to provide additional information in a timely manner.

Thank you once again for your valuable input and the time you have invested in reviewing our work.

Best regards, The Authors