OpenPL: Realistic Evaluation of Prompt Learning for VLM in Open Environments

Question 7: Adding recent works on prompt learning:

A7: We have reviewed GalLoP ("Learning Global and Local Prompts for Vision-Language Models") and conducted some experiments to assess its performance. Due to computational and space limitations, we are only able to present a subset of the results related to GalLoP in the rebuttal. We report the average performance of the algorithm under dynamic distribution shifts and dynamic co-evolution of distribution and class variation across different t-values. However, we believe these results provide useful insights and demonstrate how this method performs in comparison to other prompt learning approaches.

Question 1: About Emerging new classes protocole

A1: We agree that the performance decline as new classes are introduced may appear intuitive due to the increased difficulty with more labels. Our intention was not to suggest that this decline is unexpected, but rather to highlight the degree of performance degradation across different methods, especially under dynamic class shifts, and to evaluate how well prompt learning methods manage this degradation. Indeed, in the final discussion, we also provide an analysis of the reasons behind the strong performance of some of the currently top-performing algorithms.

Question 2: About Varying ratio of classes

A2: We have conducted multiple rounds of experiments and reported both the mean values and the variance of the results to ensure robustness. While it is possible to perform additional rounds of experimentation to further mitigate the variability in CLIP's performance across different categories, the final conclusions remain unchanged. Additionally, in our analysis of the robustness of prompt learning methods, we have carefully considered the impact of CLIP’s inherent performance fluctuations. We have reported the relative improvements and degradations in performance compared to CLIP to provide a more comprehensive understanding of the effects.

Question 3: About Distribution shifts

A3: We would like to clarify that the performance of prompt learning methods varies across different algorithms in response to dynamic distribution shifts. While most prompt learning methods do improve over CLIP on ImageNet variants at each t-value, the improvement may not always be significant in all cases. In the paper, when we say "no significant improvement," we mean that, despite some improvement, the robustness of the methods does not show substantial progress under distribution shifts. Our aim is not to discredit current algorithms, but to highlight the need for designing more robust methods, possibly through approaches like combining with Test-Time Adaptation (TTA).

Question 4: Co-evolution of distribution and class variation: What are the domain shifted datasets used in figure 4?

A4: In our experiments, we trained on ImageNet and used samples from ten smaller datasets as sources for unknown classes and distribution shifts. This approach aligns with your point: we cannot expect prompts learned on a specific and fine-grained dataset like Eurosat to generalize to new classes such as those in ImageNet. This is why cross-dataset generalization benchmarks are typically performed with prompts learned on ImageNet and evaluated on more fine-grained datasets, rather than the other way around.

Question 5: Given the identified weaknesses, do you have any insights or potential solutions for mitigating these issues in prompt learning methods?

A5: At this stage of our work, we have primarily focused on evaluating the robustness of prompt learning methods in dynamic environments, and we do not yet have concrete solutions to address the weaknesses identified. While we have pointed out areas where current prompt learning methods struggle—such as performance degradation in more complex and open environments—offering specific solutions requires further exploration. The combination of prompt learning methods and Test-Time-Adaptation may be a promising direction.

Question 6: About the description of ProDA

A6: Thank you for pointing out the inaccuracy in our description of ProDA. We apologize for the confusion. As you correctly mentioned, ProDA learns an ensemble of prompts in the input embedding space and optimizes a distributional objective over the output space, rather than independently using cross-entropy loss for each prompt. We will revise the description in the related works section to reflect this more accurately.

Thank you for your suggestions. Before addressing your concerns, we would like to take a moment to clarify the motivation and contributions of our work.

Our primary goal is to systematically and comprehensively evaluate the robustness of prompt learning methods in Vision-Language Models (VLMs) within open-world settings. Previous studies in this area have typically been limited to comparing performance in one or two specific scenarios, which often fail to provide meaningful insights into the true robustness of these methods. In contrast, our work extends the evaluation of prompt learning robustness to a series of more complex and dynamic environments.

Rather than focusing solely on simple performance comparisons, we shift the emphasis towards analyzing robustness curves across a range of challenging, real-world scenarios. We also propose a unified framework for evaluating algorithmic robustness under different metrics, which allows us to derive valuable conclusions about the performance and limitations of prompt learning methods in open-world tasks. Through this comprehensive evaluation, we aim to contribute to a deeper understanding of prompt learning methods' capabilities and their potential for adaptation in dynamic environments.

Moreover, the conclusions are also not "obvious". We provide an analysis to highlight the degree of performance degradation across different methods and to evaluate how well prompt learning methods manage this degradation. We also provide an analysis of the reasons behind the strong performance of some of the currently top-performing algorithms. We believe these discussions are insightful and could promote more research direction of prompt learning such as combining with TTA methods to address the continue changing problems.

Therefore, the novelty and contribution of our paper lies in the new metrics, new experimental settings, comprehensive evaluations, and insightful discussion. We believe these have enough contribution for the community.

Dear Reviewer 3ZDh,

We have responded to your comments in detail. As the discussion period will end in less than two day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors

Thank you for your suggestion. We appreciate your thoughtful and thorough comments on our paper.

Weakness 1(a):Does the selection of base classes influence the performance of different methods, for example, by selecting either easy or difficult base classes?

A1(a): Firstly, we acknowledge that the selection of different base classes can influence the performance of various methods. To address this, we utilized three distinct random seeds and reported the variance across multiple trials to ensure robust results. Furthermore, for the random seeds used in our experiments, we explored related text clustering methods, and our findings confirmed that the selected seeds effectively cover base classes of varying levels of difficulty. This approach ensures a balanced representation and strengthens the generalizability of our results.

Weakness 1(b):Is the number of newly added classes the same at each time step $t$?

A1(b):Yes, you are correct. The number of newly added classes remains consistent at each time step $t$.

Weakness 2: The parameter t appears frequently and represents different aspects in each scenario. I suggest using different notation to distinguish these parameters clearly.

A2: We thank the reviewer for pointing out this issue. We will make adjustments to the notation in our paper and add subscripts to the parameter t to distinguish it under different paradigms.

Weakness 3: Since the test process is continuous (with changing classes/distributions), a forgetting score should also be applied to measure performance across methods. For example, performance on the original distribution or class set could be used to assess forgetting [a].

A3: Thank you for the comment, but we cannot fully agree with it. As stated, our benchmarking focuses on prompt learning in the few-shot scenario. In this scenario, trained models do not involve knowledge forgetting. Instead, it encounters both classes or distributions it is "proficient in" and those it is unfamiliar with.

Weakness 4: Why is the initial prompt set to “XXXX” instead of the widely used “a photo of a” in prompt learning? Would different prompt initializations affect the performance?

A4: Thank you for your question. Firstly, we believe that the initialization of the prompt has little impact on the model's performance after multiple iterations. As the number of iterations increases, the influence of initialization diminishes. If it had a significant impact, KgCoOp wouldn't need to specifically set a regularization term to push the prompt towards "a photo of a".

Additionally, we set the initial prompt as 'n' occurrences of "X" to standardize the prompt length across models. Setting the initial prompt to something like "a photo of a" could result in some algorithms requiring the prompt length to exceed a certain value on certain datasets. A longer prompt can harm the model's robustness, making it harder to analyze the performance of prompt learning methods effectively.

Weakness 5: Most of the methods tested are few-shot prompt learning methods. Would the proposed benchmark also apply to unsupervised prompt tuning methods [b, c] and test-time prompt tuning methods [d, e, f]? The authors are suggested to provide some discussion about this point.

A5: Thank you for your insightful comments and suggestions. Our work primarily focuses on few-shot prompt learning methods and aims to provide a robust benchmark specifically tailored to this context. While we appreciate the importance of exploring other types of prompt tuning, such as unsupervised and test-time methods, these are beyond the primary scope of our current work.

Dear Reviewer owVA,

We have responded to your comments in detail. As the discussion period will end in less than two day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors

Question 3: I request the authors to clarify that this is different from Test Time Adaptation setting. Nothing changes continually during test time. They are all just different test scenarios.

A8: Thank you for your observation and for pointing this out. You are correct in your understanding: in our setup, the new classes do not emerge dynamically during the test phase. Instead, $t$ characterizes the difficulty of different test scenarios, where the model is trained on base classes and then tested on a mix of base and new classes (with the assumption that the new class names are known). Our mention of "as $t$ increases, new classes continually emerge while base classes diminish" was intended to describe the design of these distinct test scenarios across different t values, not a dynamic or incremental change during the actual test phase. We agree that this is different from the Test Time Adaptation setting and appreciate your suggestion to clarify this distinction in the paper. We will revise the phrasing to avoid any potential ambiguity.

Thank you again for highlighting this important point.

Question 4: Clarify what it means as $t$ increases in line 163.

A9: Thank you for your comment. The increase in $t$ reflects a progression in the difficulty of the test scenarios. Specifically, as $t$ increases, each class in the test scenario includes more samples from ImageNet variants and fewer samples from ImageNet.

Question 5: In Figure 2, what is the total number of classes fixed to. Is it fixed for all datasets. If so, what if this is changed? A 50 base + 50 new classes is a very different scenario than 500 base + 500 new classes scenario.

A10: Thank you for your question. To clarify, using ImageNet as an example: in the simplest scenario, when $t=0$, our test scenario consists only of 500 base classes. However, when $t=0.2$, the scenario includes both 500 base classes and 100 new classes simultaneously.

Weakness 1: The descriptions of charactezing difficulty of a test scenario using $t$ while is fairly well defined, it can be done in a more detailed manner.

A1: We thank the reviewer for pointing out this issue. We will make adjustments to the notation in our paper and add subscripts to the parameter $t$ to distinguish it under different paradigms.

Weakness 2: Please describe how the datasets are set up for this problem in more detail.

A2: For the scenario with dynamic class changes, we randomly select half of the classes as base classes and the other half as new classes. We then perform multiple experiments, averaging the results and reporting the variance.

For the dynamic distribution shift, since the number of categories in ImageNet variants may be fewer than in ImageNet itself, we use all the categories from the variant for testing.

For the coupled distribution and class variation, due to the significant differences in the number of classes and sample sizes between different datasets, we randomly sample from each dataset before merging them. This ensures that, during testing, the number of ImageNet categories is comparable to that of smaller datasets, and the sample sizes for each category are similar to those of the smaller categories.

For example, when combining ImageNet and ImageNet-A, we ensure that the categories come from the 200 classes of ImageNet-A, and the number of samples from either ImageNet or ImageNet-A does not exceed 16 per class in the test scenario. If ImageNet and SUN397 are combined, the number of categories from ImageNet in the test scenario will not exceed 50.

Weakness 3: All the tables report $Acc(0)$. Based on the definition(line 144, 151, 160), $t=0$ corresponds to evaluation on base task with no new classes or distribution shifts or both. Why is $Acc(0)$ reported in all tables. A suggestion, it is more appropriate to report $Acc(1)$ which corresponds to the most severe case in each scenario which can rather explain your case well.

A4: We report $Acc(0)$ because this metric reflects the adaptability of algorithms to downstream tasks. Many algorithms, such as MaPLe, demonstrate a relatively high $Acc(0)$; however, as seen with MaPLe under classes dynamic changes, their performance degradation in complex environments is significant. This is precisely the aspect we aim to highlight through our analysis.

Additionally, regarding your suggestion to report $Acc(1)$ as a measure of the worst case performance, we have already introduced $WA$ in Table 1 (line 196) as a metric to capture the worst performance of the algorithms. Thank you for raising this point, and we hope this clarifies our motivation and coverage of performance metrics.

Weakness 4: EVM is low for CLIP for most cases. So, can we infer that one is better off using Zero-shot CLIP when one doesn't know the difficulty of test scenario apriori?

A5: Indeed, you make a valid point. In situations where the difficulty of the testing environment is unknown, the performance of Zero-shot CLIP tends to be more stable.

Question 1: In implementation details, it is mentioned that the maximum number of classes and sample size is fixed for all datasets. But, in Section 3.1, line 137, it is mentioned that half the classes from the dataset serve as base classes and the other half as new classes. How is it actually set?

A6: Thank you for raising this question. We realize that our explanation may have caused some misunderstanding. The statement in the Implementation Details section was intended to clarify that, under the Dynamic Distribution Shifts and Dynamic Co-evolution of Distribution and Class Variation scenarios, we ensure consistency in the maximum number of classes and sample sizes across datasets. For instance, in the Dynamic Distribution Shifts scenario, the maximum number of classes from ImageNet during testing is not 1000 but is instead aligned with the 200 classes of ImageNet-A. We will revise the wording in the paper to avoid such ambiguities in the future. Thank you for pointing this out.

Question 2: Are the names of new classes assumed to be known before testing?

A7: Yes, you are correct on this point. Before testing, the model is aware of all the categories within the label space, as this is a classification task. Without the classification labels, one might need to rely on clustering methods instead. However, the prompt learning methods reported in our benchmark are not yet capable of achieving this. We are indeed exploring the issues you mentioned as part of our ongoing research.

Dear Reviewer o46u,

We have responded to your comments in detail. As the discussion period will end in less than two day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors

Question 1: Title of the Paper.

A1: Thank you for your feedback. The term "realistic evaluation" in the title refers to our aim of assessing model robustness under more challenging, evolving test scenarios, which are closer to real-world situations than traditional static test settings. Moreover, it is noteworthy that various similar papers adopted the term "realistic," such as [1], and they also conduct experiments on benchmark datasets such as ImageNet and synthetic settings to simulate more difficult tasks. We believe this is a widely adopted description in the evaluation and benchmark paper.

[1] Realistic Evaluation of Deep Semi-Supervised Learning Algorithms. NeurIPS 2018.

Question 2: Common terminology in the literature on prompt learning is not respected in this work.

A2: Thank you for your valuable feedback. We understand your concern regarding the terminology used in the paper. Our intention in introducing new terms was to provide a more structured framework to evaluate the robustness of prompt learning methods across different types of shifts and challenges.

Question 3: About novelty.

A3: Thank you for your detailed feedback. We appreciate the opportunity to address the concerns raised.

• While it is true that our evaluation scenarios share similarities with prior works, we believe the novelty of our paper lies in the way we structure and analyze the robustness of prompt learning methods. The introduction of parameter t allows us to systematically control and vary the difficulty of the test scenarios across multiple metrics, which provides a more unified and comprehensive framework for evaluating robustness compared to existing works.

• While the decline in performance of prompt learning methods in complex environments is quite evident, we believe that prior experimental conclusions do not adequately explain whether prompt learning methods can mitigate the extent of performance degradation. As Reviewer o46u mentioned: "It is a bold and much needed evaluation of prompt learning methods. The analysis presented could be very insightful for the research community interested in the adaptation of VLMs." We consider the observations we present to be important.

• Furthermore, previous experiments have primarily relied on direct performance comparisons. In contrast, by introducing a simple modification, we are able to unify the robustness analysis of prompt learning across different scenarios using a series of evaluation metrics. We believe this approach is meaningful and provides more comprehensive insights into the performance of prompt learning methods under various conditions.

We hope this clarifies our approach and the contributions to our work. If you have further questions or concerns, we would be happy to discuss them.

Question 4: The authors make some strong negative claims about prompt learning methods while providing no strong evidence, and in some cases, the experiments in the paper itself are not consistent with the claims.

A4: Thank you for your thoughtful comment.

• First, as you noted, our benchmark extends and complements the experimental setups from prior work. The reason earlier algorithms were evaluated under these open-environment scenarios was to demonstrate that prompt learning can be applied to more open settings and exhibit generalization to unseen classes and distributions. Our work aims to provide a more comprehensive analysis of how well prompt learning methods actually perform under such conditions and whether they are robust. Through this effort, we hope to contribute meaningful insights to the development of the field.

• Second, we are not dismissing the contributions of prompt learning methods in improving performance on source (training) datasets or classes, as these improvements are evident. However, this does not necessarily imply that their performance on unseen data (unseen classes or distributions) meets expectations. Our work focuses on systematically evaluating the robustness of these methods through a series of well-defined metrics and highlighting the challenges they still face in unseen scenarios.

• Finally, our goal is not to expect prompt learning methods to maintain unchanged (or even improved) performance in generalized scenarios. Instead, we aim to see a slower performance degradation and more stable behavior in complex scenarios. This aligns with the potential direction for future work, where the focus could be on reducing performance volatility under challenging conditions.

Question 5: About Observation 4

A5: Thank you for your valuable feedback. We acknowledge that Observation 4, while offering insights into prompt learning methods, lacks the level of evidence and experimentation that would strengthen its contribution. We do not have additional experimental results at this time to provide further backing for this observation; this observation is based on initial trends observed in the current experiments, rather than a definitive conclusion. We view this as an area for further exploration in future work.

Question 6: About Dynamic Co-evolution of Distribution and Class Variation

A6: Thank you for your comment. In the Dynamic Distribution Shift paradigm, for example, when mixed with ImageNet-A, the test environment consists of only the 200 classes shared with ImageNet-A. However, within these 200 classes, there are samples from both ImageNet and ImageNet-A distributions. This distribution is unknown and is unified into a series of metrics using the parameter $t$.

In contrast, in the Dynamic Co-evolution of Distribution and Class Variation paradigm, both the distribution and the class set may be unknown. We will clarify these points in the paper for better understanding.

Question 7: Please clarify what refers to in lines 203 and 204 inside the table.

A7: First, we sincerely apologize for the incorrect representation of the formula in line 204. We will revise it as follows:

$\int_{t\in D} Acc_{zs}(t)-Acc(t)dt, D=\lbrace x|Acc(t)< Acc_{zs}(t) \rbrace$

In this context, the parameter $t$ represents different levels of difficulty in the test scenarios. These two metrics are designed to capture the total area where the performance of the prompt learning algorithm exceeds or falls below the zero-shot CLIP performance.

Question 8: Please revise both robustness definitions. AUC, PA, and NA do not depend on $t$.

A8: Thank you for your question. While it is true that $t$ does not explicitly appear in the definitions, this does not mean that $t$ plays no role. In fact, the entire robustness analysis curve (DRC) is constructed based on $t$. The key purpose of the analysis is to evaluate performance changes across different test scenarios as $t$ varies. Although metrics like AUC, PA, and NA are not directly dependent on $t$ in their definitions, they are derived through a global analysis of the performance over varying $t$. This approach provides a unified and systematic way to evaluate robustness under diverse test conditions.

Question 9: It is unclear what kind of robustness the Decay-Gain-Ratio Robustness metric is supposed to measure; please explain in detail what it actually means.

A9: Thank you for pointing this out. The Decay-Gain-Ratio Robustness metric is designed to measure the actual improvement of prompt learning methods compared to zero-shot performance. Specifically, it provides a way to balance and quantify the trade-off between the areas where the performance of prompt learning methods surpasses that of zero-shot CLIP and the areas where it falls behind. By focusing on these regions of the performance curve, the metric aims to evaluate the net benefit of using prompt learning methods in varying test scenarios.

Question 10: It might be beneficial to add grids to the diagrams.

A10: Thank you for the suggestion. Adding grids to the diagrams could indeed make them more readable and allow readers to better interpret the data points and trends. We will incorporate grids into the figures in the revised version of the appendix to enhance clarity and visualization. We appreciate your input.

Question 11: About the writing mistakes and errors in the Introduction and Related Work sections.

A11: Thank you for bringing this to our attention. We appreciate your feedback regarding the clarity of our writing and recognize that vague sentences can hinder comprehension.

To address this:

• We will carefully review the entire manuscript to identify and rewrite vague or ambiguous sentences. Our goal is to ensure that all descriptions are precise and easily understandable.

• Particular attention will be paid to the Introduction and Related Work sections.

Thank you for your patience and valuable input.

Dear Reviewer JRV8,

We have responded to your comments in detail. As the discussion period will end in less than two day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors