CARPRT: Class-Aware Prompt Reweighting for Pre-Trained Vision-Language Models

Due to the formatting inconsistency between LaTeX and Markdown syntax within the OpenReview system, we find it difficult to include all the math formulas in this response directly. To this end, we have incorporated detailed explanations and derivations in the revision, please kindly refer to the updated PDF for the full information. We appreciate your understanding.

Response to how different priors can be obtained and used (Q2):

Thanks for the question! When prompt weights can be updated continuously, such as in TTA settings, different priors (e.g., uniform, global Dirichlet, or class-specific Dirichlet) define initial beliefs about weight distributions before observing test data. In Appendix G, we supplement a comprehensive discussion on the derivation and impact of various priors (e.g., uniform, global Dirichlet, class-specific Dirichlet) on weight updates in our proposed method. However, since Dirichlet priors would introduce additional steps (e.g., estimating concentration parameters $\alpha$), in our preliminary investigation, we used a uniform prior to keep simplicity. Despite this most simplest setup, our CARPRT prompt reweighting strategy effectively facilitated TTA methods. We leave a more systematic exploration of alternative priors (e.g., Dirichlet) into future work.

Response to corrections and typo (Q3 & Q4):

Thank you very much for pointing them out! Ln 880-883 are indeed typos, which should have been $y_1, y_2, y_3$ for each line. We have also completely revised the proof in Appendix E. Thank you once again for helping us to improve the quality of our manuscript.

Response to correspondence between CARPRT formulation and the probabilistic framework (W1 & Q1):

Thanks for raising this up. We agree with that we should have been set up clearer correspondence between CARPRT formulation and the established probabilistic framework in Section 3.

Concretely, the practical implementation Eqs.(13-16) align with Eqs.(7-11) in the following manner.

Score Calculation. (Eq. (13)) implements the likelihood term $\\Pr(\\boldsymbol{x}\_j|y_c, W, \\mathbb{P})$ from Eq. (11) by defining $s\_{j,i,c} = \frac{\exp(a_{j,i,c} / \lambda)}{ \sum_{y \in \mathcal{Y}} \exp(a_{j,i,c} / \lambda)}$. This formulation align with the EBM in Eq. (1) by

• using cosine similarity $a_{j, i, c}$ as the negative energy term and
• normalizing through softmax to obtain proper probabilities

Weight Calculation. (Eq. (15-16)) correspond to estimating $\\Pr(W | \\mathbb{P}, \\mathbb{D})$ from Eq.(8) through a two-step process.

• Eq. (15) first obtains the pseudo-labels for samples as the empirical estimates $\hat{\Pr}(y\_c | W, \mathbb{P})$ (i.e., Eq. (9)). It then estimates intermediate weights by aggregating scores across pseudo-labeled samples by multiplying the scores $\Pr(\boldsymbol{x}_j|y_c, W, \mathbb{P})$ (i.e., $s\_{j,i,c}$) with $\hat{\Pr}(y_c | W, \mathbb{P})$
• Eq. (16) applies softmax to ensure the resulting weights form a valid probability distribution over prompts for each class, which satisfies the simplex constraint implied by our probabilistic framework.

We appreciate your valuable comments and will merge this into the revision.

Response to influences of pseudo-labels and prompt template pool (W2):

Thank you very much for the question and suggestion.

The influences of pseudo-labels. Upon further investigation, we realized that defining pseudo-label accuracy is non-trivial in our problem setting compared to standard classification tasks. As detailed in Section 4.2, we create a pseudo-label set $\hat{\mathbb{Y}} = \\{ \hat{y}_{j, i} \\}^{m, n}\_{j=1, i=1}$ for each sample $\boldsymbol{x}_j$, where $\hat{y}_c$ corresponds to the class with the highest relevance score for each of $n$ templates. This could lead to multiple pseudo-labels being predicted to the same sample by different templates, challenging the clear definition of conventional pseudo-label accuracy.

As an alternative metric, we "define" a pseudo-label accuracy as the proportion of templates among all templates that correctly predict the ground truth label for each sample. Using this definition, we compute the Pearson correlation coefficient between pseudo-label accuracy and the test accuracy improvement of CARPRT over the Equal Weight baseline across all fine-grained datasets. The resulting correlation coefficient of 0.5021 suggests a weak to moderate positive relationship.

This finding indicates that, while pseudo-label accuracy may contribute to the effectiveness of CARPRT, the relatively weak correlation with test accuracy improvement suggests it may not be the dominant factor. We appreciate this observation and acknowledge the importance of investigating additional factors influencing our method's success.

Regarding your suggestion of "injecting noises into pseudo-labels", we are not that familiar with this strategy and thus have no concrete idea of how to implement this within our setting. Could you please provide more details or references regarding these methods so we can understand your suggestion better?

---

Alternatively, inspired by your suggestion, we conduct additional experiments to investigate whether and how the quality of the template pool matters. The influences of the template pool. We performed additional experiments to further investigate the impact of template quality more comprehensively. We attempted to generate two new template pools (Pool1 and Pool2) using LLMs. The details and results are summarized below:

• Pool0: Original templates (as reported in the paper).
• Pool1: We used Claude 3.5 to generate 300 templates by introducing the ImageNet’s label space as context, consisting of 100 templates each in the following formats: "A photo of a {}, a type of XXX.", "A XXX photo of a {}." and ""A XXX of a {}."
• Pool2: Using Phi 3.1, we generated five descriptive templates per ImageNet category, resulting in a total of 5,000 templates.

Key observations.

1. Template Quality Matters:

   • Pool1: Task-specific templates aligned with ImageNet labels improved both ZPE and CARPRT performance.
   • Pool2: Class-specific templates resulted in the most significant gains for CARPRT, demonstrating its ability to utilize detailed, class-aware information more effectively than ZPE, highlighting the necessity of class-aware prompt reweighting.

2. Consistent CARPRT Gains:

   • CARPRT consistently outperformed ZPE across all template pools, particularly with high-quality, class-specific templates.

This indicates that enhancing template quality indeed yields better performances, which could be one of the promising research directions in the future.

We hope this additional finding could help to address your concern.

Dear Reviewer CBgL,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission6458

Dear Reviewer CBgL,

Following your valuable comments and suggestions, we have provided a detailed reply to address additional questions and concerns raised after our initial reply. We are happy to discuss them with you in the openreview system if you feel that there still are some concerns/questions. We also welcome new suggestions/comments from you!

On the other hand, if our reply has adequately addressed the concerns, we would greatly appreciate it if you could consider updating the review score accordingly.

Thank you once again for your time and effort in reviewing our work. We look forward to hearing from you!

Best regards, Authors of Submission6458

Response to Complexity Concern (W1):

Thank you for raising this concern. We would like to first emphasize that CARPRT introduces no significant computational overhead compared to ZPE, while achieving consistently better performance on almost all datasets. To address your concern, we provide a concise complexity analysis below.

The computational complexity of reweighting methods (including class-wise, prompt-wise, and class-aware prompt reweighting) arises from three main sources: embedding calculations using CLIP, similarity score computations, and weight matrix calculation. We note that the embedding calculation is the most computationally expensive part, as each image and all prompt-class combinations must pass through the image and text encoder. This computation is shared by all prompt reweighting methods. We denote the complexity of the text (image) encoder pass as $\Delta_T$ ($\Delta_I$). Thus, the overall complexity of CARPRT is:

$O(m\Delta_I + nC\Delta_T + mnC + nC) \approx O(m\Delta_I + nC\Delta_T),$

and for prompt-wise reweighting:

$O(m\Delta_I + nC\Delta_T + mnC + n) \approx O(m\Delta_I + nC\Delta_T).$

We note that $\Delta$ (encoder complexity) is significantly larger than $m$ due to the multi-layered Transformer architecture, making encoder computations the dominant cost. Furthermore, $m$ (the number of images) is much larger than $n$ (the number of prompts) and $C$ (the number of classes), underscoring that the main burden lies in the encoder passes rather than subsequent operations.

In conclusion, CARPRT matches ZPE in computational efficiency while delivering superior results, as verified across diverse datasets. We will include this complexity analysis in the revision to clarify this point.

We hope this clarification addresses your concern.

Response to integration to Prompt Engineering methods (W1 & W2):

Thank you for the question. We demonstrate that our method can enhance existing prompt engineering methods. We extend our prompt reweighting strategy CARPRT to ProDa [1], a prompt tuning method that trains multiple invisible text prompts per dataset. Since the problem setting transitions from zero-shot to few-shot, our estimation in Eq. (15) evolves from pseudo-labels to ground-truth labels. Our combined approach demonstrated enhanced performance across various datasets, as shown in the table below:

This indicates that our method can effectively complement and enhance existing prompt engineering techniques. We will include these results in the revised manuscript.

---

In addition to learnable prompts, we have illustrated that our method yields better results once we have obtained a better template pool. Please check this response for your reference.

We hope our additional results on both learnable prompts, and higher-quality prompt templates sort out this question.

[1] Lu, Y., Wei, F., Lu, W., and Yu, D. "Prompt Distribution Learning." CVPR, 2022.

Thank you for your comments. We appreciate your positive feedback on the complexity analysis and your interest in additional statistical evidence. Below, we provide further statistical evidence to support complexity analysis and the experiment results for for higher-quality prompt templates .

The complexity analysis is good. Is there more statistical data to support it?

Thank you for the question. To validate the complexity analysis, we compared the actual computation times for CARPRT and ZPE across datasets. The table below summarizes the results (time in seconds):

For instance, on EuroSAT ($m=8100, C=10$), CARPRT took 11.55 s, and ZPE took 11.35 s, showing negligible overhead. Similarly, on SUN397 ($m=19,850, C=397$), CARPRT (107.50 s) closely matched ZPE (104.42 s). This consistency across datasets confirms that encoder operations dominate computational costs, as reflected in the complexity formulas.

ProDa is the learnable prompts, and Where are the additional results for higher-quality prompt templates ?

Thanks for your comments. We also conduct the experiment over the higher-quality prompt templates. We attempted to generate two new template pools (Pool1 and Pool2) using LLMs. The details and results are summarized below:

• Pool0: Original templates (as reported in the paper).
• Pool1: We used Claude 3.5 to generate 300 templates by introducing the ImageNet’s label space as context, consisting of 100 templates each in the following formats: "A photo of a {}, a type of XXX.", "A XXX photo of a {}." and ""A XXX of a {}."
• Pool2: Using Phi 3.1, we generated five descriptive templates per ImageNet category, resulting in a total of 5,000 templates.

Key observations.

1. Template Quality Matters:

   • Pool1: Task-specific templates aligned with ImageNet labels improved both ZPE and CARPRT performance.
   • Pool2: Class-specific templates resulted in the most significant gains for CARPRT, demonstrating its ability to utilize detailed, class-aware information more effectively than ZPE, highlighting the necessity of class-aware prompt reweighting.

2. Consistent CARPRT Gains:

   • CARPRT consistently outperformed ZPE across all template pools, particularly with high-quality, class-specific templates.

The results indicate that enhancing template quality does yield performance improvements.

We hope these revisions clearly address your concerns.

Dear Reviewer a8Wx,

Thank you very much for raising the score, and we are glad that your concerns are addressed!

We will continuously merge the suggestions and modifications into our revision. We appreciate your time and efforts in helping us to improve the quality of our paper.

Best,

Authors of submission 6458

Response to Limited Performance Gains (W1):

Thank you for your comments. In our paper (Section 5.2 and 6), we have noted that this limited improvement may be associated with the quality of the prompt templates, as well as the intrinsic difficulty and task complexity of ImageNet classification tasks.

In response, we performed additional experiments to further investigate the impact of template quality more comprehensively. We attempted to generate two new template pools (Pool1 and Pool2) using LLMs. The details and results are summarized below:

• Pool0: Original templates (as reported in the paper).
• Pool1: We used Claude 3.5 to generate 300 templates by introducing the ImageNet’s label space as context, consisting of 100 templates each in the following formats: "A photo of a {}, a type of XXX.", "A XXX photo of a {}." and ""A XXX of a {}."
• Pool2: Using Phi 3.1, we generated five descriptive templates per ImageNet category, resulting in a total of 5,000 templates.

Key observations.

1. Template Quality Matters:

   • Pool1: Task-specific templates aligned with ImageNet labels improved both ZPE and CARPRT performance.
   • Pool2: Class-specific templates resulted in the most significant gains for CARPRT, demonstrating its ability to utilize detailed, class-aware information more effectively than ZPE, highlighting the necessity of class-aware prompt reweighting.

2. Consistent CARPRT Gains:

   • CARPRT consistently outperformed ZPE across all template pools, particularly with high-quality, class-specific templates.

The results indicate that enhancing template quality does yield performance improvements.

Plans for Places365 Evaluation. We acknowledge the importance of testing CARPRT on a diverse, large-scale dataset like Places365. However, preparing (download and setup) the dataset is currently underway (it takes more than 1 week), and we will report these results in the revision once we obtained them.

Response to Pseudo-Label Accuracy Dependency (W2):

We appreciate your insightful observation. Upon further investigation, we realized that defining pseudo-label accuracy is non-trivial in our problem setting compared to standard classification tasks. As detailed in Section 4.2, we create a pseudo-label set $\hat{\mathbb{Y}} = \\{ \hat{y}_{j, i} \\}^{m, n}\_{j=1, i=1}$ for each sample $\boldsymbol{x}_j$, where $\hat{y}_c$ corresponds to the class with the highest relevance score for each of $n$ templates. This could lead to multiple pseudo-labels being predicted to the same sample by different templates, challenging the clear definition of conventional pseudo-label accuracy.

As an alternative metric, we "define" a pseudo-label accuracy as the proportion of templates among all templates that correctly predict the ground truth label for each sample. Using this definition, we compute the Pearson correlation coefficient between pseudo-label accuracy and the test accuracy improvement of CARPRT over the Equal Weight baseline across all fine-grained datasets. The resulting correlation coefficient of 0.5021 suggests a weak to moderate positive relationship.

This finding indicates that, while pseudo-label accuracy may contribute to the effectiveness of CARPRT, the relatively weak correlation with test accuracy improvement suggests it may not be the dominant factor. We appreciate this observation and acknowledge the importance of investigating additional factors influencing our method's success.

An iterative refinement of pseudo-label prediction and weight estimation. Importantly, we find that we can leverage the predicted prompt weights to update pseudo-labels iteratively. Here, pseudo-labels are defined differently from those in the main text. Specifically, for each sample, we compute a single pseudo-label by aggregating predictions across all prompts using the predicted prompt weights. This enables dynamic refinement of both the pseudo-labels and the class-specific weights.

The table below reports the performance comparison of introducing iterative refinement to CARPRT (denoted by ICARPRT).

We observe that this iterative strategy greatly enhances overall performance across almost all datasets. We will include these updated results in the revision.

Response to Distribution Modeling (Q1):

Thank you for your suggestion. We defined Energy-Based Models (EBMs) in Lemma 1 because CLIP's logits naturally align with the EBM likelihood formulation. To our knowledge, alternative distribution modeling like Gaussian Mixture Models (GMMs) might not yield better results, as EBMs are versatile and flexible enough (because EBM does not force the likelihood to be normalized, so it is much less restrictive in functional form [1]) to capture a wide range of distributions, including those representable by GMMs. In other words, EBMs can be seen as the generalization form of GMMs. Furthermore, we are not that familiar with the Local Manifold models you mentioned. Could you please provide more details or references regarding these methods so we can understand your suggestion better?

Response to Imbalance Datasets (Q2):

Thank you for your thoughtful question. To evaluate CARPRT under class imbalance, we manually construct an imbalanced CIFAR-10 dataset following previous work [2], using an imbalance ratio $\beta$ that quantifies the degree of imbalance, defined as $\beta = N_{\max} / N_{\min}$, where $N_{\max}$ and $N_{\min}$ are the number of samples in the majority and minority classes, respectively. We experiment with different levels of class imbalance by setting $\beta = 10$, $\beta = 50$, and $\beta = 100$.

Table: Performance Comparison on CIFAR-10 under Different Imbalance Factors

We observe that CARPRT consistently yields the best performance in comparison with the Average baseline and ZPE, across all imbalance levels. Compared to the Average baseline, the gains are more evident as the imbalance increases, highlighting CARPRT’s adaptability to such conditions. We also notice that the performance gap between CARPRT and ZPE narrows under severe imbalance. This gain reduction likely arises from ZPE's reliance on a single global weight for the entire dataset, which seems to remain effective even when some classes contain very few samples (e.g., 10 instances). In contrast, CARPRT's class-specific weight estimation could be less reliable for extremely underrepresented classes.

These results highlight CARPRT's robustness and adaptability in imbalanced settings. We will include these findings in the revised manuscript to address this important aspect.

[1] Song, Y., and Kingma, D. "How to Train Your Energy-Based Models." In ArXiV, 2021

[2] Cao, K., Wei, C., Gaidon, A., Arechiga, N., & Ma, T. "Learning imbalanced datasets with label-distribution-aware margin loss." NeurIPS, 2019.

Dear Reviewer 6Qg1,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission6458

Response to Complexity Concern (W1):

Thank you for your comments. We will first clarify that: CARPRT does NOT introduce significant computational overhead compared to ZPE. Please see the complexity analysis below for detailed reasons.

The computational complexity of reweighting methods (including class-wise, prompt-wise, and class-aware prompt reweighting) arises from three main sources: embedding calculations using CLIP, similarity score computations, and weight matrix calculation. We note that the embedding calculation is the most computationally expensive part, as each image and all prompt-class combinations must pass through the image and text encoder. This computation is shared by all prompt reweighting methods. We denote the complexity of the text (image) encoder pass as $\Delta_T$ ($\Delta_I$). Thus, the overall complexity of CARPRT is:

$O(m\Delta_I + nC\Delta_T + mnC + nC) \approx O(m\Delta_I + nC\Delta_T),$

and for prompt-wise reweighting:

$O(m\Delta_I + nC\Delta_T + mnC + n) \approx O(m\Delta_I + nC\Delta_T).$

We note that $\Delta$ (encoder complexity) is significantly larger than $m$ due to the multi-layered Transformer architecture, making encoder computations the dominant cost. Furthermore, $m$ (the number of images) is much larger than $n$ (the number of prompts) and $C$ (the number of classes), underscoring that the main burden lies in the encoder passes rather than subsequent operations.

In conclusion, CARPRT matches ZPE in computational efficiency while delivering better results, as verified across diverse datasets. We will include this complexity analysis in the revision to clarify this point.

Response to Limited Performance Gains (W2):

Thank you for your comments. In our paper (Section 5.2 and 6), we do have noted that this limited improvement may be associated with the quality of the prompt templates, as well as the intrinsic difficulty and task complexity of ImageNet classification tasks.

In response, we conducted additional experiments to investigate the impact of template quality more comprehensively. We attempted to generate two new template pools (Pool1 and Pool2) using LLMs. The details and results are summarized below:

• Pool0: Original templates (as reported in the paper).
• Pool1: We used Claude 3.5 to generate 300 templates by introducing the ImageNet’s label space as context, consisting of 100 templates each in the following formats: "A photo of a {}, a type of XXX.", "A XXX photo of a {}." and ""A XXX of a {}."
• Pool2: Using Phi 3.1, we generated five descriptive templates per ImageNet category, resulting in a total of 5,000 templates.

Key observations. Compared with Pool0

• Pool1 targets more task-specific information (templates are generated with respect to the ImageNet label space), leading to performance gain for both prompt reweighting strategies ZPE and CARPRT.
• Pool2, as the generated templates contain more class-specific information, CARPRT exhibits greater performance gains. Its class-aware prompt reweighting may have effectively utilized the class-specific descriptive information, outperforming ZPE's class-independent approach.

The results indicate that enhancing template quality does yield performance improvements.

To further substantiate the contribution of our work, We additionally explore whether our method can benefit broader research communities. We extend our prompt reweighting method to ProDa [1], a prompt tuning method that trains multiple invisible text prompts per dataset. Since the problem setting transitions from zero-shot to few-shot, our estimation in Eq. (15) evolves from pseudo-labels to ground-truth labels. Our combined approach demonstrated enhanced performance across various datasets, as shown in the table below:

Key Observations. These results demonstrate that CARPRT effectively complements ProDA, delivering tangible improvements in most datasets and a significant overall boost in the average performance from 78.2 to 80.2. By integrating CARPRT with an established prompt engineering technique, we show its versatility and potential to enhance existing prompt engineering frameworks, demonstrating that CARPRT can not only help with the usage of visible (natural language) prompts but also facilitate methods based on invisible (text embedding) prompts.

We will also include these results in the revision to emphasize the broader relevance and robustness of our method.

[1] Lu, Y., Wei, F., Lu, W., and Yu, D. "Prompt Distribution Learning." CVPR, 2022.

Response to Hyperparameter Sensitivity (W3):

Thank you for your thoughtful feedback.

In the zero-shot classification setting, where only test data is available, conventional hyperparameter selection (typically on a validation set) is not feasible due to the absence of training and validation sets. Our goal is to identify hyperparameters that exhibit robust and consistent performance across diverse datasets.

To address your concern, we conducted further experiments regarding hyperparameter sensitivity, on three representative datasets: Caltech, DTD, and Oxford Pets. As shown in the table below, we experimented with various temperature settings. Among these results, it is evident that a temperature of 3.0 provides a balance of performance across the datasets, making it a reasonable choice given the constraints in zero-shot setting.

We recognize the importance of comprehensive hyperparameter sensitivity analysis and will include results for all datasets in the revision.

We hope these additional results help to address your concerns.

Dear Reviewer rVkh,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission6458

Dear Reviewer rVkh,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses and let us know if they sufficiently address your concerns?

If anything requires further clarification, we would be glad to provide additional details. We look forward to heading from you.

Best,

Authors of Submission 6458

Priority rationale of conducted experiments

The experiment for new template pools is conducted on ImageNet.

Thank you for your comments. The experiment for new template pools is conducted on ImageNet to specifically address the concern originally raised in W2 about "limited improvement observed in Table 2". Since this issue refers to ImageNet results, we prioritized these additional experiments on this dataset to provide a direct response.

The experiment of ProDA+CARPRT is conducted on the other 8 datasets.

Thank you for raising this concern. The ProDA+CARPRT experiments were carried out on the other 8 datasets because both ProDA (in their original paper) and our method (in this paper, Table 1) were evaluated on these datasets. Also, we kindly note that these fine-grained datasets are commonly used benchmarks in the prompt tuning literature. These additional experiments aim to demonstrate that CARPRT enhances performance not only for visible prompts but also for approaches using invisible prompts, further demonstrating its potential contribution to a broader range of communities beyond zero-shot classification.

Ongoing comprehensive experiments

We acknowledge the need for more comprehensive evaluations and thus are conducting additional experiments, including (1) ProDA+CARPRT for ImageNet and (2) generating high-quality template pools for fine-grained datasets, following your queries.

However, it is important to note that, while CARPRT itself introduces negligible computational overhead, generating high-quality template pools using LLMs and training ProDA's learnable prompts require the availability of GPU resources. Owing to resource limitations and queuing for the computing server in the university, these experiments are currently underway. We will report the results as soon as they are available.

Selection of temperature hyperparameter

Moreover, the experiment of temperature settings is conducted only on 3 datasets, which may cause concerns regarding the generalization ability.

This table summarizes the hyperparameter analysis results across all fine-grained datasets. It is important to emphasize that in zero-shot classification, where only test data is available, conventional hyperparameter selection is inherently difficult due to the lack of training or validation data. In our paper, a temperature of 3.0 was chosen as it consistently yields strong results across datasets. While it may not be optimal for each dataset, it provides a practical and generalizable option under the constraints of the zero-shot setting.

We hope this response addresses your concerns and clarifies the priority rationale of our reported supplementary analysis given the limited rebuttal period. Further results will be provided as soon as ongoing experiments are completed. Thank you for your constructive feedback.

Dear reviewer rVkh,

During our attempt to quickly reproduce the implementation of ProDA and report additional results within a very short period, we encountered some minor issues in our implementation that required adjustments. As a result, we have updated our findings. The corrected results are shown below:

In the above table:

• RN50-ep50 refers to results using CLIP-ResNet50, trained for 50 epochs.
• VIT16-ep50 refers to results using CLIP-ViT-B/16, trained for 50 epochs.

The corrected results demonstrate that CARPRT consistently outperforms ProDA under a fair setting across all datasets. This consistent improvement highlights CARPRT’s robustness and its effectiveness in complementing ProDA to enhance performance in various scenarios.

We sincerely apologize for the oversight and any confusion caused. Thank you for your understanding and continued support.

Additionally, we have, per your request, completed all the additional experiments, including those for ImageNet, and included all the corresponding results in the revised manuscript. We hope these new experiments effectively address your concerns.

Best regards,
Authors of Submission6458

Response to Reviewer a8Wx

1. Computational Complexity.

• Concerns: Does CARPRT introduce computational overhead compared to ZPE?
• Response: We clarify that no significant overhead is introduced by CARPRT compared to ZPE. We conduct a complexity analysis, showing that both methods rely on the computationally intensive embedding step via the CLIP encoder, which dominates the cost. We also report actual evaluation elapsed time for both CARPRT and ZPE.

2. Effectiveness with Learned Prompts

• Concern: Is CARPRT effective for learned prompts?
• Response: We also supplement experiments that integrate CARPRT with learnable prompts method (ProDA), demonstrating CARPRT can facilitate both training/non-training prompt engineering scenarios, regardless of general datasets (Imagenet) or fine-grained recognition datasets. We append these results in Appendix K.

Response to Reviewer CBgL

1. Probabilistic Formulation and Correspondences

• Concern: The probabilistic correspondences are not explicitly presented. Can a mathematical derivation be included to assist understanding?
• Response: We include detailed mathematical derivations in Appendix F, connecting implementation (Eqs. (13-16)) to the probabilistic framework. We correct the presentation error in Eq. (13) to align the theoretical framework with our implementation.

2. Robustness to Noisy Pseudo-Labels

• Concern: Is CARPRT more sensitive to noisy pseudo-labels than ZPE?
• Response: We analyze the pseudo-label accuracy obtained across different datasets and calculated its correlation with CARPRT's performance improvement over ZPE, demonstrating a weak-to-moderate correlation.

3. Impact of Template Pool Diversity

• Concern: How does template pool diversity affect CARPRT?
• Response: We supplement experiments by introducing LLM-generated template pools with increased template diversity, under which CARPRT consistently outperforms ZPE. We add a section (Appendix J) to include the impact of template quality.

4. Applying Priors in Appendix D

• Concern: How can the priors discussed in Appendix D be obtained and used in Test-Time Adaptation?
• Response: We expand Appendix G to provide practical examples and guidelines for deriving and applying these priors, particularly for Test-Time Adaptation.

5. Proposition 3 Clarifications

• Concern: In Proposition 3, why do all the three (Ln 880-883) have $y_1$ as their variates? How are Ln 886-889 derived?
• Response: We revise the proof Proposition 3 to include step-by-step derivations in Apendix E and clarify the role of $y_1$ and how the related equations are derived.

---

We believe these responses have comprehensively addressed all reviewer concerns, including follow-up issues raised by reviewer rVkh. The revision incorporates additional experiments, elaborated theoretical derivations and clarifications. The reviewers (6Qg1, a8Wx, CBgL) have not raised further issues and have indicated scores leaning towards acceptance.

Once again, we appreciate the reviewers’ time and efforts, which have greatly enriched the quality and clarity of our work.

Best regards,

Authors of submission 6458