Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models

Thank you for the constructive and elaborate feedback. Please find our response below:

1. Comparison with calibration methods for CLIP [W1]

Following the reviewers' suggestion, we compare it with two recent calibration approaches for CLIP including post-hoc scaling (ZS-TS [1]) and regularization-based training (CaRot [2]). For ZS-TS, we perform the post-hoc calibration on the fine-tuned model using ImageNet-1k and evaluate the learnable temperature $\tau$ on both the base and new classes. For CaRot, we easily incorporate DOR with it. We report the average ECE on the base-to-new datasets.

The results in the table show that the optimized temperature $\tau$ significantly improves the calibration of the fine-tuned CLIP on new classes, but such calibration in ZS-TS can not be transferred to the base classes. In contrast, our approach effectively mitigates the miscalibration issue on new classes while maintaining the calibration performance on the base classes. We add the detailed results in Appendix I and Table 15.

For CaRot, we present the results in the table below. We find that our DOR method can improve the calibration of CaRot on both base and new classes. This demonstrates that DOR is a flexible regularization strategy compatible with various fine-tuning methods. We add the detailed results in Appendix I and Table 16.

2. The generalizability of DOR [W2]

We apologize for any potential misunderstanding. We clarify that the columns in Table 1 denote different measures of calibration error, instead of different cases. Notably, our method still achieves comparable performance in those non-bold results, instead of underperforming the baselines. In fact, we only bold the results that are significantly improved. In addition, we focus on the overall average performance ("HM") in the base-to-new evaluation, so counting the number of the bold results in all columns is meaningless. We add the description in Line 313.

Additionally, we add more results from two other prompt tuning methods including PromptSRC and Coprompt in Table 1 of the revised manuscript. The results show that our approach significantly improves calibration on new classes while preserving base class calibration, which leads to better average performance. In summary, the results demonstrate our method's flexibility as a regularization strategy compatible with various fine-tuning techniques.

3.Ablation on lexical databases [W3]

Thanks for your great suggestion. To evaluate whether different lexical databases significantly affect the results, we performed an ablation study on the choice of lexical databases. Specifically, we consider two additional textual databases: CLIP’s vocabulary and ConceptNet 5.7. For CLIP’s vocabulary, it includes 49,407 characters and words. For ConceptNet, we use raw sentences and filter out those exceeding CLIP's input limit (77 tokens). Finally. it consists of 705,662 short sentences.

As shown below, we find that all databases can achieve better results than the baseline. Among the three databases, using WordNet achieves the best calibration performance. Additionally, we observed that short sentences may not perform as well as prompts like “a photo of [class].” We add the detailed results in Appendix G.4 and Table 12.

[1] LeVine W, et al. Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models. ICLR workshop, 2023

[2] Changdae O, et al. Towards Calibrated Robust Fine-Tuning of Vision-Language Models. NeurIPS, 2024

6. Detailed clarification about the contribution of DOR [W2, W3]

We clarify the contribution of DOR in detail here since the reviewer is curious about why outliers would be preferable to these existing regularizations and how DOR breaks the calibration trade-off.

To illustrate why outliers can address the calibration trade-off, we can understand it from the perspective of gradient conflicts [3]. Specifically, we calculate the cosine similarity of the gradients of the prompt between $L_{\mathrm{reg}}$ and $L_{\mathrm{ce}}$ to reflect the degree of gradient conflicts. we conduct an ablation of recent regularization item in Appendix B.2. We can decouple CLIP's fine-tuning objective as $L_{\text{clip}} = L_{\mathrm{ce}} + \lambda \cdot L_{\mathrm{reg}}$, where $L_{\mathrm{ce}}$ is the cross-entropy loss for classification, and $L_{\mathrm{reg}}$ denotes the regularization term. Previously methods introduce regularization terms include $L_{\mathrm{kg}}$ (KgCoOp), $L_{\mathrm{distill}}$ (CoPrompt), and $L_{\mathrm{scl}}$ (PromptSRC).

As shown in Figure 8 of the revised manuscript, the gradient conflict distributions for KgCoOp, CoPrompt, and PromptSRC are predominantly within the range of $[-1, 0]$, which indicates a conflict with the fine-tuning objective $L_{\mathrm{ce}}$. Considering that CoOp with vanilla $L_{\mathrm{ce}}$ is an efficient calibrator for base classes (See Figure 1 (a)), we can infer that these regularization terms may hinder the calibration performance for base classes. As an alternative, we observe that the gradient conflicts for DOR are primarily concentrated within the range $(-0.1, 0.1]$. Compared to previous regularization terms, it leverages outliers to construct the regularization term and shows significantly fewer conflicts in the $[-1, 0]$ range. The phenomenon supports our claim that outliers can be used in regularization without interfering with the original fine-tuning objective for classification.

To further illustrate the actual impact of outliers, we conducted an ablation for the proportion of outliers in the regularization item. Considering KgCoOp uses a fixed number of base classes as the regularization term, we progressively replaced these texts with textual outliers at different proportions $p_{\text{out}}$ from 0.1 to 1.0. We report the average performance on base-to-new datasets.

As shown in the table below, as the proportion of outliers increases, the miscalibration of base classes begins to decrease while the performance of new classes remains largely unaffected. The results strongly support our claim that the outlier can effectively mitigate the miscalibration issue on new classes while maintaining the calibration performance on the base classes. We add the detailed discussion in Appendix B.2 and Figure 8-9.

[1] Khattak M U, et al. Self-regulating prompts: Foundational model adaptation without forgetting. ICCV, 2023

[2] Roy S, et al. Consistency-guided prompt learning for vision-language models. ICLR, 2024.

[3] Shi G, et al. Recon: Reducing conflicting gradients from the root for multi-task learning. ICLR, 2023.

We appreciate the reviewer for the insightful and detailed comments. Please find our response below:

1. Comparison with related methods [W1]

Thank you for the suggestion. Before providing the new results, we clarify that PromptSRC and CoPrompt do not study the calibration issue and do not evaluate the performance of confidence calibration throughout their paper.

As suggested by reviewers, we present (in the table below) new results with recent prompt-tuning methods: PromptSRC and CoPrompt. The results in the table show that our DOR can consistently improve calibration on new classes. Compared with the vanilla baselines, DOR can result in better overall calibration performance, while maintaining the calibration performance on the base classes. Furthermore, as shown in Table 2 of the revised manuscript, our method can also boost their generalization on new classes. We present the detailed calibration results in Table 1 and Appendix F of the revised manuscript.

2. The contribution of DOR [W2]

Again, we clarify that PromptSRC and CoPrompt do not study the calibration issue and do not evaluate the performance of confidence calibration throughout their paper. These two papers do not affect the novelty of this work. Indeed, the trade-off of accuracy between base and new classes is widely discussed in the literature of prompt tuning, but their discussions do not include confidence calibration. In addition, We clarify that we do not claim the problem statement as our primary contribution. In this work, we provide an insightful analysis of how current prompt-tuning methods affect confidence calibration. Then, we propose DOR regularization to effectively mitigate the miscalibration of fine-tuned CLIPs.

3. The analysis for using outliers [W3]

Thanks for your insightful question. In the analysis of Section 3, we show that the textual divergence is the key to the calibration performance of fine-tuned CLIP. Therefore, we regularize the textual divergence of new classes without restricting those of base classes, to preserve the calibration of zero-shot CLIP. This is why we utilize outliers in the regularization. We leave a detailed analysis in answer 6 of the response below.

4.The selection strategy of the outlier set [W4]

Thanks for your careful review. We clarify that DOR is not very sensitive to the strategy of outlier selection. In the original paper, we provide a discussion about the effect of the selection strategy in Table 4 and Section 6 of our original manuscript. Specifically, we evaluated DOR with three selection strategies: near-OOD (ours), far-OOD, and random-OOD. We report the average performance on base-to-new datasets using CoOp.

As shown below, the results indicate that the near-OOD strategy achieves the best performance in the base-to-new evaluation. Additionally, compared to the baseline (w/o regularization), models with regularization consistently reduce the ECE. Notably, the random selection serves as a strong baseline, highlighting the robustness of our proposed regularization term. To clarify this further, we add a detailed discussion in the first paragraph of Section 6 alongside Table 4.

5. The broader influence of DOR [W5]

Yes, the latest prompt-tuning methods perform better than those earlier methods. Note that our method still improves their calibration performance, we highlight the flexibility of DOR that integrates seamlessly with existing prompt-tuning methods without requiring significant modifications, making it easy to adopt within existing pipelines.

Importantly, we show that the latest methods still achieve suboptimal performance of confidence calibration in some domain-specific tasks. As shown below, our experiments on the medical dataset PathMNIST demonstrate that DOR can significantly improve the calibration of the latest method - CoPrompt. For example, DOR reduces the miscalibration of CoPrompt by 5.99% on base classes and 0.82% on new classes respectively, which highlights its great potential for real-world applications. We illustrate the detailed experiment in Appendix H and Table 13-14.

Thank you for raising the concern of statistical significance. Due to the space limit, we use bold number to indicate the statistical significance of superior results. In particular, we adopt $t$-test among 3 runs to determine the statistical significance (We bold those results with a p-value lower than 0.05). As shown in tables, our method does not make significant improvements for some cases indeed. We justify that the effectiveness of our method is validated by universally improving the performance of current prompt-tuning methods on most cases. As you suggested, we will add the number of std in the appendix of the final version.

Additionally, we'd like to clarify that improving the accuracy is not the goal of this work. In Tables 2 and 3, we present the classification accuracy to show that our method will not decrease the generalization performance with comparable (even improved) results. This is the point that the reviewer may have some misunderstandings.

Thank you for the prompt reply. We respectly disagree that SOTA methods don't need the proposed solution to work well. In the discussion, we add empirical results on several domain-specific datasets like pathMNIST, as reviewers suggested. For your convenient, we attach the results in the table below. The results show that our method can significantly improve current SOTA methods to achieve better performance in confidence calibration (e.g., from 10.33 to 6.92). This indicates the potential benefits of DOR for SOTA promp-tuning methods.

Besides, our work provides a deep understanding of the calibration issue of prompt tuning methods. While some SOTA methods perform well on improving the performance, the community was unclear about why they can benefit the calibration and whether they can perform consistently in some unseen scenarios. This highlights the significance of this work for the community.

We appreciate the reviewer for the insightful and detailed comments. Please find our response below:

1. The strategy of outlier selection [W1]

Thank you for your question. We have discussed this point in Section 6 (Table 4) of our original manuscript. Given a fixed lexical database (e.g., WordNet), we mainly considered three selection strategies: near-OOD (ours), far-OOD, and random-OOD.

The results are presented in the table below. We find that the near-OOD strategy can produce the best performance under the base-to-new evaluation. Moreover, compared to the baseline (w/o regularization), fine-tuned model with DOR consistently reduces the ECE. Interestingly, the random selection can serve as a strong baseline, demonstrating the robustness of our proposed regularization item. For clear illustration, we add a detailed discussion in the first paragraph of Section 6 and Table 4.

2. Applying DOR to domain-specific tasks [W2]

Thank you for the great suggestion. Here, we conduct our experiments on PathMNIST [1] as the medical benchmark. PathMNIST is a multi-class classification task and consists of 9 types of tissues. We fine-tune the CLIP with 16 shots from the first 5 classes and evaluate the model on all 9 classes under the base-to-new evaluation protocol.

As shown in the table below, we find that DOR can effectively help with the calibration of fine-tuned CLIP on medical datasets. For example, DOR significantly reduces the ECE on new classes from 61.28% to 14.99% in CoOp. Moreover, DOR can incorporate with latest methods like CoPrompt and significantly reduces the overall ECE from 10.33% to 6.92%. In summary, DOR can notably improve the calibration performance of prompt-tuning methods and is capable of real-world domain-specific tasks. We add the detailed results in Appendix H and Table 13-14.

3. Ablation on similarity metric [Q1]

Thank you for the suggestion. As you suggested, we provide this ablation in Appendix G.1 and Figure 10. We conduct the experiments with three metrics: Cosine similarity, Euclidean distance (L2), and Mahalanobis distance. We report the average performance on base-to-new datasets using CoOp.

As shown in the table below, we observed that DOR consistently reduces the calibration error with all three metrics. Surprisingly, L2 may outperform Cosine similarity on new classes. Following the common practice in vision-language models [2, 3], we defaultly adopt the cosine similarity to measure the semantic similarity between textual features throughout this work. We add the detailed results in Appendix G.1 and Table 10.

4. Ablation on the number of outliers [Q2]

In this work, we sample the outliers with the K highest scores, without defining a specific threshold. In particular, we set K=5000 throughout the paper. We guess your concern might be related to the number of outliers. As setting a score threshold is the same as changing the value of K, we add a sensitivity analysis of outlier numbers in the revised version.

As shown in Figure 10, we find that DOR is insensitive to the number of selected outliers when the number is relatively large (K>500). Therefore, we suggest a moderate number (e.g., 5000) to yield better calibration performance. We add the detailed results in Appendix G.2 and Figure 10.

5.Frequency of outlier update [Q3]

We demonstrate that the dynamic update strategy in DOR is essential for calibration in fine-tuned CLIP. In our method, we randomly sample a batch of textual outliers from the selected textual outlier set, in each iteration. Therefore, the textual outliers used in each iteration can be different, which establishes a dynamic regularization.

As for the effect of update frequency, we conduct an ablation on the outlier frequency. Here we use large update intervals to represent update frequencies. We report the average ECE on the base-to-new datasets with CoOp. As shown in the table below, low update frequencies (or large update intervals) are observed to negatively impact the calibration performance of DOR. The experimental results highlight the benefits of employing a dynamic update strategy in DOR, since it helps mitigate overfitting to noise [4] and achieves superior calibration performance.

6. Some typos [Q4]

Thank you for pointing it out. We have fixed it in the revised version (Line 406).

7. Clarification on visual outlier selection [Q5]

Thank you for pointing out the ambiguous description. OOD (Out-of-Distribution) data primarily refer to Semantic Shifts, which involve changes in both the label space $P(Y)$ and the input space $P(X)$ [5]. Hence, we suggest using images that do not share similar semantic concept relations with the images seen in the fine-tuning. Specifically, we use class names to filter out images from semantically similar classes. We construct the outlier set by retaining images from 50% of the classes of ImageNet-1k, which ensures these classes differ as much as possible from those used during fine-tuning. To make it clearer, we add a detailed description in Section 6.

8.Feature space when incorporating visual outliers [Q6]

Thanks for your great suggestion. We clarify that DOR-V maintains the performance on base classes while does not affect the visual feature significantly. To quantify the difference between visual distributions, we measure the distance between distributions via Maximum Mean Discrepancy (MMD) and Wasserstein distance. We compare the visual distribution between Zero-Shot CLIP (Z) and CLIP-Adapter (C). DOR-V (D) denotes that our method applied on CLIP-Adapter. We report the results from the base classes of DTD dataset.

As shown in table below, compared with ZS-CLIP, the gap between CLIP-Adapter and DOR-V is relatively smaller. The results show DOR does not significantly affect the feature space of the original fine-tuning method. To have a better view, we also conduct a visualization via t-SNE in Figure 11 of the revised manuscript. In summary, DOR-V does not DOR does not significantly affect the feature space and achieves better calibration as evidenced in Table 5 of the manuscript. We add the detailed results and visualization in Appendix J (Figure 11 & Table 17).

[1] Yang J, et al. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 2023.

[2] Prasanna, et al. Does CLIP’s generalization performance mainly stem from high train-test similarity?. ICLR, 2024

[3] Yi C, et al. Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification. CVPR, 2024

[4] Wei H, et al. Open-set label noise can improve robustness against inherent label noise. NeurIPS, 2021

[5] Yang J, et al. Generalized out-of-distribution detection: A survey. IJCV, 2024

Dear Reviewer 5mjQ,

We truly appreciate your dedication to reviewing our work. As the rebuttal period is ending soon, we eagerly await your feedback and would like to know if our response has adequately addressed your concerns. Are there any points that remain unclear or require further explanation?

Best regards, Authors.

Thanks for your valuable suggestions. Please find our response below:

1. Results with more methods for motivation [W1]

Thank you for the suggestion. As suggested by reviewers, we extend the analysis of motivation to more prompt tuning methods, including CoCoOp, MaPLe, DEPT, PromptSRC and CoPrompt. The results in the table below can make consistent conclusion as provided in Section 3.2: existing prompt tuning methods can significantly increase the FD score of textual features on both base and new classes, as well as the confidence. This leads to the overconfidence issue on new classes, as the accuracy is almost unchanged for those classes. We add a detailed visualization in Appendix B.1.

In addition, we clarify that KgCoOp leads to underconfidence on base classes, instead of novel classes (Line 227). Notably, we politely disagree with the reviewer's view that KgCoOp lacks adequate consideration for novel classes. As claimed in the Abstract of KgCoOp's paper, KgCoOp is designed to mitigate the generalization issue on unseen classes, which is also emphasized in Section 3. They also demonstrate the effectiveness of KgCoOp on unseen classes with empirical results in Table 2. This is why we use KgCoOp as the representative method to show the calibration issue of recent prompt learning methods.

2. Explanation regarding "Texture divergence". [W2]

Yes, this proposed feature divergence is closely related to the diversity of textual representation. As discussed in Section 3.2 (Line 206), we clarify that textual diversity is introduced by the cross-entropy loss, which is widely employed in not only CoOp but also many existing prompt tuning methods.

To further verify that larger textual divergence is correlated with higher output confidence, we employed Spearman correlation coefficients $\rho_{s}$ to compute the relationships between class-wise confidence and FD score on the DTD dataset. The average confidence of each class is derived from samples predicted to the corresponding label. The results of Spearman correlation reveal that textual divergence is positively correlated to output confidence, which supports our observation in Figure 6 of the manuscript.

3. Comparisons with latest prompt tuning algorithms [W3]

Thanks for the great suggestion. As suggested by reviewers, we provide new results with recent methods including PromptSRC [1] and CoPrompt [2]. The average performance on base-to-new datasets is presented in the table below. DOR improves the overall calibration and accuracy of models fine-tuned by PromptSRC or CoPrompt, which is consistent with our former results. Notably, DOR consistently reduces miscalibration on new classes and maintains a comparable performance on base classes. In summary, the results support our claim that our proposed DOR can consistently boost performance on new classes upon existing prompt tuning methods, without compromising the vanilla fine-tuning objective. We add the empirical results into Tables 1 and 2. The detailed results for each dataset can be found in Appendix F of the manuscript.

[1] Khattak M U, et al. Self-regulating prompts: Foundational model adaptation without forgetting. ICCV, 2023

[2] Roy S, et al. Consistency-guided prompt learning for vision-language models. ICLR, 2024.

Dear Reviewer 2Y4k,

We truly appreciate your dedication to reviewing our work. As the rebuttal period is ending soon, we eagerly await your feedback and would like to know if our response has adequately addressed your concerns. Are there any points that remain unclear or require further explanation?

Best regards, Authors.

Thank you so much for diligently responding to my opinion. I have read your review, and I will increase my rating to 6. Thank you.