Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models

Thank you for your insightful and positive suggestions. Here’s our response below:

1. The choice of Near-Outliers or Random-Outliers

We agree that random outliers offer a practical and efficient approach for real-world out-of-distribution (OOD) applications. To investigate their feasibility, we conduct a detailed analysis of when random outliers are sufficiently effective. We hypothesize that the performance gap between near OOD and random OOD may depend on the task’s scope. Accordingly, we compare them across datasets of varying breadth: ImageNet, spanning diverse object classes, versus the fine-grained DTD (textures) and Flowers (flowers). In the table below, our results reveal a larger gap between random and near OOD on ImageNet compared to the two fine-grained datasets, suggesting that random outliers may be particularly viable for tasks with broad and diverse categories in our method.

Thanks for your helpful and positive feedback. Our response is below:

1. Theoretical analysis on multi-class settings

Thank you for the insightful suggestion. We presented the theoretical results in binary classification to help readers quickly understand the relationship between feature divergence and output confidence. Here, we draft a theoretical analysis in the multi-class setting. In particular, a set of logits $\\{z_j\\}_ {j=1}^K$ sampled from a Gaussian distribution $\mathcal{N}(\mu, \sigma^2)$ in $K$-class classification and maximum softmax probability is denoted by $p_{\max} = \max_j \frac{e^{z_j}}{\sum_k e^{z_k}}$. Now, we prove that the expected maximum softmax probability $\mathbb{E}[p_{\max}]$ increases strictly with $\sigma$.

Similar to our former analysis, we assume $\mu = 0$ without loss of generality. We then standardized the logits as $z_j = \sigma u_j$, with $u_j \sim \mathcal{N}(0, 1)$ i.i.d., so $p_{\max} = \max_{1 \leq j \leq K} \frac{e^{\sigma u_j}}{\sum_{k=1}^K e^{\sigma u_k}}$. To analyze the monotonicity of $\mathbb{E}[p_{\max}]$ with respect to $\sigma$, we compute the derivative $\frac{d}{d\sigma} \mathbb{E}[p_{\max}]$. By using the Dominated Convergence Theorem, we interchange differentiation and expectation. The derivative simplifies to $\frac{d}{d\sigma} \mathbb{E}[p_{\max}] = \mathbb{E} \left[ p_m \left( u_m - \sum_{k=1}^K p_k u_k \right) \right]$, where $m = \arg\max_j p_j$. We ensure this step is valid by bounding the derivative $\left| \frac{\partial p_m}{\partial \sigma} \right| \leq 2 \max_j |u_j|$, which has a finite expectation for fixed $K$. Now, since $u_m = \max_j u_j$ and $e^{\sigma u_j}$ is increasing in $u_j$, we have $u_m - \sum_{k=1}^K p_k u_k = \sum_{k \neq m} p_k (u_m - u_k)$. Since $u_m > u_k$ for $k \neq m$ almost surely, and $\sum_{k \neq m} p_k = 1 - p_m > 0$, this expression is strictly positive. Given $p_m > 0$, the expectation $\mathbb{E} \left[ p_m \left( u_m - \sum_{k=1}^K p_k u_k \right) \right]$ is positive, implying $\frac{d}{d\sigma} \mathbb{E}[p_{\max}] > 0$. Hence, for $\sigma_2 > \sigma_1$, we conclude $\mathbb{E}[p_{\sigma_2}] > \mathbb{E}[p_{\sigma_1}]$ in the multi-class setting. We will incorporate this detailed analysis into the final manuscript.

2. Minor improvements on some datasets

Thank you for raising the concern. We also identify that the severity of the calibration problem differs on various datasets: current prompt-tuning methods are miscalibrated on some datasets (e.g., DTD, EuroSAT, and FGVCAircraft) and well-calibrated on some other datasets (e.g., Caltech101 and ImageNet). We conjecture that the calibration performance might be relevant to the distribution distances between the pretraining dataset of CLIP and downstream datasets (Larger distances may cause worse calibration). Notably, our method can consistently improve the calibration performance on various datasets, especially significant on those challenging cases.

3. Add standard deviation to results

Thanks for your valuable suggestion. We present the main results with standard deviation in Table 1 of [link]. The results confirm the significance of the improvement from our method.

4. The choice of cosine similarity

We use the cosine similarity since it is widely adopted in the literature of CLIP. In addition, the features of CLIP are typically normalized so that using either Euclidean distance or cosine distance achieves similar impacts. In particular, the squared Euclidean distance is proportional to the cosine distance: $||\mathbf{x} - \mathbf{y}||_2^2 = 2 - 2 \cos \langle \mathbf{x}, \mathbf{y} \rangle$. Therefore, we default to using the cosine similarity due to its popularity in vision-language models.

1. Clarification on "information leakage"

Thanks for your insightful review. We believe this concern is closely related to the response #5 for Reviewer YQTf, which is about the overlap between outliers and new classes. We'd like to clarify that our method does not leak the information of new classes, as the outliers are selected with base classes. Moreover, the ablation study in Table 6 demonstrates that using even far-OOD and random-OOD can significantly improve the overall calibration performance. In the response #5 for Reviewer YQTf, we also provide an additional ablation study by removing the new classes from the outlier pool (e.g., WordNet). The results demonstrate that the effectiveness of our method does not rely on the overlap with new classes. Instead of causing "information leakage", our work opens up the possibility of utilizing cheap public texts for improving calibration performance. In summary, leverages general semantic information from language space rather than memorizing target classes, which ensures its fairness and generalizability. We will add this discussion to the final version.

2. Results on PromptKD

Thank you for the great suggestion. we provide new results by incorporating DOR into a recent method - PromptKD. In particular, we apply DOR in the first stage to train a large teacher model(ViT-L-14), and keep the second stage unchanged. The results below show that our method can achieve meaningful improvements over PromptKD despite its strong performance.

3. Performance decrease on base classes

Thank you for the careful review. We'd like to clarify that our method achieves comparable accuracy to baselines on base classes. To illustrate this, we provide the accuracy with standard deviation in the table below. In particular, the performance gaps are negligible with most of them are less than 0.2%. We conjecture that adding regularization may slightly affect the training dynamic during the prompt tuning, leading to trivial changes on the accuracy (also increasing a little on new classes). Overall, our method can maintain the generalization performance of baselines on Base classes.

4. Visualization on DOR

Thank you for the great suggestion. we provide new visualization in Figures 1-2 [link]. The results show that KgCoOp significantly decreases the FD scores on both Base and New classes, whereas our method primarily impacts New classes. Consequently, our approach provides confidence scores that align with the improved accuracy on Base classes. We believe the visualization suggested by the reviewer can enhance the clarity of this paper.

Thank you for the constructive feedback. Please find our response below:

1. Visualization of DOR

Thanks for the great suggestion. We'd like to clarify that our method outperforms KgCoOp as our method does not fix the confidence level on the base classes while preventing the increase of textual divergence on new classes. In contrast, KgCoOp anchors the confidence level, leading to underconfidence on Base classes. To illustrate this, we provide new visualization in Figures 1-2 [link]. The results show that KgCoOp significantly decreases the FD scores on both Base and New classes, whereas our method primarily impacts New classes. Consequently, our approach provides confidence scores that align with the improved accuracy on Base classes. We believe the visualization suggested by the reviewer can enhance the clarity of this paper.

2. Clarification on result inconsistency

Thank you for your thorough review. We have carefully revisited the CoOp performance reported in our manuscript and DAC. We found that the slight differences occur specifically on two datasets—FGVCAircraft and EuroST—while the results for the remaining nine datasets align closely with those in DAC. We attribute these minor variations to differences in experimental settings, such as the random seed, hardware specifications, or software environment, which can subtly affect model performance in certain cases. Notably, these minor variations do not impact the contribution of DOR, which largely improves the calibration performance. For reproduction, we will release the full code of baselines on GitHub after publication.

3. Comparison between DOR and KG

Thank you for the great suggestion. We conduct new experiments by comparing DOR and KgCoOp on 3 baselines (CoOp, MaPLe, and CoPrompt), and present the results in the table below. The results show that integrating with our method can consistently outperform those with KgCoOp on overall performance. In particular, our method achieves much better performance than KgCoOp on Base classes, while KgCoOp performs well on New classes. This is consistent with the analysis presented in Subsection 3.2: KgCoOp anchors the confidence level, leading to underconfidence on Base classes. In short, the results demonstrate the superiority of DOR in breaking the calibration trade-off between base and novel classes.

4. Ablation on the size of the outlier set

Yes, we present the ablation study in Appendix H2. The results show that our method is not sensitive to the outlier set size, especially when the size is larger than 500. In particular, the impact of set size is negligible on Base classes, while a larger set size promotes better performance on New classes. In an extreme case with $k = 10$, our method can still significantly improve the performance on new classes. Due to the low cost of outlier texts, we recommend $K=5000$ as the default to achieve optimal calibration performance.

5. Overlap between outlier texts and new classes

Thank you for raising the concern. In Table 8, we present all selected outlier texts for 6 datasets, showing almost no class overlap especially on StanfordCars and FGVCAircraft. Moreover, the ablation study in Table 6 demonstrates that using even far-OOD and random-OOD can significantly improve the overall calibration performance. To further analyze the impact of overlap, we conduct a new ablation study by excluding all new classes from the outlier pool (denoted as DOR$^\dagger$). The average results on 11 datasets below show that DOR$^\dagger$ without overlap achieves comparable performance to DOR, significantly improving the performance on all three baselines. Therefore, the effectiveness of our method does not rely on the overlap with new classes.