Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting

[Q1] Does the paper utilize the validation dataset in experiments, if not, this should be clarified.

[A1] We do not use a validation set because our method involves no hypermeter searching.

[Q2] How does the proposed method compare with the methods[1][2] in the few-shot settings?

[A2] The methods[1][2] refer to LFA and Tip-Adapter. These methods boost the CLIP's generalization using labeled training samples. In contrast, our Frolic doesn't require any labeled samples. We evaluate our method with LFA and Tip-Adapert on the ImageNet and its variants dataset, where the LFA and Tip-Adapter only utilize the labeled samples from the ImageNet dataset. The results below show that our method achieves the best performance, except compared to LFA on the ImageNet. 

[Q3]   Although Frolic is described as training-free, there may still be hyperparameters involved in the method, and how to choose these parameters?

[A3] The only hyperparameter in our method is tolerance $\epsilon = 0.01$, which is critical for the precision of numerical calculations rather than being a traditional model hyperparameter.  As illustrated in Figure 4, when the iteration extends over 20 steps, $\epsilon$ approaches zero. While theoretically, a lower $\epsilon$ is preferable for finer precision, we have selected $\epsilon = 0.01$ to ensure a practical balance between computational efficiency and convergence reliability.

[Q4] How does the prompt distribution learning contribute to the performance? Does it outperform the original CLIP?

[A4] As compared in Table 3 ($f_{\rm c}$ and $f_{\rm g}$), the prompt distribution learning ($f_{\rm g}$, shown in the third row) significantly outperforms the original CLIP ($f_{\rm c}$​, shown in the first row). For example, $f_{\rm c}$ increases the performance from 65.1% to 68.8% on the 10-datasets and 68.7% to 69.8% on the ImageNet.

[Q5]   Can the method be performed with a few labeled samples?

[A5] Our method can effectively incorporate labeled samples by replacing the class description to estimate ${\bf z}_1$ to ${\bf z}_K$ in Equation (3). To demonstrate this capability, we have conducted additional experiments in a 16-shot setting using the ViT-B/16 backbone, the results below show that the 16-shot setting achieves better performance than the class descriptions.

[Q6] The discussion of the limitations of the methods is unclear. The authors should discuss it in depth.

[A6] The quality and distribution of data used in pre-training can significantly impact the performance of pre-trained models. Our method relies on the capabilities of pre-trained models for downstream tasks, if the pre-trained knowledge differs from the downstream tasks, the efficacy of our method may be limited. We have included a detailed discussion of limitations in the revised manuscript. Additionally, as you suggested in [Q5], few-shot learning presents a viable solution to these challenges. We appreciate your insights, which help us to further exploit the potential of our method.

[Q1] The computation of second-order moments and the covariance matrix from the marginal distribution as discussed in Eq.(3) and (5) might be computationally intensive.

[A1] We have evaluated the running time as presented in Table 5. The results show that while Frolic requires slightly more computation time (0.0078 seconds/per sample) compared to the original CLIP (0.0072 seconds/per sample), it yields improved performance, increasing from 68.7% to 71.1%. 

[Q2] The method relies on the pseudo-labels estimated by CLIP. How does this process influence the final results?

[A2] Our method can improve performance regardless of the quality of the pseudo-labels. As evidenced in Table 4, the original CLIP achieves 92.95% accuracy on the Caltech dataset, indicative of high-quality pseudo-labels, while 24.8% accuracy on the Aircraft dataset, which represents lower-quality pseudo-labels. Our method effectively improves results across these varied quality levels, boosting accuracy from 92.95% to 95.1% on the Caltech and from 24.8% to 31.4% on the Aircraft.

[Q3] The setting of results on the ImageNet variants dataset is unclear.

[A3] We utilize the model learned on ImageNet to evaluate its performance across the ImageNet variants. 

We have provided details about each variant to ensure clarity in the revision:

ImageNet-V2: sampling from the original ImageNet and including 10,000 images of 1,000 ImageNet categories.

ImageNet Sketch: including 138 50,000 images and covering 1,000 ImageNet categories.

ImageNet-R: containing renditions (e.g., art, cartoons, graffiti) for ImageNet classes, comprising 30,000 images from 200 ImageNet categories.

ImageNet-A: collecting real-world images that are misclassified by ResNet-50, totaling 7,500 images from 200 of ImageNet categories.

ObjectNet: including 50,000 test images with rotation, background, and viewpoint, and overlapping 113 classes with ImageNet

These details have been included in the revised manuscript.

[Q4] The results lack comparison with other prompt distribution methods, such as CoOp and CoCoOp.

[A4] The CoOp and CoCoOp require a training procedure with labeled samples while our method does not involve any training.  To ensure a fair comparison, we compare our Frolic with CoOp and CoCoOp on across-dataset results, where the CoOp and CoCop are trained only with the labeled samples from the ImageNet dataset and then directly tested on the remaining datasets. The results shown below demonstrate that our Frolic not only avoids the complexities of training but also exhibits superior generalization performance compared to these methods.

[Q1] The work works with a uniform prior that classes are balanced distributed. It is better to discuss the scenario with imbalanced prior.

[A1] We consider that most of the datasets such as ImageNet and its variants are uniformly distributed,  leading us to assume $\pi_j=\frac{1}{K}$ in line 134 as a special case for simplicity. However, if the downstream datasets are imbalanced, we can derive the distribution vector $\bf \pi = Z^{-1}\bf \mu$ as outlined in Equation (5). This strategy allows our method to adapt flexibly to both balanced and imbalanced dataset distributions.

[Q2]  Besides CLIP, there are many other vision-language pre-trained models, e.g., BLIP, etc. It is better to include other pre-trained models to evaluate the performance of the proposed method.

[A2] Thank you for your valuable suggestion. We have conducted additional experiments using BLIP with the  ViT/B-16 backbone.  The results presented below indicate that our method consistently outperforms BLIP across various datasets.

[Q3] Why does $e_i$ follow a Gaussian distribution in L446?

[A3] The original $\bf x$ follows a Gaussian distribution ${\cal N}{({\bf z}_j, \Sigma)}$. We express the covariance matrix $\Sigma$ of the original $\bf x$ as an expansion in terms of its eigenvectors as Equation (24), then we can interpret the variable $e_i = {\bf u}_i^T(\bf{x}-\bf{z}_j)$ to a new coordinate system defined by the orthogonal vectors ${\bf u}_i$. Since these transformations are linear and $\bf x$ is Gaussian, the transformed variables $e_i$ ​ also adhere to a Gaussian distribution. Specifically, as the variance of the $i$ -th coordinate is $\lambda_i$ ,  $e_i$ follows the Gaussian distribution ${\cal N}(0, \lambda_i)$

[Q1] the downstream task is balanced (line 134), which is not always hold true in reality. The long-tail nature of real world does not guarantee that the testing class distribution will be balanced even though the benchmarks do.

[A1] We acknowledge that real-world data often exhibits an imbalanced distribution. The balanced assumption in line 134 is a special case for simplicity. In our method, if the downstream datasets are imbalanced, we can derive the distribution vector $\bf \pi = Z^{-1}\bf \mu$ as outlined in Equation (5). This strategy allows our method to adapt to balanced and imbalanced dataset distributions flexibly.

[Q2]  A decent scale of the testing data on downstream tasks is available (e.g, testing data does not come in online fashion like one sample at a time) for estimating the beta term in bias correction. Would the approach for bias estimation still holds true in online testing scenario? I would assume this is more aligned with reality too.

[A2] Thank you for your insightful question. Our method can be easily extended to an online scenario. This involves updating the estimation $S$ incrementally as each new test example is processed. Specifically, we first initialize the matrix $S$ with the identity matrix. Suppose we receive the $n$ -th test sample with the predicted label as $j$ and the predicted probability as $\bf p$, we first update the ${\mathbf{s}}_j = \frac{n-1}{n}{\mathbf{s}}_j + \frac{1}{n}\mathbf{p}$. Then we compute the estimated $\beta$ and update the $f\_{\text{d}}$. We have conducted additional experiments in the online setting to validate this extension, and the results below show that the online Frolic achieves comparable performance to the original Frolic.

[Q3] The methods did not seem to evaluate on all benchmark datasets like previous works do. For example, CIFAR10/100 and RESISC. While some of these datasets have saturated performance, it might be still helpful to include results there.

[A3] We have conducted experiments with ViT-B/16 on CIFAR-10, CIFAR-100, and RESISC datasets, and the results are presented below:

We observe that on these three datasets, our method improves the performance over the original CLIP.

[Q4] It's not clear to me why [24] is not listed in tables for comparison as I would assume 24 to be the most direct baseline. It shares the same motivation of this paper with the requirement of accessing pretraining data. Arguably, while LAION has been taken back, [24] already does the job for us. I would not think an argument like "[24] requires large-scale pretraining data for estimating bias thus we do not compare with it" as a valid argument because this zero-shot visual recognition task itself is solely working with CLIP-ish models and estimating bias from the datasets like LAION sounds like a fair practice to me.

[A4] Thank you for your suggestions. We acknowledge that [24], referred to as REAL, represents a crucial baseline and shares a similar motivation with our work. We conducted a comparison between REAL, which utilizes the LAION 400M dataset, and our Frolic, using the OpenCLIP (ViT-B/16 model) across several datasets. The summarized results below demonstrate that our Frolic outperforms REAL obviously and achieves an average improvement of 1.1% in accuracy. We have included these results in the revised manuscript.