PRO: Pseudo-label Regularized Optimization on Unlabeled Test Data

We thank the reviewer for their suggestions to improve the paper. We provide some additional clarifications to the questions below:

While this paper empirically validates which setups/parameters are useful for test time training, it does not provide theoretical understanding on why they do not collapse

We note that imposing a penalty against deviance from the initial label distribution intuitively discourages collapsed solutions. The primary aim of this paper is to conduct an empirical investigation to substantiate this intuition and explore methods useful for test-time training. The potential for theoretical exploration into this aspect is left for future works.

Details of algorithm (PRO) are missing. It is unclear how the loss functions are combined

We will add equations detailing each loss function in the updated draft.

Can all experiment/analysis results of section 5 be included in appendix? While Section 5 contains various insights from experiments, full experiment setup and results are omitted

We will expand on the discussion on the full experiments in the appendix in the updated version.

What’s the failure mode of the suggested method other than class imbalance?

As shown in Figure. 3, we do not observe any significant failure mode of the suggested method.

Can it be shown that PRO is effective in setups other than CLIP?

Our primary focus lies in understanding the distinctive distribution shifts associated with the adaptation of CLIP models in zero-shot tasks. While PRO may demonstrate effectiveness in handling other shifts and architectures, we consider these settings more appropriate for exploration in future works.

We would like to thank the reviewer for their comments. We are glad that you appreciate the simplicity of our method.

The authors have neglected to benchmark PRO against other related test-time adaptation methods employing semi-supervised learning strategies, such as self-training [2]

We thank the reviewer for bringing relevant works to our attention. However, we note that many test-time adaptation methods, including the one reviewer suggested, are designed for non-CLIP models. At a fundamental level, the distribution shifts addressed in these works differ substantially from those encountered in our context. ResNet models, for example, are trained on supervised classification tasks with training data resembling the test data. CLIP models, however, were trained via contrastive losses on data that may be significantly different or irrelevant to the downstream data we aim to adapt to. Additionally, it is unclear whether the adaptation should be applied to either or both of the visual and textual towers of CLIP, and improbable that the hyperparameter choices are transferable.

We remark that comparison of PRO against a test-time adaptation method developed for CLIP (MUST) is provided in Table 1, where we find significant advantage in PRO over MUST. We additionally implement test-time prompt tuning (TPT) of Shu et. al [1] and found that on ViT-B/32, TPT results in 43% and 64% in accuracies for DTD and UCF-101 datasets, which are correspondingly 6% and 3% lower than the performance of PRO-CLIP with textual layer (see Table. 1)

The efficacy of PRO seems limited to specific architectures; it yields noticeable improvements when applied to VIT models but is virtually ineffective with ResNet architectures, resulting in a marginal improvement of only 0.1% (See Appendix B).

We appreciate the reviewer's keen observation. It's important to highlight that the distribution shifts examined in Appendix B with the ResNet architecture differ significantly from those addressed in the main text with CLIP. While we acknowledge that PRO might be more suited for the latter scenario, we emphasize that as foundation models like CLIP gain widespread use, the latter setting becomes increasingly relevant.

[1] Shu et. al. Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. In Advances in Neural Information Processing Systems (2022).

We would like to thank the reviewer for their constructive feedback on our work. We are glad that you appreciate the clarity of our presentation and find merit in our experimental design.

The experimental results of the paper are not convincing enough. First of all, PRO-CLIP in Figure 3 has little performance gains on commonly used image datasets

We'd like to highlight that PRO-CLIP leads to an average accuracy improvement of 2.5%, with a maximum enhancement of 6.1% for CLIP. Unlike prior works, we do not perform hyperparameter selection by peeking at the resulting accuracies of individual datasets. Instead, we adopt a consistent set of hyperparameters and conduct careful experiments to avoid meta-overfitting, thereby ensuring the robustness of our approach. We substantiated the robustness of our method in Sec. 6.2, where we find that PRO-CLIP with the same hyperparameter selection results in significant gain (up to 7.8% increase in accuracy) on additional, previously unseen datasets.

Some trivial tips

We thank the reviewers for these suggestions and will incorporate them in the updated draft.

I would like to extend my appreciation for the authors' response and acknowledge their efforts in this work. However, it is regretely that the method innovation and performance improvement claimed by the authors do not convince me.

1. In terms of method innovation:

   I still believe that this method share a high degree of similarity within other methods in the same sub-field.

   The relationship between them more like different combinations of functional/module levels to handle different test scenarios.

2. In terms of performance improvement:

   Based on a model as large as CLIP, the proposed method solely achieves a tiny gains (<= 3.5) in average absolute accuracy on almost all adopted image classification datasets.

   A considerable part of the datasets are common and simple (such as CIFAR100, Flower102, Caltech101, CIFAR10, etc.).

   Most importantly, these datasets are already present in the pre-training phase of CLIP, so even changes in certain hyperparameters may alter their performance gains. In-depth ablation analysis in this regard is also worth discussing, which will help demonstrate the effectiveness of this approach.

Finally, the absence of experiments on textual datasets still leaves me concerned that the applicability of this approach is limited.

We are thankful for Reviewer A2sn’s thoughtful comments. We are glad that you appreciate the experiments and the writing.

My biggest concern is the lack of mean and standard deviation in the reported results to see if the results are significant. This would require repeating the experiments multiple times

We wish to emphasize that our approach lacks inherent randomness, as a result we are not able to calculate standard deviation. We remark that the experimental results show that our proposed method consistently enhances the accuracy of CLIP over 18 datasets, and hope that this addresses the reviewer’s concern on the robustness of our method.

Another concern is the lack of generalization to other tasks like image segmentation and text-related problems which have become high in demand these days

We agree with the reviewer that these are essential tasks for test-time adaptation. As an initial evaluation of the proposed method, we have chosen to concentrate on image classification tasks as a starting point. The investigation of appropriate methods for test-time adaptation in other tasks is deferred to future work.

An additional concern is that the authors are using ImageNet pretrained model which has been pre-trained on millions of images that share similar classes as the ones given by the unlabeled set - to me it doesn't seem like this work would strictly fit within the low-data learning framework

We want to clarify a confusion around this. We do not claim that we are in the low-data learning framework. Instead, our primary focus is on adapting a pretrained model using unlabeled downstream data.