Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models

• 3. CLIPCap and CapDec with the CLIP-ViT-L/14 architecture

  We show the image captioning results with CLIP-ViT-L/14 in Tabel uHKV-2. With CLIP-ViT-L/14 as the image embedding extractor, RLCF with CLIP-ViT-L/14 as the reward model still achieves significant improvements. Specifically, RLCF+CLIPCap achieve more than +6% improvement in the CIDEr metric on the out domain data of NoCaps.

  With different configurations of image embedding extractor and CLIP reward model, RLCF can always achieve substantial improvements in the near domain and out domain data. This shows that RLCF is robust to the domain shift. It is desirable for the TTA problem.

Tabel uHKV-2 CapDec and CLIPCap with CLIP-ViT-L/14 architecture on NoCaps (train on COCO image or text, test on NoCaps). The reward model is always CLIP-ViT-L/14.

We also add CapDec and CLIPCap with CLIP-ViT-B/16 as the reward model in Table 7 in the revised paper.

• 4. More implementation details of TPT+KD

  For test time prompt tuning with TPT+KD in image classification, we inherit the prompt weights of CoOp. During TTA, we minimize the cross entropy between logits from the teacher and student model for the samples after confidence selection. The teacher model is always CLIP-ViT-L/14.

  As for the learn hyper-parameters, the learning rate and TTA steps are the same as RLCF, i.e., 7e-3 and 3 TTA steps. More TTA steps will not benefit TPT+KD as shown in the retrieval task in Table 2.

[1] Cho, Jaemin, et al. Fine-grained Image Captioning with CLIP Reward. In Findings of NAACL, 2022.

[2] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang.Decoupled Knowledge Distillation. CVPR 2022.

[3] Guo, Jia, et al. "Reducing the teacher-student gap via spherical knowledge disitllation." arXiv preprint arXiv:2010.07485 (2020).

• 1. Different between RLCF and the previous work [1]

  • In [1], CLIPScore works with grammar regularization or CIDEr metric during training, while RLCF works with independent CLIP reward function at test time. The training of the grammar head and calculation of the CIDEr metric in [1] both need reference text data, which is unavailable at test time. When CLIPScore is used as the independent reward function in [1] (its Table 2), 11.2 CIDEr is achieved, while the baseline has 110.3 CIDEr. [1] actually demonstrate that CLIP cannot be used as the sole reward function. This is different from our findings that CLIP reward can also be used as an independent reward function at test time.
  • The difference of the problems is non-negligible. [1] adopt CLIPScore together with grammar regularization or CIDEr metric at the training stage, where batched image-text pairs are available. Thus no sampling strategies are required. By contrast, RLCF works at the test time, where only a single test sample is available. Therefore, task-specific sampling strategies are necessary when using CLIP as the reward model. [1] cannot be directly transferred to test time due to the lack of reference text data, this is also why we do not provide a direct comparison with [1].
  • We contribute the novel reward baseline for CLIP reward as well as task-specific TTA pipelines along with sampling strategies. These contributions are important for CLIP reward to work as the independent reward function across different tasks at test time. We also demonstrate the effectiveness of CLIP reward across different tasks. These contributions are non-trivial to the community.

  We add more discussion with [1] at the Section RELATED WORK in the revised paper.

• 2. More advanced KD and the ensemble of the student and teacher model.

  We provide the ensemble results and two more advanced KD results (DKD[2] and ATKD[3]). The learning rates and TTA steps of the two methods are set the same asn RLCF. We set the hyperparameters of the two methods as the recommendations of the original papers.

  The results are shown in Tabel uHKV-1. RLCF is also better than the two KD methods. It is worth noting that RLCF with CLIP-ViT-B/16 is even better than the teacher CLIP-ViT-L/14 on ImageNet-A/ImageNet-V2/ImageNet-R. By contrast, the performance of the student models with KD methods is generally worse than the teacher models (This is a general belief in KD for the image classification problem).

  RLCF is also better than the ensemble results of CLIP-ViT-B/16+CLIP-ViT-L/14 by ~2% on average on the 4 OOD datasets. Compared to the ensemble method, RLCF can adapt to the test distribution with the feedback mechanism. This is why RLCF shows better performance than the ensemble results.

Tabel uHKV-1 More advanced KD and ensemble results on OOD data

We add ATKD in Table 1 and ensemble results in Table 7 in the revised paper.

[1] Cho, Jaemin, et al. Fine-grained Image Captioning with CLIP Reward. In Findings of NAACL, 2022.

[2] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang.Decoupled Knowledge Distillation. CVPR 2022.

[3] Guo, Jia, et al. "Reducing the teacher-student gap via spherical knowledge disitllation." arXiv preprint arXiv:2010.07485 (2020).

• 1. Different choices of reward models and how robust RLCF is to unreliable rewards

  To figure out how robust RLCF is to different reward models, we first test RLCF with different reward models in image classification. As shown in Table efhx-1, RLCF is robust to different reward models. Compared to the baseline CoOp, RLCF can achieve improvements even with a CLIP-RN50x4 as the reward model, which is worse than the prompt tuning model CLIP-ViT-B/16. During TTA, RLCF will first sample a few image-text pairs and the reward model will only give feedback to the sampled feedback. RLCF will not require the reward model to provide feedback for the image-text pairs not sampled. Namely, RLCF will not totally follow the output of the reward model. This reduces the risk of model collapse when the reward model is less informative than the tuning model.

Table efhx-1. Different reward models with RLCF for prompt tuning with CLIP-ViT-B/16 on OOD data.

We also test different reward models in image captioning. Results are shown in Table efhx-2. CapDec and CLIPCap both use CLIP-ViT-B/16 as the image embedding extractor. RLCF with different reward models can always achieve significant improvements in the near and out domain data. This shows the robustness of RLCF to open domain scenarios with different CLIP reward models.

Table efhx-2. Different reward models with RLCF in Image captioning. Train on COCO and test on NoCaps.

We add these contents to Section B.2 DIFFERENT REWARD MODELS in the appendix of the revised paper.

• 1. Can RLCF be applied within a few-shot learning framework.

  Actually, in Table 1 for image classification, for prompt tuning with CLIP-ViT-B-16, RLCF inherits the prompt weights from CoOp[1]. CoOp is a few-shot learning method (16 shots on ImageNet). RLCF works well with weights from CoOp and surpasses the CoOp baseline by a large margin. This case demonstrates that RLCF can also work well with a few-shot method.

We modify Table 1 in the revised paper to make this point clearer. Specifically, we add the test prompt tuning without CoOp pre-trained prompt weights and highlight the usage of CoOp weights (In the original submission version, we mention the usage of CoOp in the implementation details). These contents are also shown in Table 9Cat-1.

Table 9Cat-1. RLCF with and without CoOp.

• 2. Data augmentation for textual data

We apply RLCF in image classification, text-image retrieval, and image captioning. Different datasets including ImageNet/MS COCO/Flickr30K/NoCaps are used. For all experiments, no augmentations are applied for the textual data during TTA.

• 3. All steps in Figure 6

  Thanks for your suggestions. We add a new Figure 7 in the appendix of the revised paper to show the results at all TTA steps.

  We display the top-4 generated captions with their CLIP reward as each TTA steps. These samples illustrate how CLIP reward helps the captioning model select a description that matches the picture more closely

[1] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision language models. IJCV, 2022.