In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-Language Model

Thank you for your detailed review of our paper and valuable feedback! We are encouraged that you found our work interesting and promising. In the following, we will answer your questions one by one.

Q1: Why use the term "in-context"? What is the "context" for an incoming test sample? What is the relationship between the context samples and the test sample?

A1: i) In-context learning, a new paradigm in NLP, enables large language models to infer unseen tasks by conditioning on specific in-context examples. To harness this emergent ability in large vision-language models like CLIP, we develop a method to learn adaptive visual prompts. This is achieved using just a few image-label pairs, which act as contextual information, aiding the model to swiftly adapt to new test samples. Therefore, these pairs are referred to as in-context examples.

ii) For an incoming test sample, the "context" is in-context samples, i.e., several provided image-label pairs, to let the model do task like these in-context samples do on the test sample. iii) The context samples only provide the task information to do on the test sample. They are usually from the same target dataset, while there is no other relationships between them, e.g., category.

Q2: The goal of TTA is to address the domain shift on the fly. The introduction of the context samples makes the model task-specific. For each dataset in Table 2 and Table 3, do we need to construct a dataset-specific context? If the answer is YES, I think this is not a TTA method. It is a domain/dataset-specific 1-shot or few-shot approach.

A2: Yes. We need to construct a dataset-specific context for each dataset in Table 2 and Table 3.

However, our InCP is not a domain/dataset-specific 1-shot or few-shot approach. The reasons are: i) domain-specific few-shot approach is for training process, while our InCP is in the test time; ii) few-shot approach needs at least one sample from each category for training, while in-context examples in our InCP have no this constraint. We only sample five in-context examples no matter what category they belong to. iii) few-shot approaches like CoOP [1] and CoCoOP [2] employ a shared prompt for all test samples, whereas our InCP adopts an adaptive, sample-specific prompt for each individual test sample.

Q3: The authors should further clarify how the context samples are selected.

A3: We employ only a few number (i.e., 5) of in-context examples. Those examples are randomly chosen from a designated subset of the target dataset, termed "candidate in-context examples". Such a subset is created by selecting one image-label pair from each category, ensuring a diverse and representative sample pool.

Q4: Please also provide the inference time of the proposed method to evaluate the efficiency.

A4: Thanks for your suggestion. We report the inference time of the proposed method in the following table.

[1] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In IJCV, 2022.

[2] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.

Thank you for your feedback! We provide responses to your comments below:

Q1: I think the dataset-specific context selection process will largely limit the application of the proposed methods. The creation of the context put a strong prior on the method. Further discussion and justification are needed.

A1: While our method is originally tailored for specific datasets, its adaptation to large-scale datasets such as ImageNet allows the extraction of significant context information. This context information serves as an in-context prompt that can be applied to different datasets or downstream tasks. The versatility introduced through this approach enables our method to generalize across various datasets, thereby enhancing its overall applicability.

Thank you for your detailed review of our paper and constructive feedback! We are encouraged that you found our task promising and our idea novel. We have revised our paper and addressed your concerns. Below, we detail each change made in response to your feedback.

Q1: I am wondering whether the terminology of "in-context prompt learning" is proper. More evidence or reference on the terminology definition is expected during discussion.

A1: We follow existing works [1,2,3] for this ``in-context learning'' concept, in which the model is updated using in-context examples. Meanwhile, CLIP itself is not able to conduct in-context learning task. To equip CLIP with this ability, our InCP introduces learnable prompt for each test sample in test-time stage. In this way, the model can automatically understand the underlying task with in-context examples. For sample selection, in-context learning has no constraint on category in testing stage. The in-context samples can either share the same category as the current test sample or the irrelevant category. For sample quantity, in-context learning uses a small, arbitrary set of labeled samples. Therefore, our InCP belongs to the in-context learning area.

Q2: I am wondering whether the performance gain is due to the information leakage caused by random sampling. A strict implementation would be using sampling from the same subset of labeled data for rather than the whole test set.

A2: There is no information leakage in our InCP. Actually, the selected 5 in-context examples do not share the same label with the test sample. These 5 in-context examples are selected from a fixed subset of labeled data, which is composed by randomly sampling 1 sample from each category. For each test sample, we use the same subset rather than the whole test set, which is the same setting as you suggested.

Q3: The figures are confusing and not well designed. First, in Figure 2, it is not clear what does token space mean. The number of input tokens are 4 and the number of output tokens are 2. Does it mean the number of input tokens equals to the number of words in the prompt, and the token net aims to compress the tokens? Second, in Figure 3, the module and tokens are denoted in many colors, but it is hard to understand what each color indicates, and what is the dfference between different tokens in (c) and (d). Third, in Figure, the dimension of (i.e., 3) does not equal to the number of prefix tokens (i.e., 4). Is it in intention? Figure 1(c) is also confusing to me to understand what happened during the cyclic learning stage.

A3: i) "token space" in Figure 2 refers to the embedding space of text tokens and visual tokens. The number of input tokens equals to the number of words in the prompt and the output tokens. Here, the 4 to 2 means the dimension decrease, not the token number. Sorry for the confusion of token number. We modify Figure 2 in the manuscript for clarification. ii) The orange token represents a learnable visual token, the light purple tokens represent static visual tokens, the light gray tokens represent static text tokens, and the light orange tokens represent static prefix tokens in both Figure 3 (c) and (d). We update Figure 3 in the manuscript. iii) Sorry for the confusion. The number of output tokens should be the same as the prefix tokens (i.e., 4). We modify it in the manuscript. iv) During the cyclic learning stage, we first optimize text prompt and then visual prompt in a sequential process. We update Figure 1 (c) for clarification.

Q4: Difference between this method and fully test-time adaptation. According to Table 1, the main difference is that this method used additional labeled data for tuning the prompts. If so, the novelty is just additional annotations in a semi-supervised manner, which is limited.

A4: Fully test-time adaptation is a self-supervised learning method, which applies its self-supervised loss on unlabeled test sample to achieve the generalization capability of model on unseen domains. We argue that self-supervised learning, without task-specific supervision or model modifications, is insufficient for test-time adaptation, and thus propose in-context prompt learning for model adaptation.

Semi-supervised learning typically incorporates labeled data during the training phase, amalgamating it with unlabeled data to fine-tune the model and improve its performance on unlabeled samples. Labeled data in semi-supervised learning often shares categories with the unlabeled data. In our method, there is no inherent relationship between in-context examples (labeled data) and the test sample, as they are both drawn from the same domain dataset. Our approach does not necessitate any category information about the test sample, distinguishing it from semi-supervised learning methods.

Thank you for your detailed review of our paper and constructive feedback! We are encouraged that you found our method well-motivated, our experiment comprehensive, our ablation studies well-designed clear. We are addressing each of your comments and questions below:

Q1: I don’t think it is appropriate to call the proposed method an "in-context learning" method. In-context learning in both the language and vision literature does not involve optimizing any parameters including exterior ones like prompts. I think the method proposed in this work is more like few-shot learning, which updates parameters on some example images and then uses the updated parameters to run inference on the test sample.

A1: i) We follow existing works [1,2,3] for this "in-context learning" concept, in which the model is updated using in-context examples. Meanwhile, CLIP itself is not able to conduct in-context learning task. To equip CLIP with this ability, our InCP introduces learnable prompt for each test sample in test-time stage. In this way, the model can automatically understand the underlying task with in-context examples.

ii) Our work is different from few-shot learning. The main difference lies in two factors, i.e., sample selection and quantity. (a) For sample selection, few-shot uses strict categories with specific number of samples, which are widely used in training stage. Differently, in-context learning has no constraint on category. The in-context samples in testing stage can either share the same category as the current test sample or the irrelevant category. It is also impractical to know the exact category of unlabeled test sample in advance. (b) For sample quantity, few-shot learning requires a predefined number of samples from each category, while in-context learning uses a small, arbitrary set of labeled samples-commonly just five samples. Therefore, our InCP belongs to the in-context learning area, rather than the few-shot learning.

Q2: It needs to provide more details about how Token Net is trained. If it is randomly initialized and only optimized for one step at test time, I don’t think it can "translate" text tokens into visual prompts that the vision encoder comprehends.

A2: Yes, the token net is randomly initialized. The feasibility of token net is based on CLIP, where the textual and visual encoders are already aligned well in the embedding space. The token net is initialized by text tokens, and then transferred to visual tokens with a few optimization steps on visual examples, which is easy to be converged.

Q3: If I understand correctly, the InCP in Section 4.1 is learned using in-context examples from the target dataset. However, CoOp and CoCoOp are learned on the ImageNet dataset. A fair comparison would be to report CoOp/CoCoOp’s results using the same few-shot examples as InCP.

A3: Thanks for your advice. We provide CoOp/CoCoOp’s results using the same examples as InCP in the following table. It shows that our InCP achieves better performance than CoOp and CoCoOp on this setting. Following your suggestion, we add this table in the manuscript for fair comparison.

Q4: In Figure 4, it seems like both $P_{\text{v}}$ and $P_{\text{t}}$ and the token net are trainable. But the learning objective in Eq (2) only involves $P_{\text{v}}$?

A4: Thanks for pointing out this. The set of $P_{\text{v}}$, $P_{\text{t}}$ and token net are all trainable in our method. The completed objective function is presented below. With step is 1 (i.e., $s=1$) for visual prompt learning, the objective function is $\mathcal{L}(x_t,\mathcal{D},P_{\text{v}},P_{t},\theta)= \underset{P_\text{v}, \theta}{\mathrm{argmin}} \{L(x_t, P_\text{v}) + \sum_{(x_i,y_i) \in \mathcal{D}} \lambda L(x_i, y_i, P_\text{v}, P_\text{t}, \theta)\}.$ With step is 2 (i.e., $s=2$) for text prompt learning, the objective function is $\mathcal{L}(x_t,\mathcal{D},P_{\text{v}},P_{\text{t}},\theta)=\underset{P_\text{t}}{\mathrm{argmin}} \{L(x_t, P_\text{t}) + \sum_{(x_i,y_i) \in \mathcal{D}} \lambda L(x_i, y_i, P_\text{v}, P_\text{t}, \theta)\},$ where $x_t$ is unlabeled test sample, $\mathcal{D}$ contains in-context examples, $\theta$ is the parameter of token net, and $s=1,2$ are the step numbers of model optimization for visual and text prompt, respectively.

Thank you for your comprehensive review, valuable comments, and constructive feedback. We are encouraged that you found our idea insightful, our method well-structured, our experiments rigorous, and our work exceptionally clear. We provide point-to-point responses to your comments below:

Q1: Efficiency is a crucial factor in test-time adaptation, given the significance of swiftly adapting to new environments. It is imperative that the paper includes explicit reporting and a comparative analysis of inference time metrics for InCP, especially when compared to existing methods like TPT.

A1: The inference time is shown in the following table. All experiments are conducted on one 2080 Ti GPU, and inference time (Infer. Time) is calculated in minutes. The table shows that needs significant inference time due to augmenting images by 64 times. In contrast, our InCP only uses few in-context examples (i.e., 5) without any augmentation, requiring less inference time. This table clearly shows the efficiency of our method.

Q2: This paper requires clarification regarding InCP's performance on the SUN397 dataset (utilized in TPT) in Table 1 and the ImageNet dataset (also used in TPT) in Table 2. Providing a comprehensive comparison of InCP's performance on these specific datasets will significantly enhance the paper's clarity and the reader's understanding.

A2: The performance on ImageNet and SUN397 datasets is shown in the following table. It shows that our InCP achieves best performance compared with previous zero/few-shot methods. In particular, we only use 5 samples yet is better than few-shot methods [2,3].

Q3: The performance improvement in Table 2 appears to be marginal. Further explanation or additional results may be needed to demonstrate the significance of the improvement.

A3: As few-shot methods, CoOP [2] and CoCoOP [3] fine-tune the prompt on ImageNet dataset using 16-shot training data per category and evaluate the generalization performance on downstream tasks. As a self-supervised method, TPT is not trained on ImageNet. It utilizes the self-supervised loss with unlabeled test samples to achieve its zero-shot generalization ability. Our InCP follows TPT, while only using a few in-context examples as domain-specific context information. Table 2 in the main paper shows that our method outperforms TPT by notable margins on two datasets (i.e., 72.27% vs. 68.98% on Flower dataset, and 70.26% vs. 68.04% on UCF101 dataset). Our InCP can also consistently outperforms CoOP and CoCoOP when using unlabeled ImageNet data (i.e., 70.26% vs. 68.44% on UCF101 dataset), and using the same in-context examples (i.e., 71.13% vs. 67.23% on Flower dataset). The following table shows the comparison with CoOp and CoCoOp using the same in-context examples.

Referece:

[1] Manli Shu, et al. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022.

[2] Kaiyang Zhou, et al. Learning to prompt for vision-language models. In IJCV, 2022

[3] Kaiyang Zhou, et al. Conditional prompt learning for vision-language models. In CVPR, 2022.

[4] Muhammad Uzair Khattak, et al. Maple: Multi-modal prompt learning. In CVPR, 2023.

[5] Muhammad Uzair khattak, et al. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV, 2023.