Towards Global Optimal Visual In-Context Learning Prompt Selection

Q1: The workload is relatively low. I think this work should also explore the impact of models with a larger number of shots on ICL. Additionally, it should investigate the impact of data quality in the alternative dataset, and include the importance of different parts of the loss in the ranking model during ablation experiments in formula (1).

A1: (1) Larger number shots:Thanks, yes, we had tried many number of shots in our experiments, which are solid enough to support our contributions in generally. And we further add more number of shots in rebuttal. Particularly, our choice of number of shot can be referred to previous methods such as MAE-VQGAN and VPR. As for multi-shot experiment, we in this paper present two trials of multi-shot experiments. First, in Tab.1, we adopt a voting strategy to merge predictions of different permutations of each alternative-query pair. Second, in Tab.4, we present multi-alternative fusion. Both results indicate the efficacy of our method. To further verify the role of multi-alternative fusion, we extend experiments in Tab.4 to top-7 and top-10 fusion. The results are shown in the following table. We can find that the results are consistent with the original conclusion, where the performance is nearly saturated with 5 alternatives. This again supports that the experiments in our paper well-supports our claim.

(2) Impact of data quality. Our choice of alternative set is the same as VPR, i.e. selecting 50 most visually similar samples based on CLIP features. Typically, this choice is good enough as shown in previous works like VPR. Here, to further investigate the impact of data quality, we thus tried the suggestion to test our method on all folds of segmentation task with 25 or 100 alternatives for each query. The results are shown in the following table. One can find that our method is also very robust against different sizes of alternative set.

alternative set size

(3) Different parts of the loss. Thanks. We have conducted an additional ablation study to compare models trained for colorization without each loss term. The results are shown in the following table. In general all three loss terms contribute the final performance, with L_sort plays the most important role.

Loss terms

Q2: The conclusions presented in some charts are not intuitive. For example, in Figure 4, can the similarity of our method and the visual similarity of VPR be displayed in the same chart?

A2: Thank you, but as analyzed in L310-L314, visualization in Fig.4(a,b) is used to show that visual similarity can only act as basic heuristic for in-context example selection. Since we do not intend to compare results in these two figures, presenting results for two methods respectively is enough to support our claim. We will clarify this in our paper.

Q3: Compared to previous methods, although it achieves a much better level, the improvement is relatively small.

A3: For better understanding we summarize the improvement of both our method and SupPR against UnsupPR in the following table. Concretely, for segmentation and detection improvement of our method against SupPR is comparable to that of SupPR against UnsupPR. For colorization SupPR cannot improve based on UnsupPR while our method improves it by 0.05 MSE. These results are significant enough to validate the effectiveness of our proposed method.

Q1: The method seems to incur additional costs due to the ranking process. Is there any further analysis on this?

A1: Thank you. Please refer to the general response. Generally, we can summarize that the usage of list-wise ranker and the ranking aggregation will inevitably introduce additional computational cost, while the increased complexity during inference is affordable in our experiments.

Q1: The paper does not include ablation studies on the critical parameters, such as δ and τ (NDCG). These studies are essential for understanding how variations in these parameters affect the model's performance. Without this information, it is difficult to assess the robustness and sensitivity of the model to changes in these parameters.

A1: In general, our method is robust against changes of hyper-parameters. To show this, we try the suggestions to conduct ablation studies on these two hyper-parameters, whose results are presented in the following table. For delta which denotes the margin, using 0 margin leads to worse results while larger margins would be better for learning ranking models. For tau which is the temperature coefficient in NeuralNDCG, we simply use the best setting tau=1 from the original paper. As can be seen in the following table, using smaller temperature would not lead to better results. Nonetheless, all hyperparameter settings enjoy better performance than SupPR, validating the effectiveness of our method.

Q2: The paper introduces heuristic methods for ranking in-context examples using a list-wise ranker and a consistency-aware ranking aggregator. However, there is no analysis provided on the computational complexity and time cost associated with these methods. Given the potential for high computational demands, especially with large datasets, it is important to understand the scalability and practicality of these approaches.

A2: Thank you. Please refer to the general response. We give a thorough analysis. Generally, we can summarize that the usage of list-wise ranker and the ranking aggregation will inevitably introduce additional computational cost, while the increased complexity during inference is affordable in our experiments.

Q3: Figure 4 of the paper presents results related to visual similarity, but it lacks a clear explanation of how this similarity is computed. The method or formula used to calculate visual similarity is not provided, which is crucial for the reproducibility of the results and for understanding the underlying methodology.

A3: Thank you. As illustrated in L310, we use commonly used CLIP to extract features for each image, and then we calculate the cosine similarity between the alternatives and query, which are then averaged. We will add the details to our paper.

Q1: Since training exclusively on a single dataset is limiting, I would appreciate a demonstration of transfer learning performance to prove its adaptability and universality across a broad range of data.

A1: Thank you for the suggestion. We would like to highlight that the current experiment results can sufficiently validate the efficacy of our method. Nevertheless, to test the transferability of the prompt selection method and further reveal the potential of our method, we can add the demonstration suggested by conducting the following experiment for both SupPR and our method: we use models trained on each fold of segmentation and apply it to other folds. The results are shown in the following two tables. We find that while both two methods degrade in the transfer learning setting, our method still outperforms SupPR in general.
This interesting results indicate that, the training data size of each fold is insuifficient for training a robust and generalizable ranking model. On the other hand, this again indicates that prompt selection for VICL cannot be simply based on visual similarity, as claimed in our paper.

Ours

SupPR

Q2: It would be beneficial to analyze the efficiency of your model to the size of the retrieval set, as complexity is a crucial aspect.

A2: Thank you. Please refer to the general response. Generally, we can summarize that the usage of list-wise ranker and the ranking aggregation will inevitably introduce additional computational cost, while the increased complexity during inference is affordable in our experiments.

Q3: It would be insightful to examine the "upper bound" of performance, such as by concatenating each demonstration with a query to see what the maximum Intersection over Union (IoU) is, and to assess how much room there is for improvement from this upper bound.

A3: Thank you for this suggestion. We want to first highlight that our experiment results can already validate the efficacy of our method against the competitor as a powerful in-context prompt selection method. To further provide insight to this task, we try to examine an upper bound: directly testing all alternatives for each query in the segmentation task and presenting the best IoU in the following table. While our proposed method is much better than SupPR, it still leaves great potential for better performance, which we will take as future works.