AutoVP: An Automated Visual Prompting Framework and Benchmark

Q4-1. Other Vision Datasets

We appreciate the reviewer's mention of these prominent datasets. We incorporated additional datasets—SUN397, RESISC, CLEVR, and StanfordCar —and their results are available in Appendix B.4 Table 5.

Our findings demonstrate that AutoVP generally outperforms ILM-VP and CLIP-VP across most datasets. When compared to linear probing (LP), the out-of-distribution (OOD) dataset, CLEVR, demonstrated a significant 15% increase in accuracy. However, the in-distribution (ID) datasets, SUN397 and StanfordCar, show inferior performance than LP. This aligns with our findings discussed in Section 5: Performance Evaluation on ID/OOD.

Q4-2. Comparison with BlackVIP

We thank the reviewer for pointing out BlackVIP, one of the pioneering works in VP. However, we did not compare it with AutoVP because BlackVIP operates in a black-box setting, treating the pre-trained backbone as a black box. In contrast, AutoVP views pre-trained backbones as white-box models. Notably, AutoVP focuses on the concept of 'reprogramming' an online API using publicly released model architecture and weights, even though direct access to the model itself is not possible.

Despite this distinction, we present a comparison across 11 datasets with BlackVIP, and the results are summarized in the table below. Since BlackVIP selects CLIP as the pre-trained backbone, we report our results under the same pre-trained model. Our results demonstrate a 16% outperformance compared to BlackVIP. This considerable gap might be attributed to the difference in update strategies: BlackVIP employs SPSA-GC for black-box models, whereas AutoVP utilizes classic gradient descent. Given the differing objectives of these two studies, direct comparisons may present certain unfairness. This discussion has been added in Appendix B.14.

Q3. FullyMap and LP

While LP and FullyMap share similarities due to both incorporating a linear head, we conducted a comparison between LP and pure FullyMap (without visual prompts). The results shown in the table below, demonstrate that FullyMap without VP exhibits inferior performance compared to LP on most datasets. This discrepancy may arise from the inability of the pre-trained model's learned features in the linear head to be interpreted for downstream classes, highlighting why FullyMap is not equivalent to LP.

For a more detailed exploration of FullyMap, Appendix B.10 Figure 15 illustrates that FullyMap can be interpreted as a weighted combination of multiple source labels, where some human-readable features may exhibit similarity. For instance, in Figure 15(a), 'bumpy' shows similarities with 'custard apple,' 'disk brake,' and 'pineapple,' while in Figure 15(b), 'scaly' shares similar features with 'boa constrictor,' 'coho,' and 'common iguana.'

While Figure 16(b) portrays IterMap as successfully mapping to classes more closely aligned with the target classes than FullyMap (Figure 16(a)), a significant accuracy gap is evident. In Figure 16, FullyMap achieves 69.96% accuracy, whereas IterMap-1 only reaches 40.77%. This highlights that merely combining source labels is insufficient for superior performance; the weighting within the combination plays a crucial role, precisely what FullyMap achieves.

Q2. The Choices for Pre-trained Model

Within the AutoVP framework, we have opted to utilize state-of-the-art classification models, comprising residual network-based, vision transformer-based, and multi-modal models (such as CLIP), within our pre-trained classifier set. The decision to select similar pre-trained backbones aims to ensure equitable comparisons with previous studies, as highlighted in the fixed model comparison in Tables 2 and 8 of the paper. However, we have also integrated new pre-trained models, such as Swin-T. Notably, in Table 9 -- AutoVP with source model selection, Swin-T emerges as the most frequently chosen model, illustrating its significant role that contributes to the overall performance of AutoVP.

Dear Reviewer aYHC,

Thank you for your time and effort in reviewing our paper. We appreciate that you enjoyed reading our paper, and recognized our proposed framework is simple yet effective. To your questions, we address our response in the following.

Q1. More Discussion on EVP

Thank you for your thoughtful reminder! EVP is indeed a very inspirational paper that motivates our work. While we did cite EVP, our discussion of their findings on input scaling was omitted. We referenced their findings in Section 3. It's important to highlight that the input scaling in AutoVP extends beyond determining the size of prompt frames and resizing images accordingly. We integrate a dynamic optimization method during the training process, elaborated in Section 3: Input Scaling. This represents the most significant departure from EVP, which merely shrinks the input image and fixes the prompt size. Our design optimizes both prompts and prompt sizes, and adjusts the input image accordingly, maximizing the effectiveness of prompts.

Dear Reviewer aYHC,

We are delighted to learn that "Most of my questions have been solved except for Q3". We understand your concern when we cast the fully map as a generalized/improved version of label mapping (i.e., soft label mapping with learnable weights). We will modify the wording in the future version to clarify this point.

To address your remaining concern, we would like to bring to your attention that AutoVP is indeed an improved version of existing VP methods, when we constrain the label mapping to be the same as baseline methods, such as IterMap and FreqMap. For example, in the table below (extracted from Table 2), when setting the same label mapping as IterMap-1, one can see that AutoVP improves the accuracy of ILM-VP. The reason is that the feature of input scaling optimization enabled by AutoVP contributes to improved accuracy. Similarly, when including FreqMap-1 (an existing label mapping method) in AutoVP, we also observe an obvious accuracy boost over ILM-VP, such as the Food101 dataset in the table below.

We hope these case studies can convince the reviewer that AutoVP is indeed an improved version of VP. We are at the reviewer's disposal to answer any remaining concerns the reviewer may have.

Thank the authors for the feedback. Most of my questions have been solved except for Q3. It is still not clear how such a technique can be viewed as an improved version of VP since they change totally different layers in a model, one for input and one for the prediction.

Dear Reviewer aYHC,

As the discussion period is closing soon, we'd like to follow up and see if the Reviewer has had a chance to consider our response. The additional experimental results on the detection task have been provided in our Update Response. We hope our responses and new results are helpful for finalizing the review rating.

Yours Sincerely,

Authors

Q2. AutoVP Novelty and Insights

We understand that as a unified automated visual prompting (VP) framework, the reviewer may feel that it lacks originality because it contains many visual prompting methods as special cases. However, we would like to respectfully point out several new components and designs in our AutoVP that the reviewer may overlook, as well as highlight our major contributions to VP.

[Our novel designs for VP] AutoVP introduces novel components that have not been studied in prior arts in VP, including the automated input scaling, as well as weight initialization when using FullyMap. These additions contribute to the uniqueness of our framework, positioning AutoVP not just as a mere automated tuning tool, but as an advanced and comprehensive framework. It amalgamates a multitude of techniques to discern optimal VP configurations for datasets that exhibit diverse characteristics. More importantly, we also show that these two components are very critical to improving VP performance. For example, the optimal configurations in Table 2 suggest that different datasets prefer distinct image scales, and weight initialization with FullyMap is one of the most frequently selected output mapping methods. In doing so, our work transcends the confines of being solely a tuning tool and stands as a powerful toolset for effectively developing and deploying VP across a range of scenarios.

[Our contribution in demonstrating the advantage of VP] It's worth noting that one of our pivotal achievements is the notable enhancement in VP performance beyond the established linear probing baseline, which had not been demonstrated in prior arts (ILM-VP [5] and CLIP-VP [2]). In particular, our data scalability analysis in Sec. 4.1 shows that AutoVP substantially outperforms LP in the few-shot settings (10% and 1% data usage). We believe this is a significant and exciting finding, because efficient few-shot learning is exactly the motivation for prompting.

[Our contribution in providing new insights for VP] Another noteworthy result is our comprehensive analysis of VP on the OOD/ID dataset, where we show that using AutoVP can significantly improve accuracy (as showcased in Figure 5(b)). This outcome highlights the capacity of AutoVP to operate with a wider prompt space when dealing with OOD datasets like SVHN and GTSRB, thereby leading to substantial accuracy gains. This aspect underscores VP's inherent adaptability to different dataset characteristics and has not been systematically studied in the existing literature.

In response to this comment, we highlighted the novelty of AutoVP in the revision within the following sections. In the introduction, we have emphasized the novel components (automated input scaling and weight initialization) in our main contributions to provide readers with a clear impression at the outset. Additionally, we delve into a detailed discussion of the contributions made by these new modules in Section 5. Furthermore, we've incorporated the reviewer's suggestions to integrate a segmentation task into our revised version, available in Appendix B.6.

Dear Reviewer dWYk,

Thank you for your time and effort in reviewing our paper. To your questions, we address our response in the following.

Q1. Variety of Vision Tasks

We appreciate the reviewer's interest in exploring AutoVP’s performance on other vision tasks. We acknowledge the need for a closer examination of various vision tasks. To address this, we integrated AutoVP into a segmentation task and compared its performance with linear probing on both ID and OOD datasets. The results are provided in Appendix B.6 and table below.

In our VP segmentation framework illustrated in Figure 12, a FullyMap is incorporated after the pre-trained model to enable pixel-wise classification with a custom class number. In contrast, the linear probing approach modifies the last 2D convolutional layer. Table 6 displays the outcomes.

We evaluated segmentation performance using two metrics: IoU (Intersection over Union) score and pixel-wise classification accuracy. AutoVP exhibited superior performance on both metrics in the ISIC dataset. Additionally, segmentation examples highlighted that predictions align more accurately with the ground truth mask when the prompt space is larger (see Figure 13). However, in the ID dataset (Pets), VP performance was inferior to LP. This aligns with our findings in the classification task, where OOD datasets derived greater benefits from visual prompts.

As for detection tasks, we are still working on the experiments. We aim to provide those results before the end of the author-reviewer rebuttal deadline.

Dear Reviewer dWYk,

As the discussion period is closing soon, we'd like to follow up and see if the Reviewer has had a chance to consider our response. The additional experimental results on the detection task have been provided in our Update Response. We hope our responses and new results are helpful for finalizing the review rating.

Yours Sincerely,

Authors

Q3. Other Vision Datasets

There are no specific criteria for selecting datasets, and the datasets were chosen mainly from state-of-the-art VP baselines [1]. We appreciate the reviewer's mention of prominent datasets. Among the mentioned datasets, such as OxfordPets and EuroSAT, these were already included in our chosen sets. Moreover, we incorporated additional datasets—StanfordCars, SUN397, RESISC, and CLEVR—and their results are available in Appendix B.4 Table 5.

Our findings demonstrate that AutoVP generally outperforms ILM-VP and CLIP-VP across most datasets. When compared to linear probing (LP), the out-of-distribution (OOD) dataset, CLEVR, demonstrated a significant 15% increase in accuracy. However, the in-distribution (ID) datasets, SUN397 and StanfordCar, show inferior performance than LP. This aligns with the findings discussed in Section 5: Performance Evaluation on ID/OOD.

[1] Chen et al. Understanding and Improving Visual Prompting: A Label-Mapping Perspective (CVPR 2023)

Q5. Randomness of AutoVP and LP

We thank the reviewer for pointing out this concern. However, the fact that we do not report the standard deviations of other baselines does not mean that they are more stable than AutoVP, but that most results are adapted from original papers, where their authors also do not report them in the paper. We apologize for causing the misunderstanding to the reviewer. To address this, we have provided the standard deviation of ISIC and EuroSAT for both AutoVP and linear probing in the table below. The results show that AutoVP and LP exhibit similar standard deviation, suggesting the stability of AutoVP.

Q4. Accuracy Improvement across Datasets

AutoVP has exhibited noteworthy accuracy improvements on out-of-distribution (OOD) datasets such as SVHN. However, it displayed comparatively lower performance on specific in-distribution (ID) datasets. The considerable variation in performance across datasets is rooted in the extent of dissimilarity between the source and target datasets. This aspect has been discussed in Section 5: Performance Evaluation on ID/OOD.

Q2. CLIP Backbone

We utilized CLIP with the ViT-B/32 vision encoder backbone. We apologize for omitting this information; we have now included it in Section 3: Pre-trained Classifier.

Dear Reviewer PZJW,

Thank you for your time and effort in reviewing our paper. We appreciate that you enjoyed reading our paper. To your questions, we address our response in the following.

Q1. Variety of Vision Task

We focused on image classification as our work is mainly based on previous VP works that also cope with classification tasks. However, we acknowledge the need for a closer examination of various vision tasks. To address this, we integrated AutoVP into a segmentation task and compared its performance with linear probing on both ID and OOD datasets. The results are provided in Appendix B.6 and table below.

In our VP segmentation framework illustrated in Figure 12, a FullyMap is incorporated after the pre-trained model to enable pixel-wise classification with a custom class number. In contrast, the linear probing approach modifies the last 2D convolutional layer. Table 6 displays the outcomes.

We evaluated segmentation performance using two metrics: IoU (Intersection over Union) score and pixel-wise classification accuracy. AutoVP exhibited superior performance on both metrics in the ISIC dataset. Additionally, segmentation examples highlighted that predictions align more accurately with the ground truth mask when the prompt space is larger (see Figure 13). However, in the ID dataset (Pets), VP performance was inferior to LP. This aligns with our findings in the classification task, where OOD datasets derived greater benefits from visual prompts.

As for detection tasks, we are still working on the experiments. We aim to provide those results before the end of the author-reviewer rebuttal deadline.

Dear Reviewer PZJW,

As the discussion period is closing soon, we'd like to follow up and see if the Reviewer has had a chance to consider our response. The additional experimental results on the detection task have been provided in our Update Response. We hope our responses and new results are helpful for finalizing the review rating.

Yours Sincerely,

Authors

Dear Reviewer nUAn,

Thank you for your encouraging review, and for recognizing our paper is well-written, the proposed toolbox is useful and has demonstrated its state-of-the-art efficacy across various image classification tasks. We are thrilled that you enjoyed our paper. To your questions, we address our response in the following.

Q1. Limited Novelty

We understand that as a unified automated visual prompting (VP) framework, the reviewer may feel that it lacks originality because it contains many visual prompting methods as special cases. However, we would like to respectfully point out several new components and designs in our AutoVP that the reviewer may overlook, as well as highlight our major contributions to VP.

[Our novel designs for VP] AutoVP introduces novel components that have not been studied in prior arts in VP, including the automated input scaling, as well as weight initialization when using FullyMap. These additions contribute to the uniqueness of our framework, positioning AutoVP not just as a mere automated tuning tool, but as an advanced and comprehensive framework. It amalgamates a multitude of techniques to discern optimal VP configurations for datasets that exhibit diverse characteristics. More importantly, we also show that these two components are very critical to improving VP performance. For example, the optimal configurations in Table 2 suggest that different datasets prefer distinct image scales, and weight initialization with FullyMap is one of the most frequently selected output mapping methods. In doing so, our work transcends the confines of being solely a tuning tool and stands as a powerful toolset for effectively developing and deploying VP across a range of scenarios.

[Our contribution in demonstrating the advantage of VP] It's worth noting that one of our pivotal achievements is the notable enhancement in VP performance beyond the established linear probing baseline, which had not been demonstrated in prior arts (ILM-VP [5] and CLIP-VP [2]). In particular, our data scalability analysis in Sec. 4.1 shows that AutoVP substantially outperforms LP in the few-shot settings (10% and 1% data usage). We believe this is a significant and exciting finding, because efficient few-shot learning is exactly the motivation for prompting.

[Our contribution in providing new insights for VP] Another noteworthy result is our comprehensive analysis of VP on the OOD/ID dataset, where we show that using AutoVP can significantly improve accuracy (as showcased in Figure 5(b)). This outcome highlights the capacity of AutoVP to operate with a wider prompt space when dealing with OOD datasets like SVHN and GTSRB, thereby leading to substantial accuracy gains. This aspect underscores VP's inherent adaptability to different dataset characteristics and has not been systematically studied in the existing literature.

In response to this comment, we highlighted the novelty of AutoVP in the revision within the following sections. In the introduction, we have emphasized the novel components (automated input scaling and weight initialization) in our main contributions to provide readers with a clear impression at the outset. Additionally, we delve into a detailed discussion of the contributions made by these new modules in Section 5. Furthermore, we've incorporated the reviewer's suggestions to integrate a segmentation task into our revised version, available in Appendix B.6.

Q4. Segmentation and Detection

We acknowledge the need for a closer examination of various vision tasks. To address this, we integrated AutoVP into a segmentation task and compared its performance with linear probing on both ID and OOD datasets. The results are provided in Appendix B.6 and table below.

In our VP segmentation framework illustrated in Figure 12, a FullyMap is incorporated after the pre-trained model to enable pixel-wise classification with a custom class number. In contrast, the linear probing approach modifies the last 2D convolutional layer. Table 6 displays the outcomes.

We evaluated segmentation performance using two metrics: IoU (Intersection over Union) score and pixel-wise classification accuracy. AutoVP exhibited superior performance on both metrics in the ISIC dataset. Additionally, segmentation examples highlighted that predictions align more accurately with the ground truth mask when the prompt space is larger (see Figure 13). However, in the ID dataset (Pets), VP performance was inferior to LP. This aligns with our findings in the classification task, where OOD datasets derived greater benefits from visual prompts.

As for detection tasks, we are still working on the experiments. We aim to provide those results before the end of the author-reviewer rebuttal deadline.

Q3. Mapping Methods

This is a great suggestion! We conducted an output mapping analysis on both FreqMap and FullyMap using the DTD dataset to gain insights into how these mappings have been learned. We have updated these results in Appendix B.10. Figure 15 illustrates that FullyMap can be interpreted as a weighted combination of multiple source labels, where some human-readable features may exhibit similarity. For instance, in Figure 15(a), 'bumpy' shows similarities with 'custard apple’, 'disk brake’, and 'pineapple’, while in Figure 15(b), 'scaly' shares similar features with 'boa constrictor’, 'coho’, and 'common iguana’.

Furthermore, when comparing FullyMap and IterMap, a significant accuracy gap is observed: FullyMap reaches 69.96%, while IterMap-1 only reaches 40.77%. However, in Figure 16, FreqMap has mapped to some classes that are indeed very close to the target. For instance, in Figure 16(b), 'braided' maps to 'knot’, 'bubbly' maps to 'bubble’, and 'cobwebbed' maps to 'spider web’. This demonstrates that a mere combination of source labels is insufficient for achieving better performance; the weighting in the combination plays a crucial role, which is precisely what FullyMap accomplishes.

Q2. Potential Overfitting

We are not sure what the reviewer’s meaning of “overfitting to specific tasks.” In our experiments, AutoVP is designed to automatically search for the best configuration of different downstream tasks given the same set of hyperparameter candidate sets, and its selection would vary according to the given datasets.

We speculate the reviewer would like to learn more about whether AutoVP would “overestimate” at the initial tuning phase. Indeed, as a hyper-parameter optimization (HPO) framework, the resulting configuration among many combinations tends to be overfitted or overestimated. This phenomenon was first discovered as “the multiple induction problem” [1], also known as the “winner’s curse” in statistics. One can search all the combinations and train those models longer to avoid the sub-optimal tuning results. However, this may cost much more computation costs. Some research also suggests that if certain datasets are of good quality (well-balanced, large enough, etc.) then they would not be too sensitive to the hyper-parameter, which means the initial tuning phase is enough, and one can regard the tuning results as the optimal configuration.

In our paper, we also observe a similar phenomenon: when selecting CLIP as the pre-trained model, AutoVP would overestimate the performance of FreqMap and IterMap. That is, FullyMap tends to perform poorly in the hyper-parameter tuning process, yet may achieve higher test accuracy in full training. To address this issue, we use the weight initialization on FullyMap to fix it, which we have discussed in Section 4.2.

[1] Jensen and Cohen. Multiple comparisons in induction algorithms. (Machine Learning)

Dear Reviewer nUAn,

As the discussion period is closing soon, we'd like to follow up and see if the Reviewer has had a chance to consider our response. The additional experimental results on the detection task have been provided in our Update Response. We hope our responses and new results are helpful for finalizing the review rating.

Yours Sincerely,

Authors