IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks

Thank you for the review and suggestions. We address your comments below.

Q: It would be more convincing to have more quantitative experiments on more tasks, e.g., one-shot or few-shot tasks. Although it might be challenging, it will give readers a bigger picture.

A: We provide results of more Image-to-X tasks in Table 5 and visualization results in Figure 7 and 8.

Q: What about using multiple prompts? For example, multiple images, or even multiple image-text pairs as the prompt. It would be interesting to see what happens and how can further unleash the potential of the model.

A: We provide the results of multiple prompts in Table 8 and Figure 14. It shows our model can handle and benefit from multiple prompts.

Q: Failure cases should be analyzed and discussed.

A: For some of the failure cases in Bar et al, our IMProv could successfully address them with text prompts, e.g. the cat colorization and bottle segmentation of “Task ambiguity”. And our model could successfully generate the image that moves the orange to the center, though fails the other Non-aligned input-output example.

Q: The authors use ViT-L be default. Did you tried backbone at different scales?

A: We tried a larger network (ViT-H) as encoder, but there was no significant improvement. We suspect that scaling up the decoder instead of the encoder may help, but we did not have sufficient computing budget to try it. We plan to explore it as a future work.

Q: If we want to further scale up, how can we get more data like Semantic Scholar Computer Vision dataset? It would be interesting to have more discussion.

A: One potential way to scale up the dataset is by collecting instructional videos and incorporating them during the training process (e.g. videos of academic talks, oral presentations from conferences, and paper summary videos). We will add it to the discussion and explore this direction in future work.

Thank you for the review and suggestions. We address your comments below.

Q: Metrics other than LPIPS for image-to-X and X-to-image tasks.

A: We generally agree, however, in [2*] we followed Wang et al. that used the images from InstructPix2Pix. These images are generated using Stable Diffusion, and there is no ground-truth annotation for the depth/edge/segmentation. Therefore, like Wang et al. we report MSE and LPIPS (see the table below).

Q: The choice of sample images in Figure 2 of the paper, which showcases the pipeline, does not seem to be well-suited and may not have a strong relevance to the tasks addressed in the paper.

A: The sample image in Figure 2 is a random sample from our dataset, which comes from one of the computer vision papers, and represents the common image pattern of our dataset.

Q: Do large language models that exclusively handle text, such as T5, outperform models that encode text using CLIP?

A: We tried both CLIP and T5 text encoder. And we found the larger T5 text encoder doesn't improve the performance. Under “Random Class” visual prompt setting, T5 vs CLIP is 33.26 vs 36.29 mIoU. We choose to use CLIP for all the other experiments to save computation costs.

Q: In Tables 4 and 7, the models trained with CCVF and LAION exhibit significantly better metrics compared to models trained solely with CCVF. Could the authors provide some analysis, possibly accompanied by visual results, to explain the reasons behind this substantial improvement?

A: As reported in the "Dataset effect." section from Bar et al., pre-training on a larger and more diverse dataset helps the model to generalize better.

Thank you for the review and suggestions. We address your comments below.

Q: Dicussion on the difference between figure captions from papers and the prompts used at inference time.

A: Although the training and testing prompts are different, the model is able to generalize to the testing prompts by training on large-scale datasets. During the pre-training, the model learns to extract meaningful representation from the noisy data. For example, CLIP’s image encoder and text encoder are pre-trained on noisy data as well, but it can zero-shot transfer to different images and texts at inference time. And according to Figure 17, the captions of our datasets include terms like “segmentation” “sketch”, “colorization” etc. So the gap between training and inference is not large.

Q: In Table 3, methods are trained with different datasets, thus leading to an unfair comparison. The authors could report the performance of IMProv trained on CCVF.

A: In Table 4, we report the accuracy of MAE-VQGAN and IMProv trained on the same dataset. CCVF just extends CVF by extracting the captions of the figures. CCVF and CVF share the same set of images. MAE-VQGAN only uses images, and IMProv is trained on both texts and images. So row 1 and 2 in Table 4 is a fair comparison between MAE-VQGAN and IMProv, indicating that text prompt can improve the model performance.

Q: For the tasks of X-to-images and images-to-X, the evaluation is not rigorous. For example, LPIPS cannot assess the performance of semantic segmentation. The reported results do not support the claim that IMProv can well generalize to these standard computer vision tasks. Commonly used metrics for these tasks should be adopted for evaluation. In addition, the results of X-to-images look poor, where the generated images are blurry and lack details.

A: In [2*], Wang et al. only reported MSE for all the vision tasks. We follow them to use a single metric for all the tasks. The images from InstructPix2Pix are generated from Stable Diffusion, there is no ground-truth annotation for the depth/edge/segmentation. We will report both LPIPS and MSE in the updated version.

Q: Dataset ablation: IMProv trained with different datasets can indicate the benefits brought from larger datasets, not different methods trained with different datasets.

A: We report the results of IMProv(S2CV+LAION) in the below table under the same “Random Class” setting. CCVF just extends CVF by extracting the captions of the figures. CCVF and CVF share the same set of images. MAE-VQGAN only uses images, and IMProv is trained on both texts and images. IMProv(CCVF) improves over MAE-VQGAN (CVF) by 2 points, indicates text prompt helps when visual prompt is randomly sampled from different classes. IMProv(S2CV + LAION) outperforms IMProv(CCVF + LAION) by 1.7 points, which justifies the effectiveness of our proposed S2CV dataset.

Q: The authors claim that Figure 4 suggests a trade-off between visual and text prompts. However, the visual prompts are the same for a query in Figure 4.

A: In Figure 4, we keep the query the same, and change the support from “without support” (first 2 columns) to “with support” (last 4 columns). We provide more examples of the trade-off between visual and text prompts in Figure 9, with different class random sample prompts, same class random sample prompts, and nearest neighbor prompts.

Q: In Figure 5, what are the specific settings for these variants? “w/o text prompt” means no text prompt during 1) training and inference or 2) just inference? I think only the former could showcase the importance of incorporating text prompts.

A: We imply without text prompt (empty string) during inference. Following MUSE, we drop 10% of the text prompt randomly during training, so our model is able to generalize to "w/o text prompt" during inference.

Q: There are no comparisons with other SOTA in-context learning methods such as Painter [1*] (only a small comparison for the task of colorization is given in the appendix which is far less than enough) and Prompt Diffusion [2*]

A: We acknowledge that our approach performs worse than them on their benchmarks because they are trained in a fully supervised way with ground truth annotations. In contrast, our model is trained on noisy data from the paper. We plan to explore the combination of supervised data and unsupervised in future works.

Dear Reviewer wsst,

Thank you so much again for the detailed feedback. We're approaching the end of the author-reviewer discussion period. However, there are no responses yet to our rebuttal. Please do not hesitate to let us know if there is any further information or clarification we can provide. We hope to deliver all the information in time before the deadline.

Thank you!

Thanks for your reply! We address your concerns as follows.

Q1: Since it’s not trivial to convert the RGB outputs to standard benchmark formats due to the time limit, we follow PromptDiffusion to report MSE and perception scores only. We will evaluate on standard benchmark in the future version.

Q2: We report the comparison with PromptDiffusion in the table below.

Although IMProv underperforms PromptDiffusion on most tasks (except Image -> Normal ), we would like to emphasize that PromptDiffusion explicitly uses the paired dataset to train in a fully supervised way, including depth, HED and segmentation annotations, while ours only uses noisy data from Semantic Scholar and LAION. As mentioned in the fail cases in the PromptDiffusion paper, their model fails to generalize to some unseen computer vision tasks, e.g. Image -> Normal, since there are no such training data pairs in their training set. However, even though our model is trained with noisy data from the web, it still generalizes to output normal maps. Moreover, PromptDiffusion initializes the weights from ControlNet (Stable Diffusion 1.5), which is already a strong model for image generation, while our IMProv is trained from scratch without annotated data.

Thank you for the review and suggestions. We address your comments below.

Q: This paper exhibits limited novelty since the proposed method is primarily a straightforward extension of Bar et al.'s work.

A: In this work, our goal is to explore the tradeoff between text and image prompts based on Bar et al. work. Nevertheless, we show that text prompts can be used to improve the model performance, and we also collect a large dataset for multimodal in-context learning.

Q: Qualitative analyses are relatively scarce within the paper.

A: We provided extensive qualitative analysis in Figures 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 13, 14, both in the main paper and in the supplementary material.

Q: It raises the question of how performance would be affected if the visual prompt is incorrect, and how this compares to using a textual prompt alone.

A: There are no guarantees whether the visual or text prompt would affect the model more. In Figure 6, we provide an example of inconsistent visual and text prompts, and in this example, the model is more likely to follow the text prompt (generate a letter "T").

Q: In the appendix, the authors compare their method to SD. However, it may not be a fair comparison. It would be more equitable if the authors also fine-tuned on CCVF and S2CV datasets to assess whether the SD model can correctly inpaint in those scenarios.

A: We agree that the off-shelf SD is not directly comparable with our method. Our motivation for including the failure of SD is to motivate the necessity to collect S2CV for large-scale multimodal in-context learning.

Q: Whether the limitation of Bar et al. can be mitigated by using a textual prompt.

A: We show the results of failure cases in Figure 16. For some of the failure cases in Bar et al, our IMProv could successfully address them with text prompts, e.g. the cat colorization and bottle segmentation of “Task ambiguity”. And our model could successfully generate the image that moves the orange to the center, though fails the other Non-aligned input-output example.

Q: Have the authors considered how different text encoders might affect the model's performance?

A: We tried both CLIP and T5 text encoder. And we found the larger T5 text encoder doesn't improve the performance. Under the “Random Class” visual prompt setting, T5 vs CLIP is 33.26 vs 36.29 mIoU. We choose to use CLIP for all the other experiments to save computation costs.

Q: The trade-off between the number of support pairs and the use of textual prompts.

A: We plot the mIoU w.r.t the number of support in Figure 15. We run the experiments under grid 4x4 setting in Table 8, with “Random Class” support image. Similar to Bar et al., as we increase the number of support visual examples, mIoU goes up. It’s worth noting that when there are only 1 or 2 examples, the text prompt doesn’t help. It is because the model couldn't generate meaningful segmentation with a small number of visual supports due to the resolution (see Figure 13 for details). Under the setting that visual support number greater than 3, the text prompt consistently improves the results. It implies that our text prompt is orthogonal to the number of visual support pairs.

Q: Fair comparison for S2CV+LAION results.

A: In Table 4, we report the accuracy of MAE-VQGAN and IMProv trained on the same dataset. CCVF just extends CVF by extracting the captions of the figures. CCVF and CVF share the same set of images. MAE-VQGAN only uses images, and IMProv is trained on both texts and images. So row 1 and 2 in Table 4 is a fair comparison between MAE-VQGAN and IMProv, indicating that text prompt can improve the model performance. To compare S2CV with CCVF, we additionally report the results of IMProv(S2CV+LAION) in the below table (Table 10 in supplementary material) under the same “Random Class” setting. We report the average mIoU over 4 splits. IMProv(S2CV + LAION) outperforms IMProv(CCVF + LAION) by 1.7 points, which justifies the effectiveness of our proposed S2CV dataset.

Dear Reviewer yR1X,

Thank you so much again for the detailed feedback. We're approaching the end of the author-reviewer discussion period. However, there are no responses yet to our rebuttal. Please do not hesitate to let us know if there is any further information or clarification we can provide. We hope to deliver all the information in time before the deadline.

Thank you!