InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists

We thank the reviewer for their comments. Please see our responses below.

Q1: The novelty is limited. Formulating computer vision tasks as generative tasks is not novel and has been used in previous works[18]. The overall pipeline highly resembles InstructPix2Pix.

A1: While it is true that the idea of creating vision generalists has been explored before, this does not mean that our method is not novel. The paper in [18] cited by the reviewer uses visual prompting for in-context learning and does not involve any natural language component. While it is true that the architecture of our model resembles that of InstructPix2Pix due to the shared Pix2Pix formulation, the goal of our model, the way its training data is generated and the evaluation are completely different.

Q2: Performance is not impressive at all.

A2: The results in Section 4 indicate that the performance of our model surpasses the state-of-the-art vision generalists [13, 15, 17, 18, 19]. For tasks such as depth estimation and segmentation, our model outperforms other vision generalists on external datasets (Table 1) as well as in open dictionary setups (Table 2). We have also added a new comparison with the Painter model in [18] as well as an additional comparison with a task specific depth estimation model (See Tables below), both showing a competitive performance by our model. Note that our goal is not to outperform specialized models optimized on the target datasets, but rather to show the feasibility of instruction-guided execution of vision tasks and compare its performance and generalization abilities with comparable vision generalists.

We thank the reviewer for the thoughtful comments and feedback.

Q1: The necessity of using a single diffusion model compared to an ensemble of experts.

A1: An ensemble of experts, where Mask-RCNN is used for segmentation, ResNet for classification, etc., necessitate training each specialized component separately on a predefined set of categories. This type of model lacks the ability to generalize to new categories not covered in the training data. In contrast, our approach employs a single unified model, eliminating the need for training separate models for each task. This unified model can effectively transfer knowledge across various tasks and categories.

Q2: Suggestions about the structure of the paper.

A2: Thank you for the suggestion, we will make the necessary modifications in the final version of the paper.

We thank the reviewer for the thoughtful comments and feedback. Please see below a point-by-point response.

Q1: Could the author provide further explanation regarding the unique advantages of this approach?

A1: Thank you for this thought-provoking comment. We realize that the way we implement instruction-tuning in our setup may seem different from its application in NLP, where the primary goal is zero-shot generalization to entirely unseen tasks. However, we attribute this distinction to the structural differences between language and vision tasks. Vision tasks inherently involve more structure, where the anatomy of a vision task combines a canonical task type (e.g., segmentation) with a specific category or object. In this narrower sense, we believe that InstructCV exhibits generalization to new tasks by effectively handling segmentation, classification, and detection for categories not encountered during the instruction-tuning dataset training. This capability arises from the T2I formulation of InstructCV, allowing it to comprehend both the intended tasks and the associated categories and apply the learned task to new categories in an open dictionary fashion. We consider this a unique advantage of our approach—the ability to leverage semantic correspondence between instructions and images to establish a shared representation capable of generalizing the four tasks to new categories within an open dictionary framework. Furthermore, we believe that our model takes initial steps toward zero-shot generalization to entirely new tasks beyond the four considered. The rephrased instruction training equips the model to understand language descriptions of tasks, which fundamentally differs from the rigid task indicators of traditional multi-task learning— this flexibility is another key advantage of our model. For instance, one could envision implementing a panoptic segmentation task in a zero-shot manner by providing a natural language description that combines semantic segmentation and classification tasks. Although such capabilities may fully emerge in a future version of InstructCV trained on a larger number of tasks, the rephrased model represents a foundational step towards achieving this goal.

We will incorporate the discussion above in the final version of the paper.

Q2: I believe the good generalization ability to new datasets is mostly because the comparison baselines lack the ability to handle open vocabulary. On the other hand, T2I models are trained on a large amount of text-to-image pairs, which naturally gives them a good open vocabulary capability. Additionally, some unified models like x-decoder also demonstrate excellent generalization ability to new categories.

A2: Yes, the strong generalization ability of T2I which is trained on a very large dataset and that is the reason why we choose T2I to achieve the Vision Generalists. This is the key idea behind InstructCV; the pretrained multimodal representation can be repurposed to create a model that implements vision tasks in open vocabulary generalization. Thank you for bringing up the paper on X-decoder. While it is indeed related to our work, the authors in that paper have slightly different motivations and goals. We appreciate this insight and will incorporate a discussion on it in the final version of our paper.

Q8: In the second paragraph of Section 1, the authors claim that “However, these prompts consist of (uninterpretable) numerical values derived from specific training datasets, which may limit their ability to generalize to new datasets, tasks, or categories”. Why do the prompts used in inpainting-based methods (which are visual examples) contain uninterpretable numerical values?

A8: We appreciate the reviewer for bringing this to our attention. The statement in question was intended to address prompt tuning approaches. These approaches involve learning task-specific "codes," such as input tokens or pixels, to guide Seq2Seq models in executing the desired task. This specifically pertains to the first category of methods discussed in paragraph 2 on page 2. It's important to note that this doesn't apply to [18] and [19], as they learn the vision task in-context using interpretable visual examples. To avoid any confusion, we will explicitly state that this statement is referring to prompt tuning and not in-context approaches.

Q9: The format of references is messy and inconsistent. For example, a bunch of references do not list the conference/journal title.

A9: Thanks for pointing this out. We will revise the formatting of references in the final version of the paper.

Q10: Just a reminder that there is a very similar work, InstructDiffusion, to this submission. The authors could consider citing that paper and provide comparisons and discussions.

A10: Thank you for your suggestions. We were not aware of InstructDiffusion at the time of submission, as it is concurrent with our work. We appreciate your diligence in bringing this to our attention. To address this, we will include a citation for InstructDiffusion and incorporate discussions about it in the paper.

Q3: More ablations should be conducted to compare InstructCV-FP with InstructCV-RP. For example, the authors can report the results of InstructCV-FP in Table 1.

A3: We have added the performance of InstructCV-FP in Table 1.

We will also include both versions of InstructCV to all the performance comparison Tables.

Q4: For the organization of this paper, I suggest the authors move the introduction of diffusion models in Section 2.2 (which takes up most of this section) to a “Prerequisites” subsection or Appendix.

A4: Thanks for the suggestion. We will make this change in the final version of the paper.

Q5: The proposed method claims to be a vision generalist but it is validated only in some visual recognition tasks. What about other vision tasks such as low-level tasks (e.g., denoise, deblur, and derain) and generation (as done in Prompt Diffusion)?

A5: The InstructCV pipeline is not limited to the 4 tasks considered in the paper, but can be applied to a much broader range of vision tasks including low-level tasks as well. Following [18], we fine-tuned InstructCV to conduct 3 more low-level vision tasks: Denoising (SIDD dataset), Deraining (Rain14000, Rain800, Rain100H, Rain100L, Rain1200 datasets) and Enhance (LOw-Light dataset). (Due to the time constraints, we were only able to evaluate the InstructCV-FP model.) Results are provided below.

We will add the results on low-level tasks in the final version of the paper.

Q6: Further comparisons with other SOTA unified models such as Painter should be conducted. Painter is currently performing the best in prediction tasks as a vision generalist.

A6: Thanks for this suggestion. We have added Painter as a baseline in Table 1 (See below). On tasks supported by Painter, our InstructCV-FP model achieved higher performance on external datasets, and on non-external datasets, InstructCV-FP also achieved higher performance except for NYUv2.

Q7: Comparing inference speeds with baselines.

A7: We believe that our claim on InstructCV exhibiting much faster inference speed than previous methods such as Unified-IO is fair, because Unified-IO also infers one class at a time, and the inference times we correspond to speed of inference per class. The inference time is 5 seconds and not 10 seconds, the discrepancy arises from an earlier version of our model. We will fix this error in the Appendix of the final paper.

Q2: Utility of training using rephrased instructions

Q2.1: What type of instructions is used during inference?

A2.1: In Tables 1 and 2, we use randomly generated rephrased prompts during inference for evaluating the InstructCV-RP model. These rephrased prompts were sampled at inference time and were not drawn from the prompts used for training.

Q2.2: Why training InstructCV using rephrased prompts and why do we need to leverage LLMs as this could be readily done by humans?

A2.2: Please note that the InstructCV-RP model was not trained only on the 7 prompts in Table 4. The 7 rephrased prompts in this table are illustrative examples of the sampled prompts from the LLM (as highlighted in the Table caption). To achieve good generalization performance, the model was trained on a large number of rephrased prompts repeatedly sampled from the LLM. We will emphasize this in the final version of the paper. The usage of the LLM is useful for both creating a scalable framework for training InstructCV on diverse rephrased prompts, as well as evaluating the model on truly diverse and seen rephrasing of the original instructions by repeatedly sampling new prompts at inference time. To further illustrate the generalization performance of InstructCV-RP to new instructions, we conducted further experiments of the segmentation performance in the ADE20k dataset as highlighted below.

Additionally, we tested the robustness of InstructCV-RP to prompt rephrasing through a different LLM (GPT-4), and calculated the average and variance of the results. The findings highlighted below indicate the robustness of InstructCV-RP to unseen instructions.

Regarding the utility of rephrased prompts: We would like to stress that the primary goal of this paper is to create a model capable of performing standard computer vision tasks based on natural language instructions. This approach aims to enhance the model's ability to generalize to both (1) previously unseen categories and (2) novel user-generated instructions.

The importance of generalizing to unseen categories lies in the model's capacity to apply knowledge gained from training on specific categories in one dataset to perform tasks on entirely new categories in a different dataset. In the context of InstructCV, its ability to achieve this (Goal 1) is attributed to its text-to-image formulation, a quality applicable to both fixed and rephrased prompts.

While it is true that the same vision tasks can be executed using fixed prompts, this is not unique to InstructCV and is true for any multi-task model not reliant on natural language inputs. The utilization of rephrased prompts serves a different purpose: it does not aim to enhance performance on standard tasks but rather to improve the model's adaptability to diverse instructions and expressions from end users (Goal 2).

These capabilities are envisioned to be particularly beneficial for achieving zero-shot generalization to entirely new tasks beyond the four fixed tasks used in training. For instance, one could envision implementing a panoptic segmentation task in a zero-shot manner by providing a natural language description, combining semantic segmentation and classification tasks. While such capabilities are only likely to emerge in a version of InstructCV that is trained on a larger number of tasks, the rephrased model represents an initial step towards this goal.

We thank the reviewer for the thoughtful comments. Please see below a point-by-point response.

Q1: Clarification of the visual encoding of outputs for different tasks.

Q1.1: Distinction between semantic segmentation and referring segmentation

A1.1: We appreciate the reviewer's comment. It is accurate to note that, like other baselines such as "Unified-IO" and "Inpainter," our approach to encoding and implementing segmentation is analogous to referring image segmentation. Note that this is a consequence of our framing of all vision tasks within a common text-to-image generation setup, which means that all segmentation tasks are prompted by natural language instruction. Semantic segmentation can be implemented by querying classes one-by-one. We focused on the more standard semantic segmentation task in this paper since many of the baselines (e.g. the specialized architectures) cannot condition segmentation on natural language inputs. We will add a discussion on this issue in the final version of the paper to clear up any confusion.

Q1.2: What do the authors mean by "cross-reference the bounding boxes with the dataset annotations" in Appendix A.3?

A1.2: The post-processing steps for the object detection task involve obtaining the coordinates of the bounding boxes displayed in the visual output of InstructCV using out-of-the-box post-processing functions in the OpenCV library. After that, we calculate the mAP@0.5 metric using the standard procedure in the COCOeval library. The “cross-referencing” statement was meant to describe the evaluation and not the post-processing procedure, but was misplaced in the Appendix. We will fix this error in the final version of the paper.

Thank you so much for the prompt feedback. We apologize for not reflecting the changes and new experiments in a revised manuscript, we did not realize that we can upload a revision during the rebuttal period. We did our best to incorporate as many of the changes mentioned above as well as the new experiments in the revised version. In particular, we incorporated the following changes:

Q1.2: We clarified the statement on "cross-referencing the bounding boxes with the dataset annotations" in Appendix A.3.

Q2.1: We clarified the types of instructions used during inference with InstructCV.

A2.2: We clarified that the InstructCV-RP model was not trained only on the 7 prompts in Table 4.

Q3: We incorporated the results of InstructCV-FP in Table 1.

Q5: We added the results on. low level vision tasks in a new Table in the Appendix.

Q6: We added the results of the Painter baseline in [18] to Table 1.

Q7: We clarified our comments about the inference speed and its comparison with that of Unified-IO.

Q8: We clarified our comment on the difference between prompt tuning and in-context learning.

A10: We added a citation to InstructDiffusion.

We will incorporate all the remaining changes in the final paper.

Q1.1: On using referring segmentation instead of semantic segmentation.

A1.1: Thank you for your suggestion. We agree that referring segmentation (and not semantic segmentation) would be a more natural task to incorporate in our pipeline, since as the reviewer points out, sequential segmentation of classes would require a manual prespecification of these classes. The reason we initially considered semantic segmentation is that this would enable us to compare performance with existing vision generalists which do not incorporate a natural language component. We conducted a new evaluation experiment focused on referring segmentation, and will replace semantic segmentation with this task in the final manuscript. We followed the work in [C1] in preparing the datasets, and used overall mean Intersection over Union (oIoU) as an evaluation metric. We considered three datasets RefCOCO val, RefCOCO test A, RefCOCO test B for evaluation. We did not evaluate other vision generalists (such as Painter and Inpainting), because they use visual prompts for semantic segmentation, and are not naturally fit for conducting referring segmentation. Please note that InstructCV does not use the RefCOCO dataset to train the segmentation task, but as the results below show, it can achieve competitive performance compared to specialized model training on the RefCOCO dataset. Further, InstructCV significantly outperformed the Unified-IO vision generalist.

Q2: I understand that one benefit brought by utilizing rephrased prompts is the better generalization to new instructions given by users during inference. However, this is not appealing given that InstructCV-FP attains better performance than InstructCV-RP (see Table 3 in the main paper and the table provided in A3). The authors further claimed that adopting rephrased prompts is beneficial for the generalization to previously unseen categories, but I found this is not the case. As shown in the table provided in A3, InstructCV-FP still outperforms InstructCV-RP on external datasets.

A2: We believe that Table 3 shows that InstructCV-RP outperforms InstructCV-FP on almost all rephrased instructions. In the new result in A3, InstructCV-FP was evaluated on fixed prompt whereas InstructCV-RP was evaluated on rephrased prompts since we believe that we would be a more fair comparison (i.e., InstructCV-FP was not designed to handle varying user-written instructions). Thus, the differences in performance result from differences in the evaluation procedure and not the capabilities of the two variants of our model. If both models are to be evaluated with the same prompt template, InstructCV-RP outperforms InstructCV-FP on most tasks as shown in the Table below.

We also believe that the reviewer may have misinterpreted our response regarding the usefulness of rephrased prompts. In our previous response A2.2, we mentioned that our model has two goals of generalizing to both (1) previously unseen categories and (2) novel user-generated instructions. As we mentioned in A2.2, the utilization of rephrased prompts does not aim to enhance performance on standard tasks but rather to improve the model's adaptability to diverse instructions and expressions from end users (Goal 2), which we believe is demonstrated in Table 3. We did not claim that rephrased prompts enable generalization to unseen categories (Goal 1), as mentioned in A2.2, these capabilities result from the text-to-image formulation of the vision tasks. Overall, we think the rephrased prompts do not hurt the performance and enable better generalization to different ways of writing the same instructions in natural language. Finally, we wanted to stress that the rephrased prompting is only a variant of our original model and is not the key message of our paper. While we have not tested the ability of our model to utilize its language comprehension to execute more complex tasks (panoptic segmentation or compositional tasks) in a zero-shot fashion, we believe InstructCV-RP takes first steps towards building models that can implement completely new vision tasks based purely on language descriptions.

References

[C1] Henghui Ding, Chang Liu, Suchen Wang, Xudong Jiang. Vision-Language Transformer and Query Generation for Referring Segmentation. ICCV 2021.

[C2] Edgar Margffoy Tuay, Juan C Perez, Emilio Botero, and Pablo Arbelaez. Dynamic multimodal instance segmentation guided by natural language queries. ECCV 2018.

[C3] Zhiwei Hu, Guang Feng, Jiayu Sun, Lihe Zhang, and Huchuan Lu. Bi-directional relationship inferring network for referring image segmentation. CVPR 2020.