UniVis: A Universal Framework for Computer Vision Tasks

We thank the reviewer for their valuable comments. We address your concerns below:

Q1: Average performance on visual understanding/low-level image processing and high training costs.

Constrained by the pretraining model (i.e., SD), UniVis performs better in image generation (Table 3) while average in visual understanding and low-level image processing (Table 1&2). This indicates that the framework is coupled with the training format of the pretraining model. Furthermore, the authors claim that there is no need for high training costs, but this method requires more parameters.

A1: (1) We would like to clarify that one major contribution of this paper is to build a unified framework for vision tasks through generative modeling rather than commonly adopted contrastive learning or masked image modeling. Different from the well-known dictum by Richard Feynman, "What I cannot create, I do not understand", we find it very challenging for the generative models (like PromptDiffusion) to perform well on other vision tasks, especially the visual understanding tasks. Hence, the main aim of this work is making the generative model simultaneously work well on three distinct categories of vision tasks. To this end, we resort to a SOTA text-to-image model, which is one of the very few trained on web-scale data, and we carefully devise an instruction tuning pipeline to adapt it to downstream tasks. As you stated, we achieve competitive results on visual understanding and low-level image processing and exhibit SOTA performance on image generation (see Figure 5 in the revised paper). In contrast, our competing methods (Painter and PromptDiffusion) experience a clear collapse or near breakdown on one of the three categories of vision tasks (see first table in general response #1). Please refer to our general response #1 for more details.

(2) We are a bit confused about the comment saying "requires more parameters". If you mean UniVis-st requires distinct model parameters to tackle different tasks (please let us know if you did not mean this), we would like to clarify that we also present models (UniVis-sc and UniVis-mc) that are jointly trained on multiple tasks (see Tables 2, 3, and 4), which use shared model parameters. The main contribution of our paper is a unified framework for visual task learning, and it can be employed to produce three types of models under different practice scenarios (e.g., the computing power at hand).

Q2: Joint training does not yield significant gains.

When using multiple datasets for joint training, there is no significant gain, even a slight decrease, for the three different categories of tasks (Table 4). This seems to suggest that this work is limited when using only one model to unify computer vision tasks (joint training doesn't yield gains). If it's just about unifying the training format, it doesn't seem to be very meaningful, as it doesn't outperform task-specific models on specific benchmarks.

A2: We would like to emphasize that our primary focus is to investigate how to induce a profound understanding of vision tasks (which involve very disparate visual signals and data domains) through a shared scheme of generative modeling instead of seeking gains with multi-task training. Please see our general response #2 for more details.

Q3: More generalization results.

Fig4 is significant as I personally believe one of the benefits of unifying computer vision tasks should be the ability to generalize to other unseen tasks (analogous to LLM), but unfortunately, only one figure is provided to demonstrate this. How to further enhance generalizability is an interesting point.

A3: We have added more results in the updated paper (please refer to Figure 17). They verify the generalization capability of the proposed framework. We totally agree that performing out-of-distribution inference is one of the intriguing properties of unified models. Gathering more diverse data for training UniVis could be promising to enhance generalizability and we plan to investigate this in future work.

We thank the reviewer for their insightful comments. We are willing to address all the concerns raised in your review:

Q1: Model performance.

While the authors claim reduced computational resource usage, the model's performance significantly lags behind the open-sourced baseline model, Painter, as evident in Table 2.

A1: Please see our general response #1.

Q2: Universality with text instructions.

The framework's universality is some kind of diminished by its reliance on task-specific text instructions. A more compelling setup would operate without any task prompts. Text instructions should be supplemental, enriching tasks like text-to-image generation with finer details, rather than being a mandatory prerequisite for all tasks.

A2: (1) Text instructions are indeed supplemental in our design. An ablation result regarding different types of text instructions is given in Table 5. To further explore this, we conducted more experiments during rebuttal and provided the results below (added to Tables 9 and 10 in the revised paper as well).

As can be seen, text instructions are beneficial for some tasks (e.g., semantic segmentation, depth estimation, and mask-to-image), but the gain by applying task prompts is very marginal for other tasks (e.g., deraining and denoising). Therefore, one can optionally apply text instructions during inference to strike a balance between better performance and extra human efforts. (2) We do not think incorporating task-specific text instructions undermines the universality of the proposed framework. From the perspective of a user, one is aware of the task name when constructing the image instruction (input-output pair), thus the text instruction can be given by 1) an empty string, or 2) a task-level prompt (e.g., "depth map"), or 3) an instance-level prompt (could use the help from some off-the-shelf tools such as image captioning methods). Our claimed "universality" lies in the shared framework for various computer vision tasks.

Q3: Task prompt limitation.

The inclusion of task prompts detracts from the intriguing properties of in-context instruction tuning. I think, could be wrong, the essence of in-context learning lies in the model's emergent properties, deciphering the logic and connections within paired VISUAL inputs to undertake related tasks. Ideally, we'd want to furnish only in-context visual cues, enabling the model to manage a myriad of downstream tasks. Yet, the current design seems to veer away from this ideal.

A3: We agree that the essence of in-context learning may lie in the model's emergent properties. The model we utilize is Stable Diffusion (SD) which translates textual inputs into realistic images, and a bunch of prior works has demonstrated that the knowledge learned in SD contains a rich understanding of both visual and linguistic signals. In light of this, we opt to exploit the inclusion of task prompts for UniVis, which we believe could be fruitful for unlocking the unifying capability of a pre-trained SD model.

Q4: Ablation studies regarding task prompts.

I'm curious how the model performs without relying on task prompts. Have you conducted any ablation studies to shed light on this aspect?

A4: An ablation result regarding different types of text instructions is given in Table 5. We conducted two more experiments during rebuttal and provided the results in Tables 9 and 10 in the revised paper. As can be seen, text instructions are beneficial for some tasks (e.g., semantic segmentation, depth estimation, and mask-to-image), but the gain by applying task prompts is very marginal for other tasks (e.g., deraining and denoising).

Thank you for the insightful review. Below, we answer all the questions.

Q1: Limited to dense image tasks.

The proposed method is limited to dense image tasks, where the output is at a high-dimensional image level. It does not demonstrate feasibility on a wide range of computer vision tasks where the output is low-dimensional, such as image classification, image captioning, VQA, etc.

A1: (1) We have evaluated UniVis on keypoint detection, where the output is low-dimensional (please refer to Figures 3, 8, and 17). (2) Other computer vision tasks with sparse outputs could be tackled by our framework as long as one can transfer the output to RGB images. Taking image classification as an example, a class label can be turned into an RGB image filled with a unique color determined by its semantic class. Due to limited time during rebuttal, we cannot provide the results on image classification at this time, but we promise to include them in the final version if the paper gets accepted. (3) We would like to clarify that this paper focuses on standard computer vision tasks instead of vision-language ones including image captioning and VQA, which are also not explored by our competing methods (Painter and PromptDiffusion).

Q2: More dense image tasks.

Even for dense image tasks, it misses multiple tasks. For example, for the task of conditional image generation, the paper misses two important tasks: class-conditional generation and text-conditional generation. Thus, I feel the claim that Univis is a "Universal Framework for Computer Vision Tasks" is exaggerated, making the contribution of the paper limited.

A2: (1) Text-conditional generation can be fulfilled by directly applying UniVis-sc trained on four conditional image generation tasks where the query is set to a black image. Please see Figure 16 in the revised paper for visual results and more details in the updated Appendix (Sec. C). (2) Class-conditional generation could also be achieved by framing it as an inverse task of image classification (just like depth-to-image generation and depth estimation). We promise to provide the results on class-conditional generation in the final version if the paper is accepted. (3) We believe that a wide range of vision tasks from three distinct categories evaluated in our paper would adequately support our major claims.

Q3: Weak performance on visual understanding and low-level image processing tasks.

Overall, the experiment results shown in the paper are weak. In two of the three categories, it is far behind the Painter baseline. It might be because of computation limitations, but it also might be because the proposed method does not scale with more computation resources. Therefore, it would be good to show at least one experiment that is trained with the same computation budget as the Painter baseline, otherwise, the results are not convincing enough to demonstrate any improvement over previous methods.

A3: Please see our general response #1. The results provided in the second table in general response #1 verify that UniVis scales well with more computation resources.

Q4: Scaling ability with more tasks and data.

The method does not show scaling ability with more tasks and data. As shown in Table 4, a model trained with multiple tasks is worse than a model trained on a single task. This raises concerns that this method may not scale up well with multi-task training, which is not desirable in a universal framework.

A4: We would like to clarify that our primary focus is to investigate how to induce a profound understanding of vision tasks (which involve very disparate visual signals and data domains) through a shared scheme of generative modeling instead of seeking gains with multi-task training. Please see our general response #2 for more details.

Thank you for the authors' rebuttal and the clarification regarding the major aim of the paper. After reviewing the rebuttal, I have reassessed my original score, leading to its reduction. This decision stems from the authors' statement that the main goal of the paper is to "reveal the potential of generative modeling in building a universal solver for vision tasks", while not "seeking gains with multi-task training". This perspective notably diminishes the significance of a 'universal solver'. If UniVis does not have the ability to improve the performance with more tasks, a basic 'universal solver' might simply be a combination of Painter + PromptDiffusion, which shows better performance across all three evaluated tasks than UniVis.

In short, if a combined approach (UniVis-mc) does not elevate performance in any of the individual tasks (UniVis-st), then the benefit of this combination is quite constrained. It prompts the question: why not select the most effective method for each specific task and combine them to be a "universal solver"? One minor benefit of a universal solver with poor performance in all tasks could be a reduction in the total number of parameters required, but this aspect is not addressed in the paper.

Thank you for the detailed and constructive review. We are happy to follow your comments to further improve this paper:

Q1: Estimation of the experiment-to-experiment variation

My biggest concern is that the results in the tables cannot be understood without knowing the variation that might be produced from one repetition of an experiment to the next. It is crucial that you state whether you only ran each experiment once, or whether the numbers you are reporting are the averages across several trials. If the latter, it is also essential to provide some measure of variation (standard deviation, confidence interval). If you have no estimation of the experiment-to-experiment variation, how do you know that what you are observing as differences in the tables is not simply noise?.

A1: We have updated some tables in the paper to include the average scores and standard deviations across three trials. From the added results, we could find that the proposed UniVis is very stable with small stds. Due to limited time and computing resources, we are unable to run all experiments multiple times during rebuttal. We promise to update all tables in the final version if this paper is accepted.

Q2-1: Vague assertions/claims in the introduction.

At the bottom of p. 1, you say that LLMs exhibit "superior linguistic understanding due to the highly organized and structured nature of language." But images also have significant structure. You go on to say that the "disparity between low-level features and high-level semantics... is typically more pronounced in vision than in text." This is all very vague. Do you need to say it? If you do, could you be more precise about how you are assessing "disparity" or the degree of organization or structure?

A2-1: In the second paragraph of Section 1, we have changed "...due to the highly organized and structured nature of language." into "...due to the semantic-dense nature of language created by humans." We find that the description regarding "disparity" is indeed vague, and we thus remove it to avoid misunderstandings.

Q2-2: Vague assertions/claims in Section 3.2.

At p. 6, you say that "patch-level inpainting that resembles word completion training schemes in NLP is not adequate for a holistic and profound understanding of vision tasks because images have much lower semantic density than texts." This is vague. I think I know what you mean: that there is a lot of redundancy and spatial correlation in images that is not present in text. Could you make this more precise?

A2-2: We have updated the description as follows (see last paragraph of Section 3.2): "...due to the fact that the correlation between pixels is much stronger than that between words (e.g., this redundancy presented in images makes the model readily inpaint a patch with neighboring patches)."

Q3-1: Under-explained concepts regarding DDIM.

DDIM is mentioned at p. 16 (B.2) without any explanation.

A3-1: We have added a brief introduction to DDIM in the revised paper as follows (see Appendix B.2): "By setting the random noise in the reverse diffusion process to 0 (i.e., a deterministic sampling), DDIM manages to generate an image with fewer sampling steps compared to DDPM."

Q3-2: Under-explained concepts regarding U-Net.

I know U-Net is a well-understood term-of-art by now, but I think it could still use a brief explanation of its purpose, given that it is what is being trained to fit the distribution of latent codes.

A3-2: We have added a brief explanation of the purpose of U-Net in the revised paper as follows (see first paragraph of Section 3.2): "Specifically, a denoising U-Net is trained to fit the distribution of latent codes, which models the reverse diffusion process. Taking the noisy latent and the time step as input, this U-Net is further conditioned on the textual embeddings extracted through a text encoder CLIP via cross-attention to produce the output at the current time step."

Q3-3: Under-explained concepts regarding spatial-wise concatenation.

You use the phrase "spatial-wise concatenation" at p. 4 (first paragraph of 3.1). Can you describe this? I think you simply mean you can stitch the images together in a grid as you visualize in Figure 2, right?

A3-3: Yes it is, we have appended a brief explanation of "spatial-wise concatenation" in the revised paper as follows (see first paragraph of Section 3.1): "we can implement spatial-wise concatenation of any input, out, and query samples (i.e., stitching all the images together into a grid as illustrated in Figure 2)."

#2. Joint training does not yield significant gains

As shown in Table 4, UniVis-mc exhibits a competitive performance very close to UniVis-st. UniVis-sc produces slightly better results than UniVis-st on most of the tasks (see Tables 2 and 3). These results indicate that the proposed framework has a large potential to unify various tasks including discrimination and generation which are typically studied exclusively, which is indeed the aim of our work and beyond the reach of competing methods (see first table in general response #1). Seeking gains with multi-task training is not our focus (we do not explore any multi-task training skills as well, especially under limited computing resources).

As pointed out by Reviewer YRyF, the ability to produce a single trained model that can generalize across diverse categories of computer vision tasks by simply altering the "instruction" would be very useful. We also showcase the generalization capability of UniVis in Figure 4 and Figure 17 (newly added to the revised paper). We hope these merits of UniVis would encourage further investigation on how to induce a profound understanding of vision tasks (which involve very disparate visual signals and data domains) through a shared scheme of generative modeling.

We address all other concerns in individual replies to the reviewers.