VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Dear reviewer nisa,

Thanks very much for your valuable comments. We hope our responses can address your concerns and clarify our contribution.

Q1: The paper title is a bit misleading because the model is an agglomeration of many powerful pretrained models instead of a single model.

A1: We suppose that a network that could be trained end-to-end is considered as a single model. In the deep learning community, It is a common practice to link the different pretrained models in one network, such as Flamingo, LISA, and GLaMM. We extend the one-to-one linking to the one-to-many linking, which still should be regarded as a single model, NOT an agglomeration of many powerful pretrained models.

Q2: Limited contribution regarding the soft prompt and special tokens.

A2: (1) InstructBLIP, ep-ALM, and MAPL use learnable queries (i.e., soft prompts) to connect the modality encoders and LLM. FROMAGe uses a special token for image-text retrieval so as to handle multimodal outputs, where the images are not generated from the network end-to-end. These works still remain constrained to text-based outputs. However, our work is significantly different from theirs in that we support hundreds of tasks by largely extending the output formats for MLLM, e.g., box, mask, keypoint, text, and image.

(2) When scaling to various tasks across a broad range of domains, it requires us to address several challenges: (i) precise decoder invocation, (ii) mitigating task conflicts, and (iii) efficient message transmission in an end-to-end manner. Moreover, as the number of tasks increases, the difficulty of addressing these challenges will also significantly rise. Our proposed super link is a simple but non-trivial solution. It provides a new approache for connecting multiple decoders and coordinating hundreds of tasks, which have not been explored by previous works. Specifically, first, the super link contains a routing token that focuses on scheduling decoders. It can precisely determine which decode should be used after being trained with large-scale of corpus. Second, we use un-shared super-link queries for different decoders, which can mitigate the issue of task conflicts. Last, super-link queries can be easily accessed by their corresponding decoder and act as conditions for different decoders in solving various tasks.

(3) Despite the simplicity of our super link, it is able to extend MLLMs to handle hundreds of tasks across various domains. Without redundant design, a simple yet effective method might provide more clear insights into this research topic. Therefore, we believe that our contributions to the study of generalists are clear and should not be overlooked. We will include the above discussion in the revised version.

Q3: Details about model training.

A3: (1) We have provided the model training details, e.g., training strategy and hyper-parameters, in Appendix D.
Regarding the training objectives for task-specific decoders, we have given brief descriptions in Appendix B.2. Specifically, we keep the original losses for the decoders. Grounding-DINO optimizes the combination of classification loss, box losses (including l1 loss and gIoU loss), and mask losses (including binary mask loss and DICE loss). The training objective for UniPose is the combination of classification loss, box losses (including l1 loss and gIoU loss), and keypoint losses (including l1 loss and OKS loss). For Stable-Diffusion and Instruct-Pix2Pix, we employ two MSE losses: one between the CLIP text features and mapping features, and the other one between the ground-truth noise and predicted noise.

(2) We simply sum up all the losses from LLM and task-specific decoders without reweighting for each exponent, i.e., $L_{\text{total}} = L_{\text{llm}} + L_{\text{gdino}} + L_{\text{unipose}} + L_{\text{sd}} + L_{\text{ip2p}}$.

Thanks to the reviewer for the suggestion. We will add more detailed descriptions of training losses in the revised version.

Q4: Experiment with one-stage training instead of three-stage training.

A4: Please refer to common questions Q2 for the experiments and explanations. We will add this part in the revised version.

Q5: Quantitative comparison for the in-context learning setting.

A5: (1) In-context segmentation

To fairly compare with state-of-the-art methods, e.g., Painter [a1] and SegGPT [a2], we construct a benchmark based on the validation set of COCO2017, where the number of in-context examples used during inference ranges from 1 to 5. The results are shown in the following table.

(2) In-context image captioning

We follow the same evaluation protocol as OpenFlamingo [a3] and use 4-shot to assess the performance between different methods. The validation set is built upon COCO2017. We present the results in the following table.

Generally, VisionLLM v2 exhibits clear performance advantages compared with state-of-the-art methods in both in-context learning settings, which demonstrates the superior in-context capacities of our method. These results will be updated in our final paper.

References

[a1] Wang, Xinlong, et al. "Images speak in images: A generalist painter for in-context visual learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[a2] Wang, Xinlong, et al. "Seggpt: Segmenting everything in context." arXiv preprint arXiv:2304.03284 (2023).

[a3] Awadalla, Anas, et al. "Openflamingo: An open-source framework for training large autoregressive vision-language models." arXiv preprint arXiv:2308.01390 (2023).

Thank you so much for taking the time to re-evaluate our paper, thoughtfully considering our responses, and agreeing to raise the score. We will carefully take your valuable suggestions and continuously improve our paper.

Dear reviewer W77r,

Thanks a lot for your insightful reviews and support for our work! We hope our responses can address your questions.

Q1: Complex training process.

A1: We kindly invite the reviewer to refer to the common question Q2 for the details. GIT [1] mainly focuses on visual tasks and uses a smaller scale of datasets (i.e., COCO, RefCOCO series, COCO caption, ADE20K) for training. And GIT is basically evaluated on these datasets without evaluating its conversation ability. Fuyu-8B [2] is a decode-only transformer that is trained for image-level visual question answering. These two works focus on a single task or related visual tasks, making their training can be completed in one stage.

Q2: Does the performance mainly depend on the task-specific decoder? Does the proposed method aim at facilitating the integration and scheduling of various decoders? Will MLLM enhance the performance of decoders?

A2: (1) The performance largely depends on the task-specific decoders. We propose the end-to-end framework to increase the openness of decoders. For example, we can extend the capacity of Grounding-DINO by adapting to more profound categories, more diverse domains, and more flexible textual conditions.

(2) The answer to the second question, "Does the proposed method aim at facilitating the integration and scheduling of various decoders?", is yes. We integrate several task-specific decoders in one network, and use MLLM to select the task-specific decoder properly.

(3) The advantage coming from MLLM is its strong language processing ability for interpreting and reasoning over the user's prompt. This endows the model with greater textual perception and sentence reasoning capabilities, thereby enhancing the performance of tasks that rely on openness. We have conducted the ablation experiments by replacing the LLM from Vicuna-7B with stronger InternLM-20B. We find that for the grounding task on RefCOCO, the model obtains the significant +2.8 points on P@0.5, which meets our expectations. While for object detection on COCO, we do not see performance improvement. This is because the traditional language model, e.g., BERT, is sufficient for encoding the category information.

Q3: Can joint training of different tasks lead to mutual enhancement?

A3: Please refer to common question Q1 for the details. We use different decoders to complete various tasks, and we do not observe obvious mutual enhancement among tasks in this work.

Q4: Should a general model use a bridging approach (LLaVA or VisionLLM) or a simple multi-layer transformer?

A4: Very good question. From our perspective, the bridging approaches are easier to implement, in order to enhance the LLM with various capabilities beyond the text output.

We agree with the reviewer that the simple multi-layer transformer architecture is more unified and concise and has advantages in simpler training. However, at the current technology level, there are many tasks and modalities in the real world that are hard to model using the next token prediction. For example, there are still some open research questions: Do image patches follow the left-to-right order? How to effectively represent structural outputs such as segmentation masks and protein structure?

In general, we believe it is necessary to integrate specialized models into general models to advance the field of generalist models in the short term.

Dear reviewer x7n8,

Thanks so much for your constructive comments and support for acceptance. We hope our responses can address your concerns.

Q1: More related works.

A1: Thanks for pointing out the missing related works. AnyGPT builds a multimodal text-centric dataset for any-to-any multimodal generation (text, image, speech, music) with sequence modeling. Chameleon uses fully token-based representations for both texts and images, capable of understanding and generating interleaved image-text sequences. CM3 series are autoregressive models for text-to-image and image-to-text tasks. All these works could unify image understanding and generation in one network. Our model can support more vision and vision-language tasks. We will respectfully cite these works and add the discussion to the related work section of our paper.

Q2: Clarification on how super-link queries are generated. Template example for text-to-image generation.

A2: (1) Here is a template example for the text-to-image generation:

USER: Please generate an image with caption: a dog near the sea.
ASSISTANT: Sure, here it is [GEN].

We will add this example in our revised version.

(2) In the following, we would like to give clearer clarification about the generation of super-link queries.

The super-link queries are initialized as learnable weights nn.Embedding. During inference, whenever the LLM predicts the special token such as [DET], [GEN], the super-link queries will be automatically appended after it. This means that, in the current generation step, the size of input embeddings for LLM is expanded from [1, c] to [1 + num_embeds, c], where num_embeds is the number of super-link queries, c is the hidden size of LLM. So, it is true that the super-link queries are generated on-the-fly based on the context.

Q3: Study on how each task influences the others.

A3: We have provided detailed responses in the common questions Q1. Based on the current experimental results, we have not yet observed the positive impact of vision tasks on image generation.

Dear reviewer, the authors have responded to your questions - are you satisfied by their response? If so, would you like to update your rating? Is there any more information you need from the authors to make that decision?