VT-PLUG: Integrating Visual Task Plugins with Unified Instruction Tuning

I have some concerns as follows:

1. The modular components of VT-PLUG largely overlap with existing multimodal model plugin integration frameworks. In addition, the proposed VD-CoT mechanism fails to clearly demonstrate its unique advantages over existing chain-of-thought (CoT) and visual decoding methods, giving the impression of merely borrowing the CoT concept.

2. Although the paper claims high performance across various tasks, the results do not significantly surpass state-of-the-art models on tasks such as Visual Question Answering (VQA) and image captioning. Additionally, it lacks a comparative analysis with advanced vision-language models, making it difficult to justify the added complexity of the proposed method without substantial performance improvements.

3. The framework's reliance on separate plugins for each visual task increases system complexity, potentially limiting its scalability and applicability in real-world scenarios. This integration approach may incur high computational costs, which the paper does not adequately address.

4. Notable comparable methods, such as the VisionLLM v2 series, which shares similarities in task and model design, are not discussed. Additionally, pioneering works introducing visual plugins, such as Visual ChatGPT, HuggingGPT, and MM-REACT, are also absent from the discussion.

Overall, this paper does not offer a truly novel and impactful solution. Due to the overlap with existing methods, limited empirical improvements, and the lack of clarity in articulating technical contributions, my rating is "reject".

1. Proposed methods use existing expect tools to decoder visual predictions, which is unfair to compare with previous MLLMs. Such as Shikra has no decoder stage after LLM decoder.
2. As MLLM in the proposed framework works as a router and generator for inputs to downstream decoders, I think this work is closer to ones using MLLM as an agent but not an end-to-end model. Thus, maybe it should be compared with works like huggingGPT.
3. Organization of this paper is somehow poor and hard to follow. I think putting the overview figure for the whole pipeline starting from input images and ending with target predictions would be better. Definition of plugins can be explained in further details.

Shen, Yongliang, et al. "Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face." Advances in Neural Information Processing Systems 36 (2024).

Cons:

1. The related works lack a comprehensive review of literature on enhancing MLLMs with visual prediction capabilities. Key examples include the text-based method VITRON [1], and end-to-end methods like Next-GPT [2], VisionLLM series [3,4]. I think it is better to move the Sec.4.1 to the related works.
2. The process of VD-CoT is unclear to me. When is VD-CoT performed? How is the model supervised with VD-CoT during training? In Figure 2, when the user prompt requests a visual task, does the model directly output the answer?
3. The statement of group hungarian matcher is very brief in Sec.4.2, while there is the ablation study of it. Could you provide more details about group hungarian matcher and its difference with standard hungarian matcher?
4. From Table 4 and examples in Appendix, It seems that the performance of segmentation is not well. What is the reason? Did the authors train the model on the segmentation tasks only?
5. Why the VD-CoT would prevent task conflict? As to me, it is a technique to select the proper decoder and give the semantic learnable queries for decoders. The performance conflict between different decoders is not well studies.
6. The paradigm of integrating MLLMs with multiple meta decoders has been explored in studies [2,4]. I would like to hear from the authors about the differences between this submission and previous methods.

[1] Fei, H., Wu, S., Zhang, H., Chua, T. S., & Yan, S. (2024). VITRON: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing.
[2] Wu, S., Fei, H., Qu, L., Ji, W., & Chua, T. S. (2023). Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519.
[3] Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., ... & Dai, J. (2024). Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36.
[4] Wu, J., Zhong, M., Xing, S., Lai, Z., Liu, Z., Wang, W., ... & Dai, J. (2024). VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks. arXiv preprint arXiv:2406.08394.

1. Although the model is able to perform various tasks, the performance in each domain is questionable. In the quantitative results, this paper seems to be deliberately comparing with weaker models. Examples include but are not limited to:

   • Table 2: Only earlier grounded MLLMs are compared. An early baseline, LLaVA-1.5-7B (the "fundamental MLLM" module in the proposed method should be comparable to it) can achieve 78.5 accuracy on VQAv2.
   • Table 3: On POPE, "F1 score" should be the main metric to compare with, as one may achieve a very high precision by making the model predicting conservatively. Previous models like LLaVA-1.5 also mainly reports the F1 score.
   • Table 4: The performance on REC and RES are clearly behind more recent models. For example, GLaMM [ref1] achieves 83.2 RES cIoU on RefCOCO TestA, and UNINEXT [ref2] achieves 89.4 REC accuracy (IoU>0.5) on RefCOCOg Test.

2. Although this work claims keypoints as one major decoding modality, there is no quantitative evaluation of keypoint detection.

3. The ablation study does not discuss the most important module designs, e.g., decoding with CoT vs. without CoT, how data mixture in VT-Instruct affects the final results.

4. The presentation is not suitable for an academic research paper. In sections 3 and 4, only a high-level overview of the method is introduced, with a few short paragraphs and 5 large-sized figures. After reading these sections, readers may get a glimpse of the model and data, but too many details are missing or hidden in the appendix.

5. The qualitative results mainly show simple cases with few objects clearly visible. Even so, there seems to be many artifacts in the visualization (e.g., duplicate "person" detections in Figure 8 top right, low-quality masks in Figure 9 top right). It is unclear how the model perform in more challenging natural scene images.

[ref1] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, Fahad S. Khan. GLaMM: Pixel Grounding Large Multimodal Model. In CVPR, 2024.

[ref2] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, Huchuan Lu. Universal Instance Perception as Object Discovery and Retrieval. In CVPR, 2023.