MiniGPT-v2: Large Language Model as a Unified Interface for Vision-Language Multi-task Learning

Q1 Comparative Advantage Over Specialized Methods

While specialized models like BEIT-3, X-VLM, and Co-DETR demonstrate impressive task-specific performances, MiniGPT-V2 offers a unified, versatile platform that can handle a variety of tasks without the need for multiple specialized systems. One advantage of MiniGPT-v2 compared to a simple combination of expert models is that MiniGPT-v2 is able to merge different abilities to handle compositional tasks.

Fig. 4 in the appendix of the updated paper shows that after successfully detecting the cow that the user refers to in a photo with multiple cows by "[refer] the right row", MiniGPT-v2 can answer the following questions about that specific cow based (prior vision-language context). Such a task cannot be done by a VQA expert model, as they are not trained to understand the detection result. In addition, by merging multiple vision-task capabilities into one single large language model, there are more advanced vision-language abilities (advanced reasoning and in-contextual learning, etc) that can also be potentially activated in the future, which could not be found from those specialized methods.

Q2 The role of task identifiers

The incorporation of task identifiers such as “[vqa]” and “[grounding]” in MiniGPT-V2 is primarily to enhance model learning efficiency and reduce ambiguity in diverse visual-language tasks, as demonstrated in Table 5 (our main paper). These task identifiers serve as a form of regularization for different vision-language task inputs and should not affect the model's conversational capabilities.

While MiniGPT-V2 utilizes task identifiers for improved performance across different tasks, it is important to note that the system is also adept at handling identifier-free interactions. Users can engage in a wide range of vision-language tasks without the need to adding task identifiers.

In addition, we conduct a new ablation study where we remove the task identifiers in the third stage to encourage our MiniGPT-v2 to take inputs without the task identifier. Results in the following table below show that this variant of MiniGPT-v2 can conduct different tasks without the task identifiers and achieve a performance close to the original version. But overall, keeping the task identifiers can still lead to better performance.

Q3 Compared to Lynx and Qwen-VL

We thank the reviewer for the references. A brief clarification: ICLR24's guidelines classify works from the past four months, including Lynx and Quen-VL, as contemporaneous, exempting authors from comparing to them. Details at: https://iclr.cc/Conferences/2024/ReviewerGuide. For example, Qwen-VL arxiv paper is released two weeks before the paper submission deadline.

Appreciating these concurrent attempts, we looked into Lynx and Qwen-VL in detail and we observed that they share some similar training strategies to our work. However, there are still several differences:

1. Consistency in image resolution: Unlike Lynx and Qwen-VL, which progressively increase image resolution during training, our model maintains a constant resolution of 448x448 throughout the whole training process.
2. The data sampling strategy is different across different stages. The sampling strategy for Qwen-VL: Qwen-VL only trains the data focusing on image caption data. Second-stage trains on diverse multi-task datasets, but still including many noisy image captions and visual grounding data such as GRIT. In the third stage, they only train the model with the instruction finetuning and dialogue dataset. The sampling strategy for our model: In the first stage, we sample the data covering noisy and very fine-grained data such as image captions, visual grounding and many visual question answering, etc. In the second stage, we only sample the fine-grained multi-task data to improve the performance over diverse multiple vision-language tasks. In the third stage, we train both the finegrained multi-task data and also the visual instruction and dialogue data.
3. Training vision encoder:. Qwen-VL trains the vision encoder in the first two-stage training, while our model always remains the vision encoder frozen throughout the whole training.

Apart from the difference in the three-stage training, there are many differences on the model architectures, training data.

I appreciate the author's response to my previous comments and raise my rating to 5. However, after a detailed review, I feel that my concerns have not been entirely addressed. I will elaborate on my points and leave additional discussion and decision to other reviewers and PCs.

1. My comparison was not between MiniGPT-V2 and any single specialist model. Rather, I compared MiniGPT-V2, which utilizes identifiers like [vqa], [grounding], and [detection], with a multi-modal LLM agent augmented by these specialist models. The key is how an agent, when given a specific task, can leverage specialist models for higher efficiency and performance.

2. The design of LLMs is to understand users' intentions from text inputs (dialogue history) alone and to complete a variety of tasks. The use of these identifiers (for 3 tasks) seems contrary to this design.

3. GPT-4V does not rely on such identifiers to achieve its capabilities.

Furthermore, the authors' response that “MiniGPT-v2 can conduct different tasks without the task identifiers and achieve a performance close to the original version. But overall, keeping the task identifiers can still lead to better performance.” seems to align with my earlier review that “as presented in Table 5, the addition of these explicit identifiers results in a relatively marginal performance improvement”.

Regarding the references to Qwen-VL and Lynx, while Qwen-VL was released just two weeks before the ICLR deadline, Lynx has been available since July. The author's additional explanations on training differences are noted, yet the significance of these differences remains unclear to me. (There are some other multi-modal LLMs released in previous months except these two.)

Lastly, I observed that the ablation study results provided (224 vs. 448) do not align with the specific experiment I had initially requested (the proposed method vs. Resampler).

Q7 CHAIR hallucination metrics and three different prompts evaluation in MiniGPT-v2.

In our study, we employ the evaluation metrics for hallucination as defined in [1] to assess the performance of MiniGPT-v2. Specifically, we use CHAIR_i and CHAIR_s metrics to quantify hallucination at different levels:

CHAIR_i (Instance-level Hallucination): This metric evaluates the degree of hallucination at the object instance level. It is calculated as the ratio of hallucinated objects to the total number of objects mentioned in the caption. Mathematically, it is expressed as:

$CHIAR_i = \frac{| \{\text{hallucinated objects}\} |}{| \{\text{all mentioned objects}\} |}$

CHAIR_s (Sentence-level Hallucination): This metric assesses hallucination at the sentence level. It is the ratio of captions containing hallucinated objects to the total number of captions generated. It is given by: $CHAIR_s = \frac{| \{\text{captions with hallucinated objects}\} |}{| \{\text{all captions}\} |}$

​​Len (Length of Caption): This refers to the number of words in the generated caption, providing a measure of caption verbosity or conciseness.

By adding the different task identifiers, we observe that MiniGPT-v2 can generate the caption with different granularity, which is also the unique advantage of MiniGPT-v2 compared with other models.

1. Without Task Identifier: MiniGPT-v2 tends to generate captions with rich details
2. With [grounding] Identifier: The model produces captions more grounded to the image objects. However, this mode exhibits a reduction in the description of visual attributes and relationships compared to the mode without any task identifier.
3. With [caption] Identifier: This tends to generate very short image caption.

[1] Li, etc. Evaluating Object Hallucination in Large Vision-Language Models.

Q8 Qualitative results in the appendix Thank you for pointing out the discrepancy regarding the location of our qualitative results. In our initial submission, these results were indeed included in as a separate file in the supplementary materials. However, for easier accessibility and reference, we have relocated them to the appendix in our revised manuscript.

Q1 More performance evaluation for other task identifiers

We also provide the ablation results on [refer] identifiers. The results are demonstrated in the following table, and it can be observed that with the [refer] task identifier can overall improve the model performance on RefCOCO/+/g by 1.3 acc, indicating the efficiency of using [refer] task identifier.

Q2 Discussion is a bit weak, for example, the paper should have some discussion on the error analysis and some detail on the strengths and weaknesses.

Thank you for your constructive feedback. In light of your comments, we have revised our paper to provide a more robust discussion. It includes the discussion of strengths and weaknesses including the discussion regarding the training efficiency, robustness, and hallucination.

Q3 Explanation of the evaluation metrics and dataset. We add a more detailed explanation of the evaluation metrics and the dataset in the appendix. This consists of the evaluation metrics for visual question answering, RefCOCO, and hallucination. It also provides a rich explanation of the dataset.

Q4 Evaluation on a subset of the tasks

In our study, we mainly focused our evaluation on the tasks of visual question answering and referring expression comprehension. This decision was driven by two key considerations:

1. Representative Tasks: These two tasks are quite essential in evaluating the foundational capabilities of vision-language models, specifically in visual knowledge querying and visual grounding.

2. Benchmark Availability: The tasks such as detailed image captioning, detailed grounded image captioning, and object parsing and grounding, etc, lack well-established benchmarks. It's important to note that the capabilities demonstrated in those tasks, being only emerged in the large language models. Hence it lacks standardized evaluation metrics to evaluate their performance. Therefore, we focus on the well-established benchmarks such as various VQA and RefCOCO

Q5 The paper used single evaluation metrics to benchmark, which is not very comprehensive In our study, we have employed three different evaluation metrics to evaluate our model in terms of open-ended VQA, visual grounding, and hallucination evaluation. (A more detailed explanation of these three evaluation metrics can be found the in Appendix). All of these evaluation metrics together can contribute to a more comprehensive evaluation of our model.

Q6 Explain that your model does not perform well on RefCOCO+ in Table 4.

Here are the dataset features of RefCOCO, RefCOCO+, and RefCOCO(g):

1. RefCOCO has expressions that are generally simpler and may include location-based descriptions.
2. RefCOCO+ is designed to exclude location-based expressions. The referring expressions here are more complex, focusing on describing the appearance of objects rather than their location.
3. RefCOCO(g) has the referring expressions that focus on more general and longer descriptions.

As we can see from Table 4, the performance of MiniGPT-v2 on RefCOCO+ also performs well and can outperform Shikra on Test-B split. The results in Table 4 indicate that the general performance of MiniGPT-v2 on RefCOCO+ does not perform as well as RefCOCO and RefCOCO(g). From the difference among RefCOCO, RefCOCO+, and RefCOCO(g), we hypothesize that our model performs better on RefCOCO and RefCOCO(g) because our model is stronger in processing location-based referring expressions and longer descriptions compared to other baseline models, which can contribute to our model design and training strategy.

Q1 Making visual tokens even smaller (8 and 16 tokens):

We perform the ablation with concatenating more adjacent visual output tokens, e.g. 8 and 16. We provide the results in the following table. Our findings reveal that with merging more tokens, visual question answering and visual grounding performance can also drop. It's particularly noteworthy that the impact on visual grounding is more pronounced, which might because merging more tokens make it lose spatial understanding.

Q2 Comparing with training on the 224x224 image resolution

We evaluate the model with training 224x224 images (without any token merging), and we demonstrate the results in the following table,

From the results shown in the table, we can see that training on 448x448 images has similar performance on visual question answering, but it has much higher (+10.2 acc) visual grounding performance than training on 224x224 images.

Q3 Random Baseline selection

Thank you for your question regarding the selection of baseline models. Here are the key points addressing your concerns:

Table 4: Our choice of baseline models for Table 4 aligns with the evaluation benchmarks established in the work [1]. This decision was made to ensure a direct and consistent comparison with the existing works

Table 6: We incorporated evaluation results and baselines from the work [2]. This is to make it consistent with previous established evaluation benchmarks.

missing mPLUG-Owl baseline: ​​We realize it important to compare with mPLUG-Owl baseline and have included it in our updated paper (Table 3).

[1] Chen, etc.. Shikra: Unleashing multi-modal llm’s referential dialogue magic. [2] Li, etc. Evaluating Object Hallucination in Large Vision-Language Models.

Q4 Missing MIMIC-IT baselines.

Thank you for pointing out this work. MIMIC-IT is a dataset with 2.8 million multi-modal instruction-response pairs, predominantly fine-tuned on the Otter model. Acknowledging its importance, we have revised our paper (in the table 3) to include a comparison with the Otter model. And the results demonstrate that our model consistently outperforms the Otter model in the VQA performance.

Q5 Explain [INST] vs [/INST]

In Section 3.2, “[INST]” and “[/INST]” are critical markers within the LLaMA-2 language model. “[INST]” is used to indicate the start of user input, defining the user role in the interaction. This token signals that the following text is a user-generated query or statement.

On the other hand, “[/INST]” marks the beginning of the assistant's response, representing the assistant role. This token signals that the following text is a user-generated query or statement.

Q6 Task identifier and spatial location tokens representation

In our research, we did not introduce any new tokens to the vocabulary. We exclusively utilized the original token vocabulary provided by LLaMA-2.

Q7 Ethical concerns.

We recognize your concerns regarding potential overlap in the cited literature due to the shared domain of language and vision models. We assure you that any resemblance was unintentional and not a result of plagiarism. Our paper's focus on vision-and-language multitask learning necessitates referencing key foundational models, which leads to some inevitable overlap in references with MiniGPT4.

In response to your feedback, we have revised the whole related work section, and also replaced the discussion on advanced language models with a focus on multi-task vision-language models.

Q1 Zeroshot evaluation of minigpt-v2

In table 3 of the main paper, many VQA benchmarks have been evaluated in a zeroshot manner. Such as VSR, TextVQA, IconVQA, VizWiz and HateMeme.

To see if all the supervised data could be removed and evaluate the model purely in a zero-shot manner, we conduct the experiment on only training caption data (LAION, CC3M, SBU) and weakly labeled visual grounding data (GRIT-20M), and we perform the zeroshot visual question answering evaluation and also RefCOCO evaluation. We train all the models on 4XA100 GPUs for 150,000 steps. We demonstrate the VQA and RefCOCO results in the following table.

From the results shown in the table, we can see that without using supervised data, we observe performance drop on the visual question answering and visual grounding tasks.

Q2 Freezing the vision encoder or not

We also ablated by finetuning the vision encoder during the model training. We evaluate the performance on VQA benchmarks. The results are demonstrated in the following table, showing that fine tuning the vision encoder performs similar to without finetuning the model, with only 1.1 acc performance drop in average of all the VQA evaluation. The performance drop might due to the challenge of optimizing the coupled vision encoder and large language model.