MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Thank you for your valuable feedback! We response to the questions as follows:

Q1 It would be beneficial to compare the proposed model with more recent open-sourced components that utilize similar base LLMs, such as LLaVA, InstructBLIP, and mPlug-Owl. It is recommended to include evaluation on recent multimedia benchmarks such as MMBench.

As mentioned by Reviewer 7oDs, MiniGPT4 has been evaluated in the MMBench benchmark paper, and also in other benchmarks like MME and VisIT-Bench, together with similar baseline methods like LLaVa and InstructBLIP. Please refer to Q0 in the general response for a detailed discussion.

Q2 What criteria did you use to select the Advanced Abilities tasks? Additionally, can you clarify if the images used for evaluation in these tasks overlap with the dataset used in stage 1 pre-training and stage 2 fine-tuning?

For our study, we chose advanced abilities that necessitate both an understanding of visual content and complex language generation. We gathered the images for our dataset using Google Image Search, employing related keywords such as "funny memes" and "photos of delicious food."

Furthermore, it's important to note that the training set consists solely of image description texts and does not include any text data pertaining to advanced abilities. MiniGPT4 was not trained with image-text paired data to learn these skills. However, it gains this skill compositionally by efficiently mapping images to the embedding space that the LLM (frozen by default) can operate on.

Q3 In Table 2, the approach of randomly selecting a ground-truth caption and asking ChatGPT if the generated text covers the main visual concepts in the label text may lead to instances where the selected "ground-truth" caption and the generated text depict the image in different aspects, yet both are considered correct. Reporting standard metrics such as BLEU and Rouge scores, which consider multiple candidate ground-truth captions, could mitigate the issues. Could you specify the version of ChatGPT that you adopted for answer revision in your experiments?

In Table 2, we focus not on the correctness of the generated captions but on their information coverage. A high score indicates that the generated descriptions encompass more information from the image, rather than “more correct”. For this evaluation, we employed the API gpt-3.5-turbo.

Traditional metrics such as BLEU and Rouge, which rely on sentence similarity, fall short when the format of evaluated image descriptions diverges from the ground truth. This discrepancy is particularly noticeable when the generated descriptions are significantly longer than the original ground truth captions. We designed a new metric based on GPT-4 that takes all the ground truth captions into account. Please refer to Q2 in the general response section for details.

Q4 Table 5 lacks the CHAIR score for different methods.

Kindly refer to Q5 in the general response where we include results from more methods.

Q5 MiniGPT-4 fails to follow the instruction's requirement of 20 words, as evidenced by the average length of 27 words for MiniGPT-4 (short). To me, satisfying the user requirement of a basic instruction should be of higher priority.

The ability to generate a sentence with exactly 20 words is constrained by the LLM's capabilities. While SOTA closed-sourced models like GPT-4 perform well in this task, open-source models like Llama2 13B struggle. We conducted a small experiment with Llama2 13B, instructing it to "generate a sentence with 20 words". The average number of words over the 50 generations is only 14.8 words. This seemingly simple instruction proves challenging for a 13B open-source LLM. Since we freeze the open-source LLM on MiniGPT-4, it's unsurprising that MiniGPT-4 can't precisely generate a 20-word caption. Nonetheless, this prompt still encourages MiniGPT-4 to significantly reduce the image description length from 175 to 28.8 words

Q6 In Section 3.2, the paper claims that no equivalent datasets exist (for multi-modal instruction tuning) in the vision-language domain. However, there have been recent studies addressing this problem, such as LLaVA, InstructBLIP, M3IT, and MultiInstruct. To accurately reflect the current state of the field, the author should properly cite these studies and consider exploring further fine-tuning with these datasets.

At the time when this project was developed, there was no other multi-modal instruction tuning dataset that existed. We updated section 3.2 (highlighted in blue) in the updated paper to better reflect this background. In addition, we updated the related work section to better represent the current landscape. Furthermore, we carried out a new experiment, substituting our second-stage dataset with a newly-released image description dataset from LAION. This dataset features descriptions generated by GPT-4v. For detailed information, please refer to Q3 in the general response section.

Thank you for your comment! To verify more directly, we conducted a simple experiment with Vicuna-13B, the LLM used by MiniGPT-4. We employed the prompt "Introduce me an animal in 20 words." Across 20 samples, the average length of the responses generated was 29.6 words, similar to the average length (28.8 words) in MiniGPT-4's generation from our ablation study. These results suggest that the over generating issue is due to the limitations of MiniGPT-4's LLM.

Thank you for your valuable feedback! We response to the questions as follows:

Q1 The comparisons between BLIP-2 and MiniGPT are unfair because BLIP-2 uses a weaker LLM and is not designed for the advanced abilities mentioned in the paper.

The essential message our paper conveys to the community is as follows: advanced multimodal capabilities can be attained by utilizing a stronger LLM with advanced pure language skills, even if the LLM is not specifically designed for advanced multimodal capabilities. The particularity of our design choices and the details of making it work in a two-stage manner are at the heart of our work/contributions. In addition, we provide a discussion of MiniGPT-4 in benchmarks like MMBench and MME in our general response. Kindly refer to Q0 of the general response for more details.

Q2 The designed new evaluation method using ChatGPT to check whether the generation covers all the objects and visual relations is not compelling. It is possible because many hallucinations are generated, as Table 5 illustrates.

From Q5 in the general response, we can see that the hallucination issue of MiniGPT-4 is less obvious compared to other baseline models like LLava or mPLUG-Owl. In addition, we introduce a more robust metric that better considers all the ground truth caption into account with a stronger judger GPT-4-turbo. Please refer to Q2 in the general response.

Q3 The necessity of using a large number of image and text pairs to train a projection layer is not addressed sufficiently.

Kindly refer to Q4 in the general response section, where we respond experimentally to this question.

Q4. The performance drops a lot when three projection layers are used instead of one. Why the drop is so huge?

The three-layer variant almost triples the learning capacity (the number of learnable parameters). This increase may necessitate more training data compared to the original one-layer version. Consequently, training this variant on the same amount of data as the original version might not suffice to achieve comparable performance.

Thanks for your comment. Regarding Q4, our design with a single linear projection layer aligns with previous studies like [1][2]. These works already demonstrated that training only a single linear projection is sufficient for transferring image representations to a frozen LM space. Similarly, another concurrent work, LLaVa, uses a single projection layer to align visual and language features. The significant performance decline observed in AOK-VQA may be attributed to increased training capacities, suggesting the need for more training data than our original single-layer version required. As the reviewer suggests, we will expand our discussion in the final version to include ablations on alternative projection designs and the use of additional alignment data.

[1 ]Merullo, Jack, et al. Linearly Mapping from Image to Text Space. ICLR 2023.

[2] Scialom, Thomas, et al. What BERT sees: Cross-modal transfer for visual question generation. INLG 2020.

Thank you for your valuable feedback! We response to the questions as follows:

Q1 Rigorous benchmarking on different downstream tasks is needed. For reference, MMBench. More baselines are needed.

As mentioned by Reviewer 7oDs, this project has been evaluated in the MMBench benchmark paper, and also in other benchmarks like MME and VisIT-Bench, together with similar baseline methods like LLaVa and InstructBLIP. Kindly refer to Q0 in the general response for a detailed discussion.

Q2 Why not use GPT-4 itself to get the image captions and perform knowledge distillation as stage 2?

Kindly refer to Q3 in the general response for the new experiment.

Q3 Does MiniGPT-4 apply a unique projection layer to all output vectors corresponding to the learned queries? Or some other processing steps are involved?

The projection layer is applied to Q-former’s output vectors (tokens), which correspond to the Q-former’s internal learnable queries. This projection layer converts each of the output tokens individually to match the input size required by the LLM model.

Thank you for your valuable feedback! We response to the questions as follows:

Q1 Extend the model’s capacity to accommodate multiple visual inputs.

Kindly refer to Q1 in the general response section.

Q2 Evaluation Setup

Kindly refer to Q0 in the general response section.

Q3 What is the maximum input and output length for the training and inference stage?

The context length is limited by the LLM itself. For the vicuna version we use which is based on Llama 1, the context length is 2048 tokens (input + output). To ensure the training process does not exhaust GPU memory, we truncate the output to a maximum of 160 tokens if it exceeds this limit during training.

Q4 In Section 3.2, the authors mention they would examine if the generated sentence exceeds 80 tokens or not – how is the length of 80 determined? Is it through empirical observations?

The choice of this threshold is grounded in empirical observations. During our initial explorations, we manually examined the sentences generated and noticed that descriptions shorter than 80 tokens tended to be incomplete. Consequently, we established the threshold at 80 tokens.

Q5 What’s the length of the learned visual queries from the Q-Former in Mini-GPT4?

It is 32.

Q6 Missing reference.

We have updated the related work section in the paper (highlighted in blue) to include more related references.

In the MME benchmark, which focuses more on advanced multi-modal reasoning abilities like visual code reasoning and counting, MiniGPT-4 exhibits impressive performance. The MME benchmark is based on the Elo rating system. Here, we list the rank of MiniGPT-4 in tasks where MiniGPT-4 is top 4 and tasks where MiniGPT-4 is bottom 5.

In this context, MiniGPT-4 has achieved the top rank in the overall leaderboard for cognition and is ranked first in 2 out of 14 subtasks (code reasoning, position reasoning). It also secured top 3 positions in 2 out of 14 subtasks (Counting, Color recognition), and top 4 in 2 out of 14 tasks (OCR, Numerical Calculation), among 11 other baselines including LLaVa, mPLUG-Owl, InstructBLIP, VisualGLM, Otter, etc.

However, MiniGPT-4 does not perform as well in subtasks like Poster Recognition (rank 10), Artwork Recognition (rank 9), and Scene Recognition (rank 9), and Landmark Recognition (rank 8). These results suggest that MiniGPT-4 excels in cognition-related multi-modal tasks but is relatively weaker in fine-grained recognition tasks such as movie poster or artwork recognition. The strength of MiniGPT-4 in cognition tasks may be attributed to the frozen LLM component. Since the LLM isn't fine-tuned, its reasoning ability remains unaltered during multi-modal training. The comparatively weaker performance in fine-grained recognition tasks might also be a consequence of the frozen LLM. Without fine-tuning, aligning the visual module with the LLM for fine-grained and less frequently occurring concepts or objects, like artworks, can be challenging. Identifying ways to better balance cognition-related tasks with fine-grained recognition tasks presents a significant research opportunity, and we aim to explore this in future studies

[1] MMBench: Is Your Multi-modal Model an All-around Player?

[2] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

[3] Visual Instruction Tuning, https://arxiv.org/abs/2304.08485

[4] InstructBLIP: Towards general-purpose vision-language models with instruction tuning, https://arxiv.org/abs/2305.06500

Q1 Extends the model’s capacity to accommodate multiple visual inputs.

The MiniGPT-4 architecture design supports multiple image inputs. We tested this capability by fine-tuning MiniGPT-4 on VideoInstruct100K. For visual content representation, we chose 50 frames from each video and included any available subtitles. The model's final input comprised these frames and subtitles. We fine-tuned MiniGPT-4 using 1,200 video question-answer pairs, representing 1.2% of the original dataset's size. Its video understanding was assessed using the Video-based Generative Performance Benchmark, developed by Video ChatGPT. Contrary to our expectations, the results revealed that MiniGPT-4, fine-tuned for video, surpassed the state-of-the-art baseline, Video ChatGPT, in four of five metrics: correctness, detail, contextual understanding, and temporal understanding. It also performed well in consistency, despite being trained with only 1.2% of Video ChatGPT's data. These findings indicate that MiniGPT-4 is adept at processing multiple visual inputs.

Q2 In Table 2, the ChatGPT based metric doesn’t consider multiple candidate ground-truth captions.

Here, we introduce a new metric based on GPT4. We utilize GPT-4 turbo (gpt-4-1106-preview) to assess whether the generated descriptions capture the content of each ground truth caption individually. In the COCO dataset, each image is accompanied by 5 ground truth captions. For every image, we calculate the number of captions covered by the generated descriptions and then average this count across all 5000 test images to derive the final score. Our experimental results shown in the table below demonstrate that MiniGPT-4 covers more ground truth captions compared to Blip-2.

Here is the prompt we use in GPT-4 turbo

Given a test image description and a list of gt image caption,
verify whether the information in gt caption is included in the test description.
The answer should be yes or no.
Input is in this format: 
Test: (test sentence)
1: (gt1)
2: (gt2)
3: (gt3)

you need to answer yes or no for each gt in the following format:
1: (yes/no)
2: (yes/no)
3: (yes/no)

Q3 Instead of having the data post-processing steps for stage 2 training, why not use GPT-4 itself to get the image captions and perform knowledge distillation as stage 2?

We agree with the reviewer that this suggestion is reasonable. GPT4-V was not available during the time when this project was developed, but we can now do this. We conducted new experiments using an image-text pair dataset collected by LAION, available at [https://huggingface.co/datasets/laion/gpt4v-dataset]. The image descriptions in this dataset were generated by GPT-4v. Currently, the dataset size is increasing, and at the time of our experiments, it contained 2,000 image-text pairs.

We used this dataset to replace the original one in stage-2, while maintaining all other fine-tuning hyperparameters. The model was fine-tuned starting from the same stage 1 weights as the original version. We evaluated this variant using the metric we proposed in Q2. The experimental results below indicate that fine-tuning on the image descriptions generated by GPT-4v can enhance the model's performance.

Q4 Is it possible to use less data from the combined dataset to train the projection layer in the first pertaining stage? It is better to provide insights on the size data necessary to align the vision and text space.

We conducted a new ablation analysis focusing on the volume of training data used in the first stage. We selected checkpoints from midway through the first stage of training, corresponding to 10%, 30%, and 50% of the original stage 1 training duration. These checkpoints were then fine-tuned using the stage 2 training process.

The experimental results demonstrate that reducing the first stage training data to 10% results in a significant performance decline. However, with 30% of the original first stage training, which is approximately 1.5 million image-text pairs and requires 3 hours of training on 4 A100 GPUs, we can achieve performance close to that of the original model. We don’t observe further improvement after reaching 50% of the first stage training. This finding suggests that the learable capacity (the one linear mapping layer), may reach saturation at this point.

Q5 Hallucination Comparison

Here, we extend table 5 in the main paper to include more baseline models in the hallucination evaluation. Compared to contemporary methods like Llava or mPlug-Owl, MiniGPT-4 generates longer descriptions with fewer hallucination.